Introduction
Regression analysis is one of the most widely used statistical methods and has extensive applications across many fields. It investigates functional relationships between output (response) and input (explanatory) variables, so that one can model and describe these relationships or can predict new response values, see Chatterjee and Hadi [5 ] , Klemelä [19 ] , James et al. [16 ] .
In this paper, we propose a semiparametric regression method that seeks both interpretability and flexibility by incorporating a machine learning technique. To outline the proposed method, Figure 1 illustrates its application to a simple estimation problem in a one-dimensional normal regression setting. Our regression estimators are based on a “parametric” crude guess and its “nonparametric” adjustment: panel (a) displays a scatterplot of the data points and the least squares regression line m ( x , θ ^ ) = θ ^ 0 + θ ^ 1 x m\left(x,\widehat{\theta}\right)=\widehat{\theta}_0+\widehat{\theta}_1x m ( x , θ ) = θ 0 + θ 1 x with θ ^ = [ θ ^ 0 θ ^ 1 ] T \widehat{\theta}=\left[\widehat{\theta}_0\ \widehat{\theta}_1\right]^{\mathsf{T}} θ = [ θ 0 θ 1 ] T , an example of a parametric regression estimator, represented by a dotted line. Here the parameters θ = [ θ 0 θ 1 ] T \theta=\left[\theta_0\ \theta_1\right]^{\mathsf{T}} θ = [ θ 0 θ 1 ] T in m ( x , θ ) = θ 0 + θ 1 x m\left(x,\theta\right)=\theta_0+\theta_1x m ( x , θ ) = θ 0 + θ 1 x are estimated to be θ ^ 0 ≈ 0.05 \widehat{\theta}_0\approx0.05 θ 0 ≈ 0.05 and θ ^ 1 ≈ 1.08 > 0 \widehat{\theta}_1\approx1.08>0 θ 1 ≈ 1.08 > 0 , suggesting that the data points globally increase with x x x in a sense. By using parametric estimators as in linear models, we can interpret such global behavior of the regression function based only on a model function m ( x , θ ) m(x,\boldsymbol{\theta}) m ( x , θ ) and estimated parameters θ ^ \hat{\boldsymbol{\theta}} θ ^ . However, as observed in panel (a), the linear model above does not fit the data points well locally. Then, allowing for model misspecification, we adjust this initial “parametric” regression with a “nonparametric” approach, which enables estimators to capture broad classes of regression functions flexibly. After this combination of parametric and nonparametric approaches, the proposed method is referred to as “semiparametric.” In this paper, we implement the adjustment via a machine learning technique called “stagewise empirical risk minimization algorithm.” Briefly, from a user-prepared set (dictionary) of functions (words), this algorithm selects one word in each iteration, and then forms a convex combination of the selected words for estimation or prediction. Based on this algorithm and using a dictionary introduced in Section 5, we now construct a term ξ ^ I ( x ) \hat{\xi}_{I}(x) ξ ^ I ( x ) for additive adjustment with the number of iterations I = 5 , 40 , 150 I = 5, 40, 150 I = 5 , 40 , 150 , as presented in panel (b) of Figure 1 . Then we compute a proposed semiparametric estimator m ( x , θ ^ ) + ξ ^ I ( x ) m(x,\hat{\boldsymbol{\theta}}) + \hat{\xi}_{I}(x) m ( x , θ ^ ) + ξ ^ I ( x ) shown in panel (a), demonstrating that the regression line is adjusted and that the proposed estimator seems to fit the data points better as I I I increases.
Figure 1. An example of regression with our proposed method. “Parametric” regression is conducted by using the least squares regression line m ( x , θ ^ ) = θ ^ 0 + θ ^ 1 x m\left(x,\widehat{\theta}\right)=\widehat{\theta}_0+\widehat{\theta}_1x m ( x , θ ) = θ 0 + θ 1 x . Its “nonparametric” adjustment function ξ ^ I ( x ) \widehat{\xi}_I(x) ξ I ( x ) is implemented via the “stagewise empirical risk minimization algorithm” with the number of iterations I = 5 , 40 , 150 I=5,40,150 I = 5 , 40 , 150 . Then we obtain a proposed semiparametric estimator m ( x , θ ^ ) + ξ ^ I ( x ) m\left(x,\widehat{\theta}\right)+\widehat{\xi}_I(x) m ( x , θ ) + ξ I ( x ) with additive adjustment.
Now we describe each of the parametric, nonparametric and semiparametric approaches. The parametric approach is fundamental in regression analysis. It models the target regression function as a function determined by a finite-dimensional parameter, as in linear models. This approach reduces the problem of estimating a “function” to that of estimating a “parameter vector” contained in the model, and parametric regression estimators are constructed by plugging an estimator for the parameters (such as the maximum likelihood estimator (MLE) in McCullagh and Nelder [21 ] , Dunn and Smyth [8 ] ) into the model function. Its simple structure facilitates interpretation of the underlying regression function: for example, whether it increases, decreases or hardly changes globally with some explanatory variables. Particularly in linear regression, the LASSO estimator is known to estimate several coefficients for explanatory variables to be exactly 0, so that this estimator can be utilized for variable selection, see Tibshirani [27 ] , Hastie et al. [14 ] , Hastie et al. [13 ] . Although this parametric approach has the merits of simplicity and interpretability and is widely used in statistics, this approach has a drawback: the assumption that the underlying structure can be captured by a finite-dimensional parameter vector is sometimes very restrictive. It is well known that parametric estimators generally lack consistency under model misspecification.
On the other hand, the nonparametric approach offers flexibility in terms of consistency. This approach includes several methods for regression analysis, such as kernel smoothing, spline smoothing, gradient boosted regression trees (GBRTs). In general, by appropriately tuning smoothing parameters, these methods can achieve consistency for broad classes of regression functions, which makes the nonparametric approach advantageous. However, the structure of nonparametric estimators is sometimes more difficult to interpret than that of parametric models. In addition, several nonparametric methods suffer from data sparsity for high-dimensional explanatory variables, a phenomenon referred to as “curse of dimensionality,” see, for example, Chapter 3.2.6 of Klemelä [19 ] .
From the above observations, the parametric and nonparametric approaches each have their own advantages and disadvantages. This motivates us to consider a fine combination of these two approaches, in order to make regression estimators both easily interpretable and flexible. Specifically, we consider a semiparametric approach in which an initial crude estimation with the parametric approach is subsequently adjusted through the nonparametric approach. The underlying idea can be found in several pioneering works. Focusing on the additive adjustment above, this approach coincides with performing an additional regression on the residuals resulting from the initial regression. This strategy appeared, for example, in Section 5C of Tukey [28 ] , which fits lines to an original data set and then to the resulting residuals. Boosting also applies this strategy by iteratively fitting the residuals, see Friedman [9 ] and Bühlmann and Yu [4 ] . The semiparametric approach based on this idea has been explored in several studies, including [23 , 11 , 32 ) and references therein] . In particular, Fan et al. [11 ] unified broad classes of semiparametric estimators in kernel smoothing: they incorporated a quasi-likelihood method including Bernoulli (logistic) and Poisson regression; they addressed both additive and multiplicative adjustment; and, through local polynomial smoothing, they proposed semiparametric variants of the Nadaraya–Watson estimator, local linear fitting and so on. Fan et al. [11 ] also noted on p.4154 that “Here we focus on local polynomial techniques although the idea can be extended to other nonparametric methods,” thereby indicating that the nonparametric part of the semiparametric approach can be implemented with some techniques other than local polynomial ones.
As the nonparametric adjustment in the above semiparametric approach, we utilize a stagewise empirical risk minimization algorithm studied by Klemelä [17 ] , Klemelä [18 ] , Naito and Eguchi [24 ] , Nishida and Naito [26 ] in density estimation and by Yoshida and Naito [33 ] in regression analysis. This machine learning technique has several advantages, as pointed out in these studies: (i) Yoshida and Naito [33 ] utilized the empirical risk derived from the Bregman divergence, which allows their proposed estimators to be applied widely in various settings, such as Bernoulli and Poisson regression. (ii) There are almost no constraints on preparing the dictionary. (ii-1) For instance, dictionaries can be designed to automate smoothing parameter selection in conventional estimators, through the algorithmic selection of words. In density estimation, Nishida and Naito [26 ] utilized a dictionary composed of scaled kernel functions with several bandwidths and they conducted bandwidth selection analogous to that in kernel density estimation. (ii-2) Another utilization of dictionaries is for variable selection. Yoshida and Naito [33 ] constructed a dictionary of univariate B B B -spline bases corresponding to each explanatory variable and demonstrated variable selection with this dictionary for sparse additive models.
In addition to proposing such a semiparametric method, we aim to investigate its performance both theoretically and numerically. In the theoretical study, we develop a non-asymptotic error bound for the proposed estimators as in Klemelä [18 ] , Naito and Eguchi [24 ] , Nishida and Naito [26 ] and Yoshida and Naito [33 ] . At the same time, we examine the asymptotic behavior of the bound, even under general settings where the dictionary and words may depend on the data and sample size. The numerical study explores accuracy of the proposed estimators with concrete dictionaries, by comparing them with several other regression estimators. It also addresses behavior of our estimators in terms of variable selection, when utilizing a dictionary composed of univariate words.
This paper is organized as follows. Section 2 describes the Bregman divergence. Using this divergence, the proposed semiparametric estimators are formulated in Section 3. Their theoretical results are developed in Section 4. Section 5 provides some concrete dictionaries for implementing the estimators. Section 6 discusses simulations investigating the finite-sample performance of our estimators in several settings. Applications to real data sets are presented in Section 7. Section 8 offers concluding remarks. All proofs and calculations for the theoretical results are provided in the Appendix.
Risk functions based on the Bregman divergence
Let Y ∈ R Y \in\mathbb{R} Y ∈ R be a response variable and X ∈ R d \mathbf{X} \in\mathbb{R}^{d} X ∈ R d be a vector of explanatory variables. We are given i.i.d. pairs of observations X = { ( Y 1 , X 1 ) , … , ( Y n , X n ) } ⊂ R × R d \mathbf{X} = \{(Y_{1}, \mathbf{X}_{1}), \ldots, (Y_{n}, \mathbf{X}_{n})\} \subset\mathbb{R} \times\mathbb{R}^{d} X = {( Y 1 , X 1 ) , … , ( Y n , X n )} ⊂ R × R d drawn from a joint distribution with the density function f ( y , x ) = p ( y ∣ x ) q ( x ) f(y,\mathbf{x}) = p(y \mid\mathbf{x})q(\mathbf{x}) f ( y , x ) = p ( y ∣ x ) q ( x ) , where p ( ⋅ ∣ x ) p(\cdot\mid\mathbf{x}) p ( ⋅ ∣ x ) denotes the conditional density of Y Y Y given X = x \mathbf{X} = \mathbf{x} X = x , and q q q is the marginal density of X \mathbf{X} X with the support D ⊂ R d \mathbb{D} \subset\mathbb{R}^{d} D ⊂ R d . Our goal is to estimate the regression function μ ( x ) = E [ Y ∣ X = x ] \mu(\mathbf{x}) = \mathbb{E}[Y \mid\mathbf{X} = \mathbf{x}] μ ( x ) = E [ Y ∣ X = x ] at points x ∈ D \mathbf{x} \in\mathbb{D} x ∈ D .
To formulate the proposed estimator, we introduce a divergence measure between functions. For a strictly convex real-valued function U : R → R U : \mathbb{R} \to\mathbb{R} U : R → R , the functional Bregman divergence between the target μ \mu μ and a candidate estimator μ ˉ \bar{\mu} μ ˉ is defined as
D U ( μ , μ ˉ ) = ∫ R d d U ( μ ( x ) , μ ˉ ( x ) ) q ( x ) d x D_{U}(\mu,\bar{\mu}) = \int_{\mathbb{R}^{d}} d_{U}(\mu(\mathbf{x}),\bar{\mu}(\mathbf{x}))q(\mathbf{x})\,d\mathbf{x} D U ( μ , μ ˉ ) = ∫ R d d U ( μ ( x ) , μ ˉ ( x )) q ( x ) d x
where
d U ( μ ( x ) , μ ˉ ( x ) ) = U ( μ ( x ) ) − U ( μ ˉ ( x ) ) − u ( μ ˉ ( x ) ) { μ ( x ) − μ ˉ ( x ) } d_{U}(\mu(\mathbf{x}),\bar{\mu}(\mathbf{x})) = U(\mu(\mathbf{x})) - U(\bar{\mu}(\mathbf{x})) - u(\bar{\mu}(\mathbf{x}))\{\mu(\mathbf{x})-\bar{\mu}(\mathbf{x})\} d U ( μ ( x ) , μ ˉ ( x )) = U ( μ ( x )) − U ( μ ˉ ( x )) − u ( μ ˉ ( x )) { μ ( x ) − μ ˉ ( x )}
and u = U ′ u = U' u = U ′ , see Bregman [2 ] , Basu et al. [1 ] , Yoshida and Naito [33 ] , Naito and Penev [25 ] . It is known that the inequality d U ( μ ( x ) , μ ˉ ( x ) ) ≥ 0 d_{U}(\mu(\mathbf{x}),\bar{\mu}(\mathbf{x})) \ge0 d U ( μ ( x ) , μ ˉ ( x )) ≥ 0 always holds, and that d U ( μ ( x ) , μ ˉ ( x ) ) = 0 d_{U}(\mu(\mathbf{x}),\bar{\mu}(\mathbf{x})) = 0 d U ( μ ( x ) , μ ˉ ( x )) = 0 holds if and only if μ ( x ) = μ ˉ ( x ) \mu(\mathbf{x}) = \bar{\mu}(\mathbf{x}) μ ( x ) = μ ˉ ( x ) , see Bregman [2 ] and Basu et al. [1 ] . These two properties of d U d_{U} d U are also satisfied by the functional Bregman divergence D U D_{U} D U . Therefore, D U D_{U} D U can be regarded as a measure of discrepancy between functions, although it is not a metric in the mathematical sense. As discussed below, different strictly convex functions U U U yield different forms of D U D_U D U , such as the squared error and the loss function used in Bernoulli regression.
We next derive an empirical risk function based on D U D_U D U . To do this, we use an alternative expression for d U d_U d U :
d U ( μ ( x ) , μ ˉ ( x ) ) = d U ∗ ( u ( μ ˉ ( x ) ) , u ( μ ( x ) ) ) = U ∗ ( u ( μ ˉ ( x ) ) ) − U ∗ ( u ( μ ( x ) ) ) − μ ( x ) { u ( μ ˉ ( x ) ) − u ( μ ( x ) ) } = { U ∗ ( u ( μ ˉ ( x ) ) ) − μ ( x ) u ( μ ˉ ( x ) ) } − { U ∗ ( u ( μ ( x ) ) ) − μ ( x ) u ( μ ( x ) ) } , (1) \begin{aligned}
d_U\left(\mu(\boldsymbol{x}),\bar{\mu}(\boldsymbol{x})\right)
&=d_{U^*}\left(u(\bar{\mu}(\boldsymbol{x})),u(\mu(\boldsymbol{x}))\right)\\
&=U^*\left(u(\bar{\mu}(\boldsymbol{x}))\right)-U^*\left(u(\mu(\boldsymbol{x}))\right)-\mu(\boldsymbol{x})\left\{u(\bar{\mu}(\boldsymbol{x}))-u(\mu(\boldsymbol{x}))\right\}\\
&=\left\{U^*\left(u(\bar{\mu}(\boldsymbol{x}))\right)-\mu(\boldsymbol{x})u(\bar{\mu}(\boldsymbol{x}))\right\}-\left\{U^*\left(u(\mu(\boldsymbol{x}))\right)-\mu(\boldsymbol{x})u(\mu(\boldsymbol{x}))\right\}, \tag*{(1)}
\end{aligned} d U ( μ ( x ) , μ ˉ ( x ) ) = d U ∗ ( u ( μ ˉ ( x )) , u ( μ ( x )) ) = U ∗ ( u ( μ ˉ ( x )) ) − U ∗ ( u ( μ ( x )) ) − μ ( x ) { u ( μ ˉ ( x )) − u ( μ ( x )) } = { U ∗ ( u ( μ ˉ ( x )) ) − μ ( x ) u ( μ ˉ ( x )) } − { U ∗ ( u ( μ ( x )) ) − μ ( x ) u ( μ ( x )) } , (1)
where U ∗ ( s ) = sup z ∈ R { z s − U ( z ) } U^*(s)=\sup_{z\in\mathbb{R}}\{zs-U(z)\} U ∗ ( s ) = sup z ∈ R { z s − U ( z )} is the convex conjugate function of U U U , with useful properties such as ( U ∗ ) ′ = u − 1 (U^*)'=u^{-1} ( U ∗ ) ′ = u − 1 , see [15 ] . By using (2.1) and defining ℓ ( y , x ∣ μ ˉ ) = U ∗ ( u ( μ ˉ ( x ) ) ) − y ⋅ u ( μ ˉ ( x ) ) \ell(y,\boldsymbol{x}\mid\bar{\mu})=U^*(u(\bar{\mu}(\boldsymbol{x})))-y\cdot u(\bar{\mu}(\boldsymbol{x})) ℓ ( y , x ∣ μ ˉ ) = U ∗ ( u ( μ ˉ ( x ))) − y ⋅ u ( μ ˉ ( x )) , D U D_U D U can be calculated as
D U ( μ , μ ˉ ) = D U ∗ ( u ( μ ˉ ) , u ( μ ) ) = ∫ R d { ℓ ( μ ( x ) , x ∣ μ ˉ ) − ℓ ( μ ( x ) , x ∣ μ ) } q ( x ) d x = ∫ R × R d { ℓ ( y , x ∣ μ ˉ ) − ℓ ( y , x ∣ μ ) } f ( y , x ) d y d x . \begin{aligned}
D_U(\mu,\bar{\mu})&=D_{U^*}(u(\bar{\mu}),u(\mu))\\
&=\int_{\mathbb{R}^d}\{\ell(\mu(\boldsymbol{x}),\boldsymbol{x}\mid\bar{\mu})-\ell(\mu(\boldsymbol{x}),\boldsymbol{x}\mid\mu)\}q(\boldsymbol{x})\,d\boldsymbol{x}\\
&=\int_{\mathbb{R}\times\mathbb{R}^d}\{\ell(y,\boldsymbol{x}\mid\bar{\mu})-\ell(y,\boldsymbol{x}\mid\mu)\}f(y,\boldsymbol{x})\,dy\,d\boldsymbol{x}.
\end{aligned} D U ( μ , μ ˉ ) = D U ∗ ( u ( μ ˉ ) , u ( μ )) = ∫ R d { ℓ ( μ ( x ) , x ∣ μ ˉ ) − ℓ ( μ ( x ) , x ∣ μ )} q ( x ) d x = ∫ R × R d { ℓ ( y , x ∣ μ ˉ ) − ℓ ( y , x ∣ μ )} f ( y , x ) d y d x .
Then the empirical version of D U D_U D U based on the observations X \mathcal{X} X is given by
D ^ U ( μ , μ ˉ ) = 1 n ∑ i = 1 n { ℓ ( Y i , X i ∣ μ ˉ ) − ℓ ( Y i , X i ∣ μ ) } = 1 n ∑ i = 1 n ℓ ( Y i , X i ∣ μ ˉ ) + ⟨ Constant for μ ˉ ⟩ \begin{aligned}
\widehat{D}_U(\mu,\bar{\mu})=\frac{1}{n}\sum_{i=1}^{n}\{\ell(Y_i,X_i\mid\bar{\mu})-\ell(Y_i,X_i\mid\mu)\} \\
&=\frac{1}{n}\sum_{i=1}^{n}\ell(Y_i,X_i\mid\bar{\mu})+\langle\text{Constant for }\bar{\mu}\rangle
\end{aligned} D U ( μ , μ ˉ ) = n 1 i = 1 ∑ n { ℓ ( Y i , X i ∣ μ ˉ ) − ℓ ( Y i , X i ∣ μ )} = n 1 i = 1 ∑ n ℓ ( Y i , X i ∣ μ ˉ ) + ⟨ Constant for μ ˉ ⟩
and we obtain the empirical risk function
L ^ ( μ ˉ ) = 1 n ∑ i = 1 n ℓ ( Y i , X i ∣ μ ˉ ) . (2) \widehat{L}(\bar{\mu})=\frac{1}{n}\sum_{i=1}^{n}\ell(Y_i,X_i\mid\bar{\mu}). \tag*{(2)} L ( μ ˉ ) = n 1 i = 1 ∑ n ℓ ( Y i , X i ∣ μ ˉ ) . (2)
This function (2.2) takes different forms depending on the strictly convex function U U U : if U ∗ ( z ) = z 2 / 2 U^*(z)=z^2/2 U ∗ ( z ) = z 2 /2 , then u = id u=\operatorname{id} u = id and ℓ ( y , x ∣ μ ˉ ) = { μ ˉ ( x ) − y } 2 / 2 − y 2 / 2 \ell(y,\boldsymbol{x}\mid\bar{\mu})=\{\bar{\mu}(\boldsymbol{x})-y\}^2/2-y^2/2 ℓ ( y , x ∣ μ ˉ ) = { μ ˉ ( x ) − y } 2 /2 − y 2 /2 , so that (2.2) is equivalent to the squared loss; the function U ∗ ( z ) = ln [ 1 + e z ] U^*(z)=\ln[1+e^z] U ∗ ( z ) = ln [ 1 + e z ] leads to u ( z ) = ln [ z / ( 1 − z ) ] u(z)=\ln[z/(1-z)] u ( z ) = ln [ z / ( 1 − z )] and ℓ ( y , x ∣ μ ˉ ( x ) ) = − { y ln [ μ ˉ ( x ) ] + ( 1 − y ) ln [ 1 − μ ˉ ( x ) ] } \ell(y,x\mid\bar{\mu}(x))=-\{y\ln[\bar{\mu}(x)]+(1-y)\ln[1-\bar{\mu}(x)]\} ℓ ( y , x ∣ μ ˉ ( x )) = − { y ln [ μ ˉ ( x )] + ( 1 − y ) ln [ 1 − μ ˉ ( x )]} , making (2.2) the Kullback–Leibler divergence between Bernoulli distributions B e ( μ ( x ) ) Be(\mu(x)) B e ( μ ( x )) and B e ( μ ˉ ( x ) ) Be(\bar{\mu}(x)) B e ( μ ˉ ( x )) ; if U ∗ ( z ) = e z U^*(z)=e^z U ∗ ( z ) = e z , then u ( z ) = ln z u(z)=\ln z u ( z ) = ln z and ℓ ( y , x ∣ μ ˉ ) = μ ˉ − y ln [ μ ˉ ( x ) ] \ell(y,x\mid\bar{\mu})=\bar{\mu}-y\ln[\bar{\mu}(x)] ℓ ( y , x ∣ μ ˉ ) = μ ˉ − y ln [ μ ˉ ( x )] , so (2.2) is exactly the negative log-likelihood function in Poisson regression, see Section 6.2.2 of McCullagh and Nelder [21 ] .
By utilizing this empirical risk function L ^ ( μ ˉ ) \widehat{L}(\bar{\mu}) L ( μ ˉ ) , an estimator based on the data can be constructed as arg min μ ˉ D ^ U ( μ , μ ˉ ) = arg min μ ˉ L ^ ( μ ˉ ) \operatorname*{arg\,min}_{\bar{\mu}}\widehat{D}_U(\mu,\bar{\mu})=\operatorname*{arg\,min}_{\bar{\mu}}\widehat{L}(\bar{\mu}) arg min μ ˉ D U ( μ , μ ˉ ) = arg min μ ˉ L ( μ ˉ ) , that is, as an optimal function that minimizes the Bregman divergence D ^ U \widehat{D}_U D U from the target μ \mu μ .
Proposed semiparametric estimators
In this section, we use the empirical risk introduced in the previous section to construct semiparametric estimators through stagewise minimization. Before detailing the construction procedure, we first describe the functional form of the estimators and their motivation. We prepare a model function m ( ⋅ , θ ) : D → R m(\cdot,\boldsymbol{\theta}):\mathbb{D}\to\mathbb{R} m ( ⋅ , θ ) : D → R determined by a finite-dimensional parameter θ ∈ Θ ⊂ R p \boldsymbol{\theta}\in\Theta\subset\mathbb{R}^p θ ∈ Θ ⊂ R p , p ≥ 1 p\ge1 p ≥ 1 . We also prepare functions (called “words”) B i : R d → R B_i:\mathbb{R}^d\to\mathbb{R} B i : R d → R , i = 1 , … , N i=1,\ldots,N i = 1 , … , N , N ∈ N N\in\mathbb{N} N ∈ N and the set (called “dictionary”) of these functions D = { B i ∣ i = 1 , … , N } \mathcal{D}=\{B_i\mid i=1,\ldots,N\} D = { B i ∣ i = 1 , … , N } . Setting a parameter γ ∈ { 0 , 1 } \gamma\in\{0,1\} γ ∈ { 0 , 1 } , we define an operator T ( θ , B ) ( x ) = m ( x , θ ) + m ( x , θ ) γ ⋅ B ( x ) T(\boldsymbol{\theta},B)(x)=m(x,\boldsymbol{\theta})+m(x,\boldsymbol{\theta})^\gamma\cdot B(x) T ( θ , B ) ( x ) = m ( x , θ ) + m ( x , θ ) γ ⋅ B ( x ) for a function B B B , and also define a set T ( θ , D ) = { T ( θ , B ) ∣ B ∈ D } T(\boldsymbol{\theta},\mathcal{D})=\{T(\boldsymbol{\theta},B)\mid B\in\mathcal{D}\} T ( θ , D ) = { T ( θ , B ) ∣ B ∈ D } . For these m m m , D \mathcal{D} D and γ \gamma γ , we consider a class of functions having the form
{ u − 1 ( m ( ⋅ , θ ) + m ( ⋅ , θ ) γ ⋅ ξ ) | ξ = ∑ k = 1 N λ k B k , B k ∈ D , ∑ k = 1 N λ k = 1 , λ k ≥ 0 , k = 1 , … , N } = { u − 1 ( ∑ k = 1 N λ k T ( θ , B k ) ) | B k ∈ D , ∑ k = 1 N λ k = 1 , λ k ≥ 0 , k = 1 , … , N } = u − 1 ( co ( T ( θ , D ) ) ) , (3) \begin{aligned}
\left\{u^{-1}\left(m(\cdot,\boldsymbol{\theta})+m(\cdot,\boldsymbol{\theta})^\gamma\cdot\xi\right)\ \middle|\ \xi=\sum_{k=1}^{N}\lambda_kB_k,\ B_k\in\mathcal{D},\ \sum_{k=1}^{N}\lambda_k=1,\ \lambda_k\geq0,\ k=1,\ldots,N\right\} \\
&=\left\{u^{-1}\left(\sum_{k=1}^{N}\lambda_kT(\boldsymbol{\theta},B_k)\right)\ \middle|\ B_k\in\mathcal{D},\ \sum_{k=1}^{N}\lambda_k=1,\ \lambda_k\geq0,\ k=1,\ldots,N\right\} \\
&=u^{-1}\left(\operatorname{co}\left(T(\boldsymbol{\theta},\mathcal{D})\right)\right), \tag*{(3)}
\end{aligned} { u − 1 ( m ( ⋅ , θ ) + m ( ⋅ , θ ) γ ⋅ ξ ) ξ = k = 1 ∑ N λ k B k , B k ∈ D , k = 1 ∑ N λ k = 1 , λ k ≥ 0 , k = 1 , … , N } = { u − 1 ( k = 1 ∑ N λ k T ( θ , B k ) ) B k ∈ D , k = 1 ∑ N λ k = 1 , λ k ≥ 0 , k = 1 , … , N } = u − 1 ( co ( T ( θ , D ) ) ) , (3)
where u u u is the function u = U ′ u=U' u = U ′ associated with D U D_U D U in Section 2, and co ( T ( θ , D ) ) \operatorname{co}(T(\boldsymbol{\theta},\mathcal{D})) co ( T ( θ , D )) is the convex hull of the set T ( θ , D ) T(\boldsymbol{\theta},\mathcal{D}) T ( θ , D ) .
The functions (3.1) are based on a combination of parametric and nonparametric approaches. In the parametric approach, the target regression function μ \mu μ is modeled as a function determined by a finite-dimensional parameter: { u − 1 ( m ( ⋅ , θ ) ) ∣ θ ∈ Θ } \{u^{-1}(m(\cdot,\theta))\mid\theta\in\Theta\} { u − 1 ( m ( ⋅ , θ )) ∣ θ ∈ Θ } . By plugging in an estimator θ ^ \hat{\theta} θ ^ of the parameter θ \theta θ , we obtain the parametric regression estimator u − 1 ( m ( ⋅ , θ ^ ) ) u^{-1}(m(\cdot,\hat{\theta})) u − 1 ( m ( ⋅ , θ ^ )) . Typical examples include generalized linear models (GLMs). In GLMs, the hyperplane is used as m ( x , θ ) = [ 1 x T ] θ m(\mathbf{x},\boldsymbol{\theta})=[1\ \mathbf{x}^{\mathsf{T}}]\boldsymbol{\theta} m ( x , θ ) = [ 1 x T ] θ , θ ∈ R d + 1 \boldsymbol{\theta}\in\mathbb{R}^{d+1} θ ∈ R d + 1 . The function u − 1 u^{-1} u − 1 serves as a known inverse link function. Moreover, θ ^ \hat{\boldsymbol{\theta}} θ ^ may be taken as, for instance, the MLE ([21 ] ) or the LASSO estimator ([14 ] ), both of which will be formulated below. This u − 1 ( m ( ⋅ , θ ^ ) ) u^{-1}(m(\cdot,\hat{\boldsymbol{\theta}})) u − 1 ( m ( ⋅ , θ ^ )) provides interpretation of the underlying population regression function μ \mu μ . For example, in GLMs, the parameter θ \boldsymbol{\theta} θ corresponds to the coefficients of hyperplanes, allowing us to get information: whether μ \mu μ increases, decreases, or hardly changes globally with some explanatory variables. On the other hand, such models with a finite-dimensional parameter are sometimes too restrictive. It has been understood that parametric estimators generally lack consistency, and this approach is justified only when the true μ \mu μ is exactly within the model { u − 1 ( m ( ⋅ , θ ) ) ∣ θ ∈ Θ } \{u^{-1}(m(\cdot,\theta))\mid\theta\in\Theta\} { u − 1 ( m ( ⋅ , θ )) ∣ θ ∈ Θ } or at least in its neighborhood.
To retain the interpretability of the parametric approach while improving flexibility, we introduce a nonparametric adjustment to the parametric crude guess. Unlike the purely parametric model u − 1 ( m ( ⋅ , θ ) ) u^{-1}(m(\cdot,\theta)) u − 1 ( m ( ⋅ , θ )) , the function in (3) includes an adjustment term ξ \xi ξ . Here γ \gamma γ in (3) controls the type of adjustment: γ = 0 \gamma=0 γ = 0 yields additive adjustment u − 1 ( m ( ⋅ , θ ) + ξ ) u^{-1}(m(\cdot,\theta)+\xi) u − 1 ( m ( ⋅ , θ ) + ξ ) , while γ = 1 \gamma=1 γ = 1 leads to multiplicative adjustment as u − 1 ( m ( ⋅ , θ ) ⋅ ( 1 + ξ ) ) u^{-1}(m(\cdot,\theta)\cdot(1+\xi)) u − 1 ( m ( ⋅ , θ ) ⋅ ( 1 + ξ )) , see [11 ] . Below we construct this ξ \xi ξ as a convex combination of I I I words B 1 , … , B I B_1,\ldots,B_I B 1 , … , B I selected from the dictionary D \mathcal{D} D : ξ ^ = ∑ i = 1 I λ i B i \hat{\xi}=\sum_{i=1}^{I}\lambda_iB_i ξ ^ = ∑ i = 1 I λ i B i , in which the same words may be selected repeatedly. These words are selected by minimizing the empirical risk (2.2) in a stagewise manner.
Now we present the construction of our semiparametric estimators. Let
C ( π ˉ ∣ g , h ) ( x ) = C ( π ˉ ∣ g ( x ) , h ( x ) ) = π ˉ ⋅ g ( x ) + ( 1 − π ˉ ) ⋅ h ( x ) C(\bar{\pi}\mid g,h)(x)=C(\bar{\pi}\mid g(x),h(x))=\bar{\pi}\cdot g(x)+(1-\bar{\pi})\cdot h(x) C ( π ˉ ∣ g , h ) ( x ) = C ( π ˉ ∣ g ( x ) , h ( x )) = π ˉ ⋅ g ( x ) + ( 1 − π ˉ ) ⋅ h ( x )
for functions g : R d → R g:\mathbb{R}^{d}\to\mathbb{R} g : R d → R and h : R d → R h:\mathbb{R}^{d}\to\mathbb{R} h : R d → R and a coefficient π ˉ ∈ [ 0 , 1 ] \bar{\pi}\in[0,1] π ˉ ∈ [ 0 , 1 ] . We introduce the notation argmin B ∈ D ε g ( B ) = { B ∈ D ∣ g ( B ) ≤ inf B ∈ D g ( B ) + ε } \operatorname*{argmin}_{B\in\mathcal{D}}^{\varepsilon}g(B)=\{B\in\mathcal{D}\mid g(B)\leq\inf_{B\in\mathcal{D}}g(B)+\varepsilon\} argmin B ∈ D ε g ( B ) = { B ∈ D ∣ g ( B ) ≤ inf B ∈ D g ( B ) + ε } for a function g : D → R g:\mathcal{D}\to\mathbb{R} g : D → R and a positive real number ε > 0 \varepsilon>0 ε > 0 . In addition, we set ξ 0 = 0 \xi_0=0 ξ 0 = 0 , π k ∈ [ 0 , 1 ] \pi_k\in[0,1] π k ∈ [ 0 , 1 ] (k ∈ N k\in\mathbb{N} k ∈ N ) and fix a sufficiently small ε > 0 \varepsilon>0 ε > 0 . The semiparametric estimators in this paper are constructed as follows ([18 ] , Naito and Eguchi [24 ] , Yoshida and Naito [33 ] , Nishida and Naito [26 ] :
⟨ S t e p 1 ⟩ \langle\mathit{Step\ 1}\rangle ⟨ Step 1 ⟩ Obtain a parametric regression estimator u − 1 ( m ( ⋅ , θ ˉ ) ) u^{-1}(m(\cdot,\bar{\theta})) u − 1 ( m ( ⋅ , θ ˉ )) with an estimator θ ˉ \bar{\theta} θ ˉ for the parameter θ \theta θ in the model function.
⟨ S t e p 2 ⟩ \langle\mathit{Step\ 2}\rangle ⟨ Step 2 ⟩ Prepare a dictionary D \mathcal{D} D . Then select B 1 , … , B I B_1,\ldots,B_I B 1 , … , B I and obtain ξ I \xi_I ξ I :
B k ∈ argmin B ˉ ∈ D L ^ ( u − 1 ( T ( θ ˉ , C ( π k ∣ B ˉ , ξ k − 1 ) ) ) ) , B_k \in\operatorname*{argmin}_{\bar{B}\in\mathcal{D}} \widehat{L}\left(u^{-1}\left(T\left(\bar{\theta},C\left(\pi_k\mid\bar{B},\xi_{k-1}\right)\right)\right)\right), B k ∈ B ˉ ∈ D argmin L ( u − 1 ( T ( θ ˉ , C ( π k ∣ B ˉ , ξ k − 1 ) ) ) ) ,
ξ k = C ( π k ∣ B k , ξ k − 1 ) \xi_k=C\left(\pi_k\mid B_k,\xi_{k-1}\right) ξ k = C ( π k ∣ B k , ξ k − 1 )
for each k = 1 , … , I k=1,\ldots,I k = 1 , … , I .
⟨ S t e p 3 ⟩ \langle\mathit{Step\ 3}\rangle ⟨ Step 3 ⟩ Construct the semiparametric regression estimator μ ^ = μ I ( θ ˉ ) = u − 1 ( T ( θ ˉ , ξ I ) ) \widehat{\mu}=\mu_I(\bar{\theta})=u^{-1}\left(T\left(\bar{\theta},\xi_I\right)\right) μ = μ I ( θ ˉ ) = u − 1 ( T ( θ ˉ , ξ I ) ) .
The above ⟨ S t e p 1 ⟩ \langle\mathit{Step\ 1}\rangle ⟨ Step 1 ⟩ is the initial parametric regression. The estimator θ ˉ \bar{\theta} θ ˉ is, for example, the MLE argmin θ ∈ Θ L ^ ( u − 1 ( m ( ⋅ , θ ) ) ) \operatorname*{argmin}_{\theta\in\Theta}\widehat{L}\left(u^{-1}(m(\cdot,\theta))\right) argmin θ ∈ Θ L ( u − 1 ( m ( ⋅ , θ )) ) , or the LASSO estimator
argmin θ ∈ Θ [ L ^ ( u − 1 ( m ( ⋅ , θ ) ) ) + λ ∥ θ ∥ 1 ] \operatorname*{argmin}_{\theta\in\Theta}\left[\widehat{L}\left(u^{-1}(m(\cdot,\theta))\right)+\lambda\lVert\theta\rVert_1\right] θ ∈ Θ argmin [ L ( u − 1 ( m ( ⋅ , θ )) ) + λ ∥ θ ∥ 1 ]
for the ℓ 1 \ell_1 ℓ 1 -norm ∥ ⋅ ∥ 1 \lVert\cdot\rVert_1 ∥ ⋅ ∥ 1 and some λ ≥ 0 \lambda\geq0 λ ≥ 0 (and for appropriately transformed data), or others that are not derived from the Bregman divergence D U D_U D U . In ⟨ S t e p 2 ⟩ \langle\mathit{Step\ 2}\rangle ⟨ Step 2 ⟩ , the adjustment term ξ \xi ξ is introduced through the stagewise minimization of the empirical risk L ^ \widehat{L} L in (2) . Finally, the proposed estimator μ ^ \widehat{\mu} μ has the form
μ ^ = u − 1 ( T ( θ ˉ , ξ ^ ) ) , ξ ^ = ξ I = ∑ k = 1 I λ k , I B k , λ k , I = π k ∏ j = k + 1 I ( 1 − π j ) , \widehat{\mu}=u^{-1}\left(T\left(\bar{\theta},\widehat{\xi}\right)\right),\qquad\widehat{\xi}=\xi_I=\sum_{k=1}^{I}\lambda_{k,I}B_k,\qquad\lambda_{k,I}=\pi_k\prod_{j=k+1}^{I}(1-\pi_j), μ = u − 1 ( T ( θ ˉ , ξ ) ) , ξ = ξ I = k = 1 ∑ I λ k , I B k , λ k , I = π k j = k + 1 ∏ I ( 1 − π j ) ,
where λ k , I ≥ 0 \lambda_{k,I}\geq0 λ k , I ≥ 0 , k = 1 , … , I k=1,\ldots,I k = 1 , … , I and ∑ k = 1 I λ k , I = 1 \sum_{k=1}^{I}\lambda_{k,I}=1 ∑ k = 1 I λ k , I = 1 are satisfied.
Theoretical results
This section provides theoretical results for the proposed estimator μ ^ \widehat{\mu} μ in § 3 . We evaluate estimation accuracy of μ ^ \widehat{\mu} μ by using a measure called “divergence risk.” We derive a non-asymptotic upper bound for this risk, and further investigate this bound asymptotically. We also compare this non-asymptotic bound for μ ^ \widehat{\mu} μ with that for a theoretically ideal estimator.
In what follows, we fix a strictly convex function U U U and the link function u = U ′ u=U' u = U ′ for μ ^ \hat{\mu} μ ^ , and put the sequence π k \pi_k π k forming the coefficients of ξ ^ \hat{\xi} ξ ^ as
π k = τ k − 1 + τ \pi_k=\frac{\tau}{k-1+\tau} π k = k − 1 + τ τ
for some τ ≥ 2 \tau\ge2 τ ≥ 2 and for each k ∈ N k \in\mathbb{N} k ∈ N . This π k \pi_k π k is similar to those in Klemelä [18 ] , Naito and Eguchi [24 ] , Nishida and Naito [26 ] and Yoshida and Naito [33 ] . Let ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 be the ℓ 2 \ell_2 ℓ 2 -norm, ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ be the L ∞ L_\infty L ∞ -norm,
W = { [ λ B ] B ∈ D ∣ λ B ≥ 0 , ∑ B ∈ D λ B = 1 } , N ∈ N \mathcal{W}=\left\{[\lambda_B]_{B\in\mathcal{D}}\mid\lambda_B\ge0,\ \sum_{B\in\mathcal{D}}\lambda_B=1\right\},\qquad N\in\mathbb{N} W = { [ λ B ] B ∈ D ∣ λ B ≥ 0 , B ∈ D ∑ λ B = 1 } , N ∈ N
and
μ ( x , Λ , θ ) = u − 1 ( T ( θ , ∑ B ∈ D λ B B ) ( x ) ) , x ∈ R d , Λ = [ λ B ] B ∈ D ∈ W , θ ∈ R p . \mu(x,\Lambda,\theta)=u^{-1}\left(T\left(\theta,\sum_{B\in\mathcal{D}}\lambda_B B\right)(x)\right),\qquad x\in\mathbb{R}^d,\quad\Lambda=[\lambda_B]_{B\in\mathcal{D}}\in\mathcal{W},\quad\theta\in\mathbb{R}^p. μ ( x , Λ , θ ) = u − 1 ( T ( θ , B ∈ D ∑ λ B B ) ( x ) ) , x ∈ R d , Λ = [ λ B ] B ∈ D ∈ W , θ ∈ R p .
We make the following assumptions:
(A1) The density function q q q for the distribution of the explanatory variables X X X has the compact support D ⊂ R d \mathbb{D}\subset\mathbb{R}^d D ⊂ R d .
(A2) The response Y Y Y satisfies E [ Y 2 ] < ∞ \mathbb{E}\left[Y^2\right]<\infty E [ Y 2 ] < ∞ .
(A3) The estimator θ ^ \hat{\theta} θ ^ for the parameter θ \theta θ satisfies θ ^ → P θ ∗ \hat{\theta}\xrightarrow{P}\theta^* θ ^ P θ ∗ as n → ∞ n\to\infty n → ∞ .
(A4) The function x ↦ m ( x , θ ) x\mapsto m(x,\theta) x ↦ m ( x , θ ) is continuous for any θ ∈ Θ \theta\in\Theta θ ∈ Θ . In addition, the function θ ↦ m ( x , θ ) \theta\mapsto m(x,\theta) θ ↦ m ( x , θ ) is continuously differentiable for any x ∈ D x\in\mathbb{D} x ∈ D .
(A5) The strictly convex function U U U is twice continuously differentiable.
We define the divergence risk as the function μ ˉ ↦ E [ D U ( μ , μ ˉ ) ] \bar{\mu}\mapsto\mathbb{E}[D_U(\mu,\bar{\mu})] μ ˉ ↦ E [ D U ( μ , μ ˉ )] for the target regression function μ \mu μ and an estimator μ ˉ \bar{\mu} μ ˉ , where D U D_U D U is the Bregman divergence associated with U U U and the expectation E \mathbb{E} E is taken with respect to the observations X \mathsf{X} X constructing μ ˉ \bar{\mu} μ ˉ . If E [ D U ( μ , μ ˉ ) ] \mathbb{E}[D_U(\mu,\bar{\mu})] E [ D U ( μ , μ ˉ )] is smaller, it means that the estimator μ ˉ \bar{\mu} μ ˉ has higher accuracy. In the statements below, words in the dictionary D \mathcal{D} D are allowed to depend on the observations X \mathsf{X} X . The divergence risk for μ ^ \hat{\mu} μ ^ is evaluated in Theorem 4.1 below, on the basis of Lemmas 4.1 and 4.2 establishing the relevant inequalities.
Lemma 4.1. For fixed I ∈ N I \in\mathbb{N} I ∈ N and for fixed θ 1 , θ 2 ∈ R p \theta_1,\theta_2 \in\mathbb{R}^p θ 1 , θ 2 ∈ R p which may depend on the observations X X X , the function μ I ( θ 1 ) \mu_I(\theta_1) μ I ( θ 1 ) defined in ⟨ S t e p 3 ⟩ \langle Step\ 3\rangle ⟨ S t e p 3 ⟩ of the algorithm in Section 3 satisfies the following inequality of random variables:
L ^ ( μ I ( θ 1 ) ) ≤ inf Λ ∈ W L ^ ( μ ( ⋅ , Λ , θ 2 ) ) + τ 2 I + τ A 1 ( θ 1 , θ 2 ) \widehat{L}(\mu_I(\theta_1)) \le\inf_{\Lambda\in\mathcal{W}} \widehat{L}(\mu(\cdot,\Lambda,\theta_2))+\frac{\tau^2}{I+\tau}A_1(\theta_1,\theta_2) L ( μ I ( θ 1 )) ≤ Λ ∈ W inf L ( μ ( ⋅ , Λ , θ 2 )) + I + τ τ 2 A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + τ 2 I + τ ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) + ε , \quad+\|\theta_1-\theta_2\|_2 A_2(\theta_1,\theta_2)+\frac{\tau^2}{I+\tau}\|\theta_1-\theta_2\|_2 A_3(\theta_1,\theta_2)+\varepsilon, + ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + I + τ τ 2 ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) + ε , where
A 1 ( θ 1 , θ 2 ) = 4 sup B ∈ D sup x ∈ D ∣ B ( x ) ∣ 2 sup x ∈ D ∣ m ( x , θ 1 ) m ( x , θ 2 ) ∣ γ A_1(\theta_1,\theta_2)=4\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}|B(x)|^2\sup_{x\in\mathcal{D}}|m(x,\theta_1)m(x,\theta_2)|^\gamma A 1 ( θ 1 , θ 2 ) = 4 B ∈ D sup x ∈ D sup ∣ B ( x ) ∣ 2 x ∈ D sup ∣ m ( x , θ 1 ) m ( x , θ 2 ) ∣ γ × sup Λ ∈ W sup x ∈ D ( U ∗ ) ′ ′ ( u ( μ ( x , Λ , θ 1 ) ) ) , \quad\times\sup_{\Lambda\in\mathcal{W}}\sup_{x\in\mathcal{D}}(U^*)''\bigl(u(\mu(x,\Lambda,\theta_1))\bigr), × Λ ∈ W sup x ∈ D sup ( U ∗ ) ′′ ( u ( μ ( x , Λ , θ 1 )) ) , A 2 ( θ 1 , θ 2 ) = sup B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] ∥ { 1 + γ m ( x , θ ) } γ − 1 B ( x ) ∂ ∂ θ m ( x , θ ) ∣ θ = θ 2 + β ( θ 1 − θ 2 ) ∥ 2 A_2(\theta_1,\theta_2)=\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}\sup_{\beta\in[0,1]}
\left\|\left\{1+\gamma m(x,\theta)\right\}\gamma^{-1}B(x)\frac{\partial}{\partial\theta}m(x,\theta)\bigg|_{\theta=\theta_2+\beta(\theta_1-\theta_2)}\right\|_2 A 2 ( θ 1 , θ 2 ) = B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] sup { 1 + γ m ( x , θ ) } γ − 1 B ( x ) ∂ θ ∂ m ( x , θ ) θ = θ 2 + β ( θ 1 − θ 2 ) 2 × 1 n ∑ i = 1 n sup B ∈ D ∣ Y i − u − 1 ( T ( θ 1 , B ) ( X i ) ) ∣ , \quad\times\frac{1}{n}\sum_{i=1}^{n}\sup_{B\in\mathcal{D}}\left|Y_i-u^{-1}\bigl(T(\theta_1,B)(X_i)\bigr)\right|, × n 1 i = 1 ∑ n B ∈ D sup Y i − u − 1 ( T ( θ 1 , B ) ( X i ) ) , A 3 ( θ 1 , θ 2 ) = 2 sup B ∈ D sup x ∈ D ∣ B ( x ) ∣ sup x ∈ D ∣ m ( x , θ 1 ) ∣ γ sup Λ ∈ W sup x ∈ D ( U ∗ ) ′ ′ ( u ( μ ( x , Λ , θ 1 ) ) ) A_3(\theta_1,\theta_2)=2\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}|B(x)|\sup_{x\in\mathcal{D}}|m(x,\theta_1)|^\gamma\sup_{\Lambda\in\mathcal{W}}\sup_{x\in\mathcal{D}}(U^*)''\bigl(u(\mu(x,\Lambda,\theta_1))\bigr) A 3 ( θ 1 , θ 2 ) = 2 B ∈ D sup x ∈ D sup ∣ B ( x ) ∣ x ∈ D sup ∣ m ( x , θ 1 ) ∣ γ Λ ∈ W sup x ∈ D sup ( U ∗ ) ′′ ( u ( μ ( x , Λ , θ 1 )) ) × sup B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] ∥ { 1 + γ m ( x , θ ) } γ − 1 B ( x ) ∂ ∂ θ m ( x , θ ) ∣ θ = θ 2 + β ( θ 1 − θ 2 ) ∥ 2 . \quad\times\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}\sup_{\beta\in[0,1]}
\left\|\left\{1+\gamma m(x,\theta)\right\}\gamma^{-1}B(x)\frac{\partial}{\partial\theta}m(x,\theta)\bigg|_{\theta=\theta_2+\beta(\theta_1-\theta_2)}\right\|_2. × B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] sup { 1 + γ m ( x , θ ) } γ − 1 B ( x ) ∂ θ ∂ m ( x , θ ) θ = θ 2 + β ( θ 1 − θ 2 ) 2 .
Lemma 4.2. For fixed θ 1 ( X ) , θ 2 ( X ) ∈ R p \theta_1(X),\theta_2(X)\in\mathbb{R}^p θ 1 ( X ) , θ 2 ( X ) ∈ R p and δ ( X ) ≥ 0 \delta(X)\ge0 δ ( X ) ≥ 0 , let Λ ˉ ( X ) ∈ W \bar{\Lambda}(X)\in\mathcal{W} Λ ˉ ( X ) ∈ W be such that
L ^ ( μ ( ⋅ , Λ ˉ ( X ) , θ 1 ( X ) ) ) ≤ inf Λ ∈ W L ^ ( μ ( ⋅ , Λ , θ 2 ( X ) ) ) + δ ( X ) . (4) \widehat{L}\bigl(\mu(\cdot,\bar{\Lambda}(X),\theta_1(X))\bigr)\le\inf_{\Lambda\in\mathcal{W}}\widehat{L}\bigl(\mu(\cdot,\Lambda,\theta_2(X))\bigr)+\delta(X). \tag*{(4)} L ( μ ( ⋅ , Λ ˉ ( X ) , θ 1 ( X )) ) ≤ Λ ∈ W inf L ( μ ( ⋅ , Λ , θ 2 ( X )) ) + δ ( X ) . (4) Then
E [ D U ( μ , μ ( ⋅ , Λ ˉ ( X ) , θ 1 ( X ) ) ) ] \mathbb{E}\left[D_U\bigl(\mu,\mu(\cdot,\bar{\Lambda}(X),\theta_1(X))\bigr)\right] E [ D U ( μ , μ ( ⋅ , Λ ˉ ( X ) , θ 1 ( X )) ) ] ≤ E [ inf Λ ∈ W D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) ] + E [ sup Λ ∈ W E n ( μ ( ⋅ , Λ , θ 2 ( X ) ) ) ] + E [ sup Λ ∈ W [ − E n ( μ ( ⋅ , Λ , θ 1 ( X ) ) ) ] ] + E [ δ ( X ) ] , \le\mathbb{E}\left[\inf_{\Lambda\in\mathcal{W}}D_U\bigl(\mu,\mu(\cdot,\Lambda,\theta_2(X))\bigr)\right]
+\mathbb{E}\left[\sup_{\Lambda\in\mathcal{W}}\mathcal{E}_n\bigl(\mu(\cdot,\Lambda,\theta_2(X))\bigr)\right]
+\mathbb{E}\left[\sup_{\Lambda\in\mathcal{W}}\left[-\mathcal{E}_n\bigl(\mu(\cdot,\Lambda,\theta_1(X))\bigr)\right]\right]
+\mathbb{E}\left[\delta(X)\right], ≤ E [ Λ ∈ W inf D U ( μ , μ ( ⋅ , Λ , θ 2 ( X )) ) ] + E [ Λ ∈ W sup E n ( μ ( ⋅ , Λ , θ 2 ( X )) ) ] + E [ Λ ∈ W sup [ − E n ( μ ( ⋅ , Λ , θ 1 ( X )) ) ] ] + E [ δ ( X ) ] ,
where E n ( h ) \mathcal{E}_n(h) E n ( h ) is a centered empirical process
E n ( h ) = 1 n ∑ i = 1 n ℓ ( Y i , X i ∣ h ) − ∫ R × R d ℓ ( y , x ∣ h ) f ( y , x ) d y d x \mathcal{E}_n(h)=\frac{1}{n}\sum_{i=1}^{n}\ell(Y_i,X_i\mid h)-\int_{\mathbb{R}\times\mathbb{R}^{d}}\ell(y,x\mid h)f(y,x)\,\mathrm{d}y\mathrm{d}x E n ( h ) = n 1 i = 1 ∑ n ℓ ( Y i , X i ∣ h ) − ∫ R × R d ℓ ( y , x ∣ h ) f ( y , x ) d y d x
associated with a function h : R d → R h:\mathbb{R}^{d}\to\mathbb{R} h : R d → R .
We now define a dictionary denoted by D ( Δ 1 , Δ 2 , c ) \mathcal{D}(\Delta_1,\Delta_2,c) D ( Δ 1 , Δ 2 , c ) with some fixed Δ 1 , Δ 2 , c > 0 \Delta_1,\Delta_2,c>0 Δ 1 , Δ 2 , c > 0 , for the theoretical results below. We first construct its words as follows. Let the realization of the observation X \mathbf{X} X be X = { ( y 1 , x 1 ) , … , ( y n , x n ) } \mathcal{X}=\{(y_1,x_1),\ldots,(y_n,x_n)\} X = {( y 1 , x 1 ) , … , ( y n , x n )} . The words should be allowed to include the estimator θ ^ ( X ) \hat{\theta}(\mathcal{X}) θ ^ ( X ) for the parameter in the parametric model, and also to depend on the observations X \mathcal{X} X themselves. Further, we introduce an index set P ⊂ R p \mathcal{P}\subset\mathbb{R}^{p} P ⊂ R p , p ≥ 1 p\geq1 p ≥ 1 , which will determine the words in the dictionary. For example, the elements of P \mathcal{P} P may be a specific i i i -th observation ( y i , x i ) (y_i,x_i) ( y i , x i ) or a bandwidth H H H or their combination ( ( y i , x i ) , H ) ((y_i,x_i),H) (( y i , x i ) , H ) . The general form of the words is a function B ( ⋅ ∣ θ , Z , P ) : R d → R B(\cdot\mid\theta,\mathcal{Z},\mathcal{P}):\mathbb{R}^{d}\to\mathbb{R} B ( ⋅ ∣ θ , Z , P ) : R d → R , θ ∈ Θ \theta\in\Theta θ ∈ Θ , Z = { ( v 1 , w 1 ) , … , ( v n , w n ) } \mathcal{Z}=\{(v_1,w_1),\ldots,(v_n,w_n)\} Z = {( v 1 , w 1 ) , … , ( v n , w n )} with ( v i , w i ) ∈ R × R d (v_i,w_i)\in\mathbb{R}\times\mathbb{R}^{d} ( v i , w i ) ∈ R × R d and P ∈ P \mathcal{P}\in\mathcal{P} P ∈ P . Then, preparing a finite set P ∗ ⊂ P \mathcal{P}^{*}\subset\mathcal{P} P ∗ ⊂ P with ∣ P ∗ ∣ = N |\mathcal{P}^{*}|=N ∣ P ∗ ∣ = N , we define
D = { B ( ⋅ ∣ θ ^ ( X ) , X , P ) ∣ P ∈ P ∗ } , \mathcal{D}=\left\{B\left(\cdot\mid\hat{\theta}(\mathcal{X}),\mathcal{X},\mathcal{P}\right)\mid\mathcal{P}\in\mathcal{P}^{*}\right\}, D = { B ( ⋅ ∣ θ ^ ( X ) , X , P ) ∣ P ∈ P ∗ } ,
where N ∈ N N\in\mathbb{N} N ∈ N may depend on the sample size n n n . An example of this dictionary will appear in Section 5.3. The dictionary utilized in the theorems below is a subset of this D \mathcal{D} D . The practical dictionary is defined as
D ( Δ 1 , Δ 2 , c ) = { B ( ⋅ ∣ θ ^ ( X ) , X , P ) ∈ D | sup x ∈ R d ∣ T ( θ ^ ( X ) , B ( x ∣ θ ^ ( X ) , X , P ) ) ∣ ≤ Δ 1 , sup x ∈ R d sup θ ∈ Θ c ∣ B ( ⋅ ∣ θ , X , P ) ∣ ≤ Δ 2 } , \mathcal{D}(\Delta_1,\Delta_2,c)=\left\{B\left(\cdot\mid\hat{\theta}(\mathcal{X}),\mathcal{X},\mathcal{P}\right)\in\mathcal{D}\ \middle|\ \sup_{x\in\mathbb{R}^{d}}\left|T\left(\hat{\theta}(\mathcal{X}),B\left(x\mid\hat{\theta}(\mathcal{X}),\mathcal{X},\mathcal{P}\right)\right)\right|\leq\Delta_1,\quad\sup_{x\in\mathbb{R}^{d}}\sup_{\theta\in\Theta_c}\left|B(\cdot\mid\theta,\mathcal{X},\mathcal{P})\right|\leq\Delta_2\right\}, D ( Δ 1 , Δ 2 , c ) = { B ( ⋅ ∣ θ ^ ( X ) , X , P ) ∈ D x ∈ R d sup T ( θ ^ ( X ) , B ( x ∣ θ ^ ( X ) , X , P ) ) ≤ Δ 1 , x ∈ R d sup θ ∈ Θ c sup ∣ B ( ⋅ ∣ θ , X , P ) ∣ ≤ Δ 2 } ,
for some fixed Δ 1 , Δ 2 , c > 0 \Delta_1,\Delta_2,c>0 Δ 1 , Δ 2 , c > 0 , where Θ c = { θ ∈ Θ ∣ ∥ θ − θ ∗ ∥ 2 ≤ c } \Theta_c=\{\theta\in\Theta\mid\|\theta-\theta^{*}\|_2\leq c\} Θ c = { θ ∈ Θ ∣ ∥ θ − θ ∗ ∥ 2 ≤ c } .
We assume that D ( Δ 1 , Δ 2 , c ) \mathcal{D}(\Delta_1,\Delta_2,c) D ( Δ 1 , Δ 2 , c ) satisfies the following assumption:
(A6) There exists a set B \mathcal{B} B such that D ( Δ 1 , Δ 2 , c ) ⊂ B \mathcal{D}(\Delta_1,\Delta_2,c)\subset\mathcal{B} D ( Δ 1 , Δ 2 , c ) ⊂ B and its covering number N ( ϵ , B , ∥ ⋅ ∥ ∞ ) N(\epsilon,\mathcal{B},\|\cdot\|_{\infty}) N ( ϵ , B , ∥ ⋅ ∥ ∞ ) satisfies ∫ 0 1 ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ < ∞ \int_0^1\sqrt{\ln N(\epsilon,\mathcal{B},\|\cdot\|_{\infty})}\,\mathrm{d}\epsilon<\infty ∫ 0 1 ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ < ∞ .
Remark. The definition of the covering number can be found in Definition 5.1 and Section 5.3.3 of Wainwright [30 ] , and Definition 2.1.5 of van der Vaart and Wellner [29 ] . Examples of sets B \mathcal{B} B satisfying ∫ 0 1 ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ < ∞ \int_0^1\sqrt{\ln N(\epsilon,\mathcal{B},\|\cdot\|_{\infty})}\,\mathrm{d}\epsilon<\infty ∫ 0 1 ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ < ∞ can also be found in Section 2.7 of van der Vaart and Wellner [29 ] .
For the proposed estimator μ ^ \hat{\mu} μ ^ constructed from such a dictionary, we can evaluate the divergence risk E [ D U ( μ , μ ^ ) ] \mathbb{E}[D_U(\mu,\hat{\mu})] E [ D U ( μ , μ ^ )] as follows:
Theorem 4.1. For ε > 0 \varepsilon> 0 ε > 0 in the algorithm, the stagewise estimator μ ^ = μ I ( θ ^ ) \hat{\mu}=\mu_I(\hat{\theta}) μ ^ = μ I ( θ ^ ) satisfies
E [ D U ( μ , μ ^ ) ] ≤ E [ inf g ∈ u − 1 ( co ( T ( θ ^ , D ( Δ 1 , Δ 2 , c ) ) ) ) D U ( μ , g ) ] + ∑ α = 0 1 E [ sup g ∈ u − 1 ( co ( T ( θ ^ , D ( Δ 1 , Δ 2 , c ) ) ) ) [ ( − 1 ) α E n ( g ) ] ] + τ 2 I + τ C 1 ( n ) + ε , (5) \begin{aligned}
\mathbb{E}[D_U(\mu,\hat{\mu})] &\leq\mathbb{E}\left[\inf_{g\in u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D}(\Delta_1,\Delta_2,c))))}D_U(\mu,g)\right] \\
&\quad+\sum_{\alpha=0}^{1}\mathbb{E}\left[\sup_{g\in u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D}(\Delta_1,\Delta_2,c))))}[(-1)^\alpha\mathcal{E}_n(g)]\right]+\frac{\tau^2}{I+\tau}C_1(n)+\varepsilon, \tag*{(5)}
\end{aligned} E [ D U ( μ , μ ^ )] ≤ E [ g ∈ u − 1 ( co ( T ( θ ^ , D ( Δ 1 , Δ 2 , c )))) inf D U ( μ , g ) ] + α = 0 ∑ 1 E [ g ∈ u − 1 ( co ( T ( θ ^ , D ( Δ 1 , Δ 2 , c )))) sup [( − 1 ) α E n ( g )] ] + I + τ τ 2 C 1 ( n ) + ε , (5) where C 1 ( n ) = E [ A 1 ( θ ^ , θ ^ ) ] C_1(n)=\mathbb{E}\left[A_1(\hat{\theta},\hat{\theta})\right] C 1 ( n ) = E [ A 1 ( θ ^ , θ ^ ) ] . Moreover,
∑ α = 0 1 E [ sup g ∈ u − 1 ( co ( T ( θ ^ , D ( Δ 1 , Δ 2 , c ) ) ) ) [ ( − 1 ) α E n ( g ) ] ] = O ( 1 n ) . \sum_{\alpha=0}^{1}\mathbb{E}\left[\sup_{g\in u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D}(\Delta_1,\Delta_2,c))))}[(-1)^\alpha\mathcal{E}_n(g)]\right]=O\left(\frac{1}{\sqrt{n}}\right). α = 0 ∑ 1 E [ g ∈ u − 1 ( co ( T ( θ ^ , D ( Δ 1 , Δ 2 , c )))) sup [( − 1 ) α E n ( g )] ] = O ( n 1 ) .
We focus on the upper bound (4.2) in Theorem 4.1: the first term in the bound represents the essential bias in approximating μ \mu μ with the function class u − 1 ( co ( T ( θ ^ , D ( Δ 1 , Δ 2 , c ) ) ) ) u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D}(\Delta_1,\Delta_2,c)))) u − 1 ( co ( T ( θ ^ , D ( Δ 1 , Δ 2 , c )))) , to which the proposed estimator μ ^ \hat{\mu} μ ^ belongs. The second term approaches 0 as n → ∞ n\to\infty n → ∞ , and the third term also converges to 0 as I → ∞ I\to\infty I → ∞ . In short, given that we can take sufficiently small ε \varepsilon ε , Theorem 4.1 states that the divergence risk for μ ^ \hat{\mu} μ ^ can be bounded above only by the essential bias term and asymptotically vanishing terms.
This Theorem 4.1 can be better understood with Theorem 4.2 below. Theorem 4.2 provides an error bound for an ideal estimator μ 0 \mu^0 μ 0 :
Theorem 4.2. For ε > 0 \varepsilon> 0 ε > 0 and for any of the following estimators
μ 0 ∈ argmin g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) ε L ^ ( g ) , (6) \mu^0\in\operatorname*{argmin}_{g\in u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D})))}^{\varepsilon}\hat{L}(g), \tag*{(6)} μ 0 ∈ g ∈ u − 1 ( co ( T ( θ ^ , D ))) argmin ε L ^ ( g ) , (6) it holds that
E [ D U ( μ , μ 0 ) ] ≤ E [ inf g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) D U ( μ , g ) ] + ∑ α = 0 1 E [ sup g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) [ ( − 1 ) α E n ( g ) ] ] + ε . (7) \mathbb{E}[D_U(\mu,\mu^0)]
\leq\mathbb{E}\left[\inf_{g\in u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D})))}D_U(\mu,g)\right]
+\sum_{\alpha=0}^{1}\mathbb{E}\left[\sup_{g\in u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D})))}[(-1)^\alpha\mathcal{E}_n(g)]\right]+\varepsilon. \tag*{(7)} E [ D U ( μ , μ 0 )] ≤ E [ g ∈ u − 1 ( co ( T ( θ ^ , D ))) inf D U ( μ , g ) ] + α = 0 ∑ 1 E [ g ∈ u − 1 ( co ( T ( θ ^ , D ))) sup [( − 1 ) α E n ( g )] ] + ε . (7)
Comparing the upper bounds in (5) and (7) , (5) differs from (7) only in that it includes the term τ 2 ( I + τ ) ⋅ C 1 ( n ) \frac{\tau^2}{(I+\tau)} \cdot C_1(n) ( I + τ ) τ 2 ⋅ C 1 ( n ) , which converges to 0 as I → ∞ I \to\infty I → ∞ . Therefore, the bound for the practically implemented μ ^ \widehat{\mu} μ asymptotically coincides with that for μ 0 \mu^0 μ 0 in (6) , which theoretically minimizes the empirical risk over the convex hull.
Implementation
In this section, we present practical implementation of the proposed estimator μ ^ \widehat{\mu} μ introduced in Section 3. In particular, we propose several dictionaries for constructing the adjustment term ξ ^ \widehat{\xi} ξ in μ ^ \widehat{\mu} μ . More specifically, we use three sets containing d d d -variate, univariate and bivariate words, respectively. The dictionaries below are based on the estimators previously proposed by Fan et al. [11 ] in kernel smoothing. These dictionaries can be utilized in several settings, including normal, Bernoulli and Poisson regression.
Dictionary D 0 D_0 D 0
Fan et al. [11 ] proposed the following adjustment term for the semiparametric estimator:
ξ ^ H ( x ) = solution ξ [ 1 n ∑ i = 1 n K H ( X i − x ) m ( X i , θ ^ ) γ { Y i − u − 1 ( m ( X i , θ ^ ) + m ( X i , θ ^ ) γ ξ ) } = 0 ] (8) \widehat{\xi}_H(x) = \operatorname{solution}_{\xi}\left[\frac{1}{n}\sum_{i=1}^{n} K_H(\mathbf{X}_i-x)m(\mathbf{X}_i,\widehat{\theta})^{\gamma}\left\{Y_i-u^{-1}\left(m(\mathbf{X}_i,\widehat{\theta})+m(\mathbf{X}_i,\widehat{\theta})^{\gamma}\xi\right)\right\}=0\right] \tag*{(8)} ξ H ( x ) = solution ξ [ n 1 i = 1 ∑ n K H ( X i − x ) m ( X i , θ ) γ { Y i − u − 1 ( m ( X i , θ ) + m ( X i , θ ) γ ξ ) } = 0 ] (8)
where θ ^ \widehat{\theta} θ is an estimator of the parameter θ \theta θ , K H ( x ) = ∣ H ∣ − 1 / 2 K ( H − 1 / 2 x ) K_H(x)=|H|^{-1/2}K\left(H^{-1/2}x\right) K H ( x ) = ∣ H ∣ − 1/2 K ( H − 1/2 x ) , K K K is a d d d -dimensional function called “kernel function,” and H H H is a symmetric positive-definite matrix named “bandwidth.”
Remark. This ξ ^ H \widehat{\xi}_H ξ H corresponds to the local constant case p = 0 p=0 p = 0 in Sections 2 and 3 of Fan et al. [11 ] . The estimators in Fan et al. [11 ] are constructed via a quasi-log-likelihood function rather than the Bregman divergence. However, these estimators coincide with those derived from the Bregman divergence, since there is a correspondence between the quasi-log-likelihood function and the Bregman divergence: for any quasi-log-likelihood function Q Q Q satisfying several conditions, there exists a strictly convex function W W W such that − Q ( x , y ) = d W ( y , x ) -Q(x,y)=d_W(y,x) − Q ( x , y ) = d W ( y , x ) , see § 3 of Zhang et al. [34 ] .
If U = x 2 / 2 U=x^2/2 U = x 2 /2 as in normal regression, the above solution (5.1) can be calculated explicitly as
ξ ^ H ( x ) = 1 n ∑ i = 1 n K H ( X i − x ) m ( X i , θ ^ ) γ { Y i − m ( X i , θ ^ ) } ( 1 / n ) ∑ j = 1 n K H ( X j − x ) m ( X j , θ ^ ) 2 γ (9) \widehat{\xi}_H(x)=\frac{1}{n}\sum_{i=1}^{n}\frac{K_H(\mathbf{X}_i-x)m(\mathbf{X}_i,\widehat{\theta})^\gamma\left\{Y_i-m(\mathbf{X}_i,\widehat{\theta})\right\}}{(1/n)\sum_{j=1}^{n}K_H(\mathbf{X}_j-x)m(\mathbf{X}_j,\widehat{\theta})^{2\gamma}} \tag*{(9)} ξ H ( x ) = n 1 i = 1 ∑ n ( 1/ n ) ∑ j = 1 n K H ( X j − x ) m ( X j , θ ) 2 γ K H ( X i − x ) m ( X i , θ ) γ { Y i − m ( X i , θ ) } (9)
On the other hand, if U ≠ x 2 / 2 U\ne x^2/2 U = x 2 /2 as in Bernoulli and Poisson regression, (5.1) cannot be solved explicitly. Then we replace the term u − 1 ( m ( X i , θ ^ ) + m ( X i , θ ^ ) γ ξ ) u^{-1}\left(m(\mathbf{X}_i,\widehat{\theta})+m(\mathbf{X}_i,\widehat{\theta})^\gamma\xi\right) u − 1 ( m ( X i , θ ) + m ( X i , θ ) γ ξ ) in (5.1) with the first-order approximation around ξ = 0 \xi=0 ξ = 0 , that is,
u − 1 ( m ( X i , θ ^ ) + m ( X i , θ ^ ) γ ξ ) ≈ u − 1 ( m ( X i , θ ^ ) ) + ( U ∗ ) ′ ′ ( m ( X i , θ ^ ) ) m ( X i , θ ^ ) γ ξ , u^{-1}\left(m(\mathbf{X}_i,\widehat{\theta})+m(\mathbf{X}_i,\widehat{\theta})^\gamma\xi\right)\approx u^{-1}\left(m(\mathbf{X}_i,\widehat{\theta})\right)+(U^*)''\left(m(\mathbf{X}_i,\widehat{\theta})\right)m(\mathbf{X}_i,\widehat{\theta})^\gamma\xi, u − 1 ( m ( X i , θ ) + m ( X i , θ ) γ ξ ) ≈ u − 1 ( m ( X i , θ ) ) + ( U ∗ ) ′′ ( m ( X i , θ ) ) m ( X i , θ ) γ ξ ,
and we obtain
solution ξ [ 1 n ∑ i = 1 n K H ( X i − ⋅ ) m ( X i , θ ^ ) γ × [ Y i − { u − 1 ( m ( X i , θ ^ ) ) + ( U ∗ ) ′ ′ ( m ( X i , θ ^ ) ) m ( X i , θ ^ ) γ ξ } ] ] = 0 = 1 n ∑ i = 1 n K H ( X i − ⋅ ) m ( X i , θ ^ ) γ { Y i − u − 1 ( m ( X i , θ ^ ) ) } ( 1 / n ) ∑ j = 1 n K H ( X j − ⋅ ) m ( X j , θ ^ ) 2 γ ( U ∗ ) ′ ′ ( m ( X j , θ ^ ) ) = 1 n ∑ i = 1 n B 0 ( x ∣ θ ^ , i , H ) , (10) \begin{aligned}
\underset{\xi}{\operatorname{solution}}\ \left[\frac{1}{n}\sum_{i=1}^{n}K_H(\mathbf{X}_i-\cdot)m(\mathbf{X}_i,\widehat{\theta})^\gamma\right.&\\
\left.\times\left[Y_i-\left\{u^{-1}\left(m(\mathbf{X}_i,\widehat{\theta})\right)+(U^*)''\left(m(\mathbf{X}_i,\widehat{\theta})\right)m(\mathbf{X}_i,\widehat{\theta})^\gamma\xi\right\}\right]\right]&=0\\
=\frac{1}{n}\sum_{i=1}^{n}\frac{K_H(\mathbf{X}_i-\cdot)m(\mathbf{X}_i,\widehat{\theta})^\gamma\left\{Y_i-u^{-1}\left(m(\mathbf{X}_i,\widehat{\theta})\right)\right\}}{(1/n)\sum_{j=1}^{n}K_H(\mathbf{X}_j-\cdot)m(\mathbf{X}_j,\widehat{\theta})^{2\gamma}(U^*)''\left(m(\mathbf{X}_j,\widehat{\theta})\right)}\\
=\frac{1}{n}\sum_{i=1}^{n}B_0\left(x\mid\widehat{\theta},i,H\right), \tag*{(10)}
\end{aligned} ξ solution [ n 1 i = 1 ∑ n K H ( X i − ⋅ ) m ( X i , θ ) γ × [ Y i − { u − 1 ( m ( X i , θ ) ) + ( U ∗ ) ′′ ( m ( X i , θ ) ) m ( X i , θ ) γ ξ } ] ] = n 1 i = 1 ∑ n ( 1/ n ) ∑ j = 1 n K H ( X j − ⋅ ) m ( X j , θ ) 2 γ ( U ∗ ) ′′ ( m ( X j , θ ) ) K H ( X i − ⋅ ) m ( X i , θ ) γ { Y i − u − 1 ( m ( X i , θ ) ) } = n 1 i = 1 ∑ n B 0 ( x ∣ θ , i , H ) , = 0 (10)
where
B 0 ( x ∣ θ , i , H ) = K H ( X i − x ) m ( X i , θ ) γ { Y i − u − 1 ( m ( X i , θ ) ) } ( 1 / n ) ∑ j = 1 n K H ( X j − x ) m ( X j , θ ) 2 γ ( U ∗ ) ′ ′ ( m ( X j , θ ) ) (11) B_0\left(x\mid\theta,i,H\right)=\frac{K_H(\mathbf{X}_i-x)m(\mathbf{X}_i,\theta)^\gamma\left\{Y_i-u^{-1}\left(m(\mathbf{X}_i,\theta)\right)\right\}}{(1/n)\sum_{j=1}^{n}K_H(\mathbf{X}_j-x)m(\mathbf{X}_j,\theta)^{2\gamma}(U^*)''\left(m(\mathbf{X}_j,\theta)\right)} \tag*{(11)} B 0 ( x ∣ θ , i , H ) = ( 1/ n ) ∑ j = 1 n K H ( X j − x ) m ( X j , θ ) 2 γ ( U ∗ ) ′′ ( m ( X j , θ ) ) K H ( X i − x ) m ( X i , θ ) γ { Y i − u − 1 ( m ( X i , θ ) ) } (11)
for x ∈ R d x\in\mathbb{R}^d x ∈ R d , θ ∈ R p \theta\in\mathbb{R}^p θ ∈ R p , i = 1 , … , n i=1,\ldots,n i = 1 , … , n and a positive-definite matrix H H H . Note that (5.3) is equal to (5.2) when U = x 2 / 2 U=x^2/2 U = x 2 /2 .
We adopt B 0 B_0 B 0 above as the words for our ξ ^ \widehat{\xi} ξ , and propose a dictionary
D 0 = { B 0 ( x ∣ θ ^ , i , H ) ∣ i = 1 , … , n , H ∈ H 0 } (12) \mathcal{D}_0=\left\{B_0\left(x\mid\widehat{\theta},i,H\right)\mid i=1,\ldots,n,H\in\mathbb{H}_0\right\} \tag*{(12)} D 0 = { B 0 ( x ∣ θ , i , H ) ∣ i = 1 , … , n , H ∈ H 0 } (12)
for a prepared set H 0 \mathbb{H}_0 H 0 of bandwidths.
Dictionaries D 1 D_1 D 1 and D 2 D_2 D 2
Here we fix the parametric model function as m ( x , θ ) = [ 1 x T ] θ m(x,\boldsymbol{\theta}) = [1\ x^{\mathsf{T}}]\boldsymbol{\theta} m ( x , θ ) = [ 1 x T ] θ , θ = [ θ 0 θ 1 ⋯ θ d ] T ∈ R d + 1 \boldsymbol{\theta} = [\theta_0\ \theta_1\ \cdots\ \theta_d]^{\mathsf{T}} \in\mathbb{R}^{d+1} θ = [ θ 0 θ 1 ⋯ θ d ] T ∈ R d + 1 .
Let 1 ≤ i ≤ n 1 \le i \le n 1 ≤ i ≤ n and 1 ≤ k < ℓ ≤ d 1 \le k < \ell\le d 1 ≤ k < ℓ ≤ d . For the variable x = [ x 1 x 2 ⋯ x d ] T x = [x_1\ x_2\ \cdots\ x_d]^{\mathsf{T}} x = [ x 1 x 2 ⋯ x d ] T , the i i i -th observation of the explanatory variables X i = [ X i 1 ⋯ X i d ] T \mathbf{X}_i = [X_{i1}\cdots X_{id}]^{\mathsf{T}} X i = [ X i 1 ⋯ X i d ] T , i = 1 , … , n i = 1,\ldots,n i = 1 , … , n and an estimator θ ^ = [ θ ^ 0 θ ^ 1 ⋯ θ ^ d ] T \widehat{\boldsymbol{\theta}} = [\widehat{\theta}_0\ \widehat{\theta}_1\cdots\widehat{\theta}_d]^{\mathsf{T}} θ = [ θ 0 θ 1 ⋯ θ d ] T of the parameter θ \boldsymbol{\theta} θ , we denote the three corresponding subvectors as follows: x k ℓ = [ x k x ℓ ] T \mathbf{x}_{k\ell} = [x_k\ x_\ell]^{\mathsf{T}} x k ℓ = [ x k x ℓ ] T , θ ^ 0 k ℓ = [ θ ^ 0 θ ^ k θ ^ ℓ ] T \widehat{\boldsymbol{\theta}}_{0k\ell} = [\widehat{\theta}_0\ \widehat{\theta}_k\ \widehat{\theta}_\ell]^{\mathsf{T}} θ 0 k ℓ = [ θ 0 θ k θ ℓ ] T and X i k ℓ = [ X i k X i ℓ ] T \mathbf{X}_{ik\ell} = [X_{ik}\ X_{i\ell}]^{\mathsf{T}} X ik ℓ = [ X ik X i ℓ ] T . In addition, we define submodels of m m m with X \mathbf{X} X : m k ( X i k , θ ^ 0 k ) = θ ^ 0 + θ ^ k X i k m_k(X_{ik},\widehat{\theta}_{0k}) = \widehat{\theta}_0+\widehat{\theta}_kX_{ik} m k ( X ik , θ 0 k ) = θ 0 + θ k X ik and m k ℓ ( X i k ℓ , θ ^ 0 k ℓ ) = θ ^ 0 + θ ^ k X i k + θ ^ ℓ X i ℓ m_{k\ell}(\mathbf{X}_{ik\ell},\widehat{\boldsymbol{\theta}}_{0k\ell}) = \widehat{\theta}_0+\widehat{\theta}_kX_{ik}+\widehat{\theta}_\ell X_{i\ell} m k ℓ ( X ik ℓ , θ 0 k ℓ ) = θ 0 + θ k X ik + θ ℓ X i ℓ . Using this notation, we define the following univariate and bivariate kernel-based functions
B 1 ( x k ∣ θ ^ , i , k , h ) = K h ( X i k − x k ) m k ( X i k , θ ^ 0 k ) γ { Y i − u − 1 ( m k ( X i k , θ ^ 0 k ) ) } ( 1 / n ) ∑ j = 1 n K h ( X j k − x k ) m k ( X j k , θ ^ 0 k ) 2 γ ( U ∗ ) ′ ′ ( m k ( X j k , θ ^ 0 k ) ) (13) B_1\left(x_k\mid\widehat{\boldsymbol{\theta}},i,k,h\right)=\frac{K_h(X_{ik}-x_k)m_k\left(X_{ik},\widehat{\theta}_{0k}\right)^\gamma\left\{Y_i-u^{-1}\left(m_k\left(X_{ik},\widehat{\theta}_{0k}\right)\right)\right\}}{(1/n)\sum_{j=1}^n K_h(X_{jk}-x_k)m_k\left(X_{jk},\widehat{\theta}_{0k}\right)^{2\gamma}(U^*)''\left(m_k\left(X_{jk},\widehat{\theta}_{0k}\right)\right)} \tag*{(13)} B 1 ( x k ∣ θ , i , k , h ) = ( 1/ n ) ∑ j = 1 n K h ( X j k − x k ) m k ( X j k , θ 0 k ) 2 γ ( U ∗ ) ′′ ( m k ( X j k , θ 0 k ) ) K h ( X ik − x k ) m k ( X ik , θ 0 k ) γ { Y i − u − 1 ( m k ( X ik , θ 0 k ) ) } (13)
for a positive real number h h h , and
B 2 ( x k ℓ ∣ θ ^ , i , k , ℓ , H ) = K H ( X i k ℓ − x k ℓ ) m k ℓ ( X i k ℓ , θ ^ 0 k ℓ ) γ { Y i − u − 1 ( m k ℓ ( X i k ℓ , θ ^ 0 k ℓ ) ) } ( 1 / n ) ∑ j = 1 n K H ( X j k ℓ − x k ℓ ) m k ℓ ( X j k ℓ , θ ^ 0 k ℓ ) 2 γ ( U ∗ ) ′ ′ ( m k ℓ ( X j k ℓ , θ ^ 0 k ℓ ) ) \begin{aligned}
B_2\left(\mathbf{x}_{k\ell}\mid\widehat{\boldsymbol{\theta}},i,k,\ell,H\right) \\
&=\frac{K_H(\mathbf{X}_{ik\ell}-\mathbf{x}_{k\ell})m_{k\ell}\left(\mathbf{X}_{ik\ell},\widehat{\boldsymbol{\theta}}_{0k\ell}\right)^\gamma\left\{Y_i-u^{-1}\left(m_{k\ell}\left(\mathbf{X}_{ik\ell},\widehat{\boldsymbol{\theta}}_{0k\ell}\right)\right)\right\}}{(1/n)\sum_{j=1}^n K_H(\mathbf{X}_{jk\ell}-\mathbf{x}_{k\ell})m_{k\ell}\left(\mathbf{X}_{jk\ell},\widehat{\boldsymbol{\theta}}_{0k\ell}\right)^{2\gamma}(U^*)''\left(m_{k\ell}\left(\mathbf{X}_{jk\ell},\widehat{\boldsymbol{\theta}}_{0k\ell}\right)\right)}
\end{aligned} B 2 ( x k ℓ ∣ θ , i , k , ℓ , H ) = ( 1/ n ) ∑ j = 1 n K H ( X j k ℓ − x k ℓ ) m k ℓ ( X j k ℓ , θ 0 k ℓ ) 2 γ ( U ∗ ) ′′ ( m k ℓ ( X j k ℓ , θ 0 k ℓ ) ) K H ( X ik ℓ − x k ℓ ) m k ℓ ( X ik ℓ , θ 0 k ℓ ) γ { Y i − u − 1 ( m k ℓ ( X ik ℓ , θ 0 k ℓ ) ) }
for a 2 × 2 2\times2 2 × 2 positive-definite matrix H H H . These B 1 B_1 B 1 and B 2 B_2 B 2 are univariate and bivariate versions of (5.4). We employ these functions as words in dictionaries. After preparing sets of bandwidths H 1 \mathbb{H}_1 H 1 and H 2 \mathbb{H}_2 H 2 , we define the following two dictionaries containing univariate or bivariate words:
D 1 = { B 1 ( x k ∣ θ ^ , i , k , h ) ∣ i = 1 , … , n , k = 1 , … , d , h ∈ H 1 } (14) D_1=\left\{B_1\left(x_k\mid\widehat{\boldsymbol{\theta}},i,k,h\right)\mid i=1,\ldots,n,k=1,\ldots,d,h\in\mathbb{H}_1\right\} \tag*{(14)} D 1 = { B 1 ( x k ∣ θ , i , k , h ) ∣ i = 1 , … , n , k = 1 , … , d , h ∈ H 1 } (14)
D 2 = D 1 ∪ { B 2 ( x k ℓ ∣ θ ^ , i , k , ℓ , H ) ∣ i = 1 , … , n , 1 ≤ k < ℓ ≤ d , H ∈ H 2 } . (15) D_2=D_1\cup\left\{B_2\left(\mathbf{x}_{k\ell}\mid\widehat{\boldsymbol{\theta}},i,k,\ell,H\right)\mid i=1,\ldots,n,1\le k<\ell\le d,H\in\mathbb{H}_2\right\}. \tag*{(15)} D 2 = D 1 ∪ { B 2 ( x k ℓ ∣ θ , i , k , ℓ , H ) ∣ i = 1 , … , n , 1 ≤ k < ℓ ≤ d , H ∈ H 2 } . (15)
Although trivariate or higher-dimensional words can be defined in a similar way, we focus on these univariate and bivariate words below.
Properties of μ ^ \widehat{\mu} μ with dictionaries D 0 D_0 D 0 , D 1 D_1 D 1 and D 2 D_2 D 2
We investigate the theoretical behavior of the proposed estimators μ ^ \widehat{\mu} μ constructed from the above dictionaries D 0 D_0 D 0 , D 1 D_1 D 1 and D 2 D_2 D 2 . As the kernel function K K K in these dictionaries, we mainly consider the Gaussian kernel K ( z ) = ( 2 π ) − d / 2 exp [ − ∥ z ∥ 2 2 / 2 ] K(z) = (2\pi)^{-d/2}\exp\left[-\|z\|_{2}^{2}/2\right] K ( z ) = ( 2 π ) − d /2 exp [ − ∥ z ∥ 2 2 /2 ] here. Recall the dictionary D ( Δ 1 , Δ 2 , c ) \mathcal{D}(\Delta_{1},\Delta_{2},c) D ( Δ 1 , Δ 2 , c ) for a dictionary D \mathcal{D} D and fixed Δ 1 , Δ 2 , c > 0 \Delta_{1},\Delta_{2},c > 0 Δ 1 , Δ 2 , c > 0 as defined after Lemma 4.2 in § 4 . Dictionaries D 0 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{0}(\Delta_{1},\Delta_{2},c) D 0 ( Δ 1 , Δ 2 , c ) , D 1 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{1}(\Delta_{1},\Delta_{2},c) D 1 ( Δ 1 , Δ 2 , c ) and D 2 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{2}(\Delta_{1},\Delta_{2},c) D 2 ( Δ 1 , Δ 2 , c ) derived from D 0 \mathcal{D}_{0} D 0 , D 1 \mathcal{D}_{1} D 1 and D 2 \mathcal{D}_{2} D 2 satisfy the following proposition:
Proposition 5.1. For fixed Δ 1 , Δ 2 , c > 0 \Delta_{1},\Delta_{2},c > 0 Δ 1 , Δ 2 , c > 0 , D 0 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{0}(\Delta_{1},\Delta_{2},c) D 0 ( Δ 1 , Δ 2 , c ) , D 1 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{1}(\Delta_{1},\Delta_{2},c) D 1 ( Δ 1 , Δ 2 , c ) and D 2 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{2}(\Delta_{1},\Delta_{2},c) D 2 ( Δ 1 , Δ 2 , c ) with the Gaussian kernel K K K satisfy the assumption (A6).
Therefore, the second term of the bound in (5) converges asymptotically to 0, when we construct the proposed estimator μ ^ \hat{\mu} μ ^ with D 0 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{0}(\Delta_{1},\Delta_{2},c) D 0 ( Δ 1 , Δ 2 , c ) , D 1 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{1}(\Delta_{1},\Delta_{2},c) D 1 ( Δ 1 , Δ 2 , c ) and D 2 ( Δ 1 , Δ 2 , c ) \mathcal{D}_{2}(\Delta_{1},\Delta_{2},c) D 2 ( Δ 1 , Δ 2 , c ) .
Simulation
In this section, we conduct Monte Carlo simulations to investigate the finite-sample performance of the proposed estimator μ ^ \hat{\mu} μ ^ introduced in § 3 . We utilize the dictionaries D 0 \mathcal{D}_{0} D 0 , D 1 \mathcal{D}_{1} D 1 and D 2 \mathcal{D}_{2} D 2 defined in § 5 , and consider normal, Bernoulli and Poisson regression in both low- and higher-dimensional settings.
Throughout this section, we assess estimation accuracy using the divergence risk R [ μ ˉ ] = E [ D U ( μ , μ ˉ ) ] \mathcal{R}[\bar{\mu}] = \mathbb{E}[D_{U}(\mu,\bar{\mu})] R [ μ ˉ ] = E [ D U ( μ , μ ˉ )] between the target regression function μ \mu μ and an estimator μ ˉ \bar{\mu} μ ˉ , as introduced in § 4 . A smaller value of R [ μ ˉ ] \mathcal{R}[\bar{\mu}] R [ μ ˉ ] indicates higher estimation accuracy. Specifically, we will compute a Monte Carlo approximation R ^ [ μ ˉ ] \widehat{\mathcal{R}}[\bar{\mu}] R [ μ ˉ ] of R [ μ ˉ ] \mathcal{R}[\bar{\mu}] R [ μ ˉ ] under the following settings: we assume that the d d d -dimensional explanatory variables X \mathbf{X} X follows the uniform distribution U ( [ − 1 , 1 ] d ) U\left([-1,1]^{d}\right) U ( [ − 1 , 1 ] d ) . The conditional distribution of the objective variable Y ∣ X = x Y\mid\mathbf{X}=x Y ∣ X = x is assumed to be: the normal distribution N ( μ ( x ) , ( 0.2 ) 2 ) N\left(\mu(x),(0.2)^{2}\right) N ( μ ( x ) , ( 0.2 ) 2 ) in normal regression; the Bernoulli distribution Be ( μ ( x ) ) \operatorname{Be}(\mu(x)) Be ( μ ( x )) in Bernoulli regression; and the Poisson distribution Po ( μ ( x ) ) \operatorname{Po}(\mu(x)) Po ( μ ( x )) in Poisson regression, where μ ( x ) \mu(x) μ ( x ) is the expectation of each conditional distribution. We independently generate n n n observations of ( Y , X ) (Y,\mathbf{X}) ( Y , X ) and repeat this procedure R R R times, then writing them as ( Y 1 [ r ] , X 1 [ r ] ) , … , ( Y n [ r ] , X n [ r ] ) \left(Y_{1}^{[r]},\mathbf{X}_{1}^{[r]}\right),\ldots,\left(Y_{n}^{[r]},\mathbf{X}_{n}^{[r]}\right) ( Y 1 [ r ] , X 1 [ r ] ) , … , ( Y n [ r ] , X n [ r ] ) for r = 1 , … , R r=1,\ldots,R r = 1 , … , R . We also independently generate S S S observations of X \mathbf{X} X : X 1 ∗ , … , X S ∗ \mathbf{X}_{1}^{*},\ldots,\mathbf{X}_{S}^{*} X 1 ∗ , … , X S ∗ . Calculating the estimator μ ˉ [ r ] \bar{\mu}^{[r]} μ ˉ [ r ] for each repetition r r r , we compute
R ^ [ μ ˉ ] = 1 R ∑ r = 1 R D U ∗ ( μ , μ ˉ [ r ] ) , D U ∗ ( μ , μ ˉ [ r ] ) = 1 S ∑ s = 1 S d U ( μ ( X s ∗ ) , μ ˉ [ r ] ( X s ∗ ) ) . (16) \widehat{\mathcal{R}}[\bar{\mu}] = \frac{1}{R}\sum_{r=1}^{R}D_U^*(\mu,\bar{\mu}^{[r]}), \qquad D_U^*(\mu,\bar{\mu}^{[r]}) = \frac{1}{S}\sum_{s=1}^{S}d_U\left(\mu(X_s^*),\bar{\mu}^{[r]}(X_s^*)\right). \tag*{(16)} R [ μ ˉ ] = R 1 r = 1 ∑ R D U ∗ ( μ , μ ˉ [ r ] ) , D U ∗ ( μ , μ ˉ [ r ] ) = S 1 s = 1 ∑ S d U ( μ ( X s ∗ ) , μ ˉ [ r ] ( X s ∗ ) ) . (16)
and its standard error
se ^ [ μ ˉ ] = Var ^ [ R ^ [ μ ˉ ] ] = 1 R 2 ∑ r = 1 R { D U ∗ ( μ , μ ˉ [ r ] ) − R ^ [ μ ˉ ] } 2 . (17) \widehat{\operatorname{se}}[\bar{\mu}] = \sqrt{\widehat{\operatorname{Var}}\left[\widehat{\mathcal{R}}[\bar{\mu}]\right]} = \sqrt{\frac{1}{R^2}\sum_{r=1}^{R}\left\{D_U^*(\mu,\bar{\mu}^{[r]})-\widehat{\mathcal{R}}[\bar{\mu}]\right\}^2}. \tag*{(17)} se [ μ ˉ ] = Var [ R [ μ ˉ ] ] = R 2 1 r = 1 ∑ R { D U ∗ ( μ , μ ˉ [ r ] ) − R [ μ ˉ ] } 2 . (17)
We set R = S = 1000 R=S=1000 R = S = 1000 .
Throughout this section, the parametric model function is set to u − 1 ( [ 1 x T ] θ ) u^{-1}([1\ x^{\mathsf{T}}]\theta) u − 1 ([ 1 x T ] θ ) for θ = [ θ 0 θ 1 ⋯ θ d ] T ∈ R d + 1 \theta=[\theta_0\ \theta_1\ \cdots\ \theta_d]^{\mathsf{T}}\in\mathbb{R}^{d+1} θ = [ θ 0 θ 1 ⋯ θ d ] T ∈ R d + 1 and the inverse link function u − 1 = U ∗ u^{-1}=U^* u − 1 = U ∗ . As the strictly convex function in the Bregman divergence, we use: in normal regression, U ( z ) = z 2 / 2 U(z)=z^2/2 U ( z ) = z 2 /2 — then u = u − 1 = id u=u^{-1}=\operatorname{id} u = u − 1 = id and U ∗ ( z ) = z 2 / 2 U^*(z)=z^2/2 U ∗ ( z ) = z 2 /2 ; in Bernoulli regression, U ( z ) = z ln z + ( 1 − z ) ln [ 1 − z ] − 1 U(z)=z\ln z+(1-z)\ln[1-z]-1 U ( z ) = z ln z + ( 1 − z ) ln [ 1 − z ] − 1 — then u ( z ) = ln [ z / ( 1 − z ) ] u(z)=\ln[z/(1-z)] u ( z ) = ln [ z / ( 1 − z )] , u − 1 ( z ) = 1 / ( 1 + e − z ) u^{-1}(z)=1/(1+e^{-z}) u − 1 ( z ) = 1/ ( 1 + e − z ) and U ∗ ( z ) = ln [ 1 + e z ] U^*(z)=\ln[1+e^z] U ∗ ( z ) = ln [ 1 + e z ] ; in Poisson regression, U ( z ) = z ln z − z U(z)=z\ln z-z U ( z ) = z ln z − z — then u ( z ) = ln z u(z)=\ln z u ( z ) = ln z , u − 1 ( z ) = e z u^{-1}(z)=e^z u − 1 ( z ) = e z and U ∗ ( z ) = e z U^*(z)=e^z U ∗ ( z ) = e z . In addition, we write x = [ x 1 x 2 ⋯ x d ] T \boldsymbol{x}=[x_1\ x_2\ \cdots\ x_d]^{\mathsf{T}} x = [ x 1 x 2 ⋯ x d ] T .
Low dimension
We compare the estimation accuracy of the proposed estimator μ ^ \widehat{\mu} μ with that of other estimators.
The settings and notation for the estimators in this subsection are listed below:
θ ^ M = arg min θ ∈ R d + 1 L ^ ( u − 1 ( [ 1 ⋅ T ] θ ) ) , (18) \widehat{\theta}_{\mathrm{M}}=\underset{\theta\in\mathbb{R}^{d+1}}{\arg\min}\ \widehat{L}\left(u^{-1}\left([1\ \cdot^{\mathsf{T}}]\theta\right)\right), \tag*{(18)} θ M = θ ∈ R d + 1 arg min L ( u − 1 ( [ 1 ⋅ T ] θ ) ) , (18)
where L ^ \widehat{L} L is (2.2) associated with the Bregman divergence D U D_U D U . This estimator θ ^ M \widehat{\theta}_{\mathrm{M}} θ M coincides with the MLE in GLMs. Under this setting, we compute R ^ [ μ ^ P ] \widehat{\mathcal{R}}[\widehat{\mu}_{\mathrm{P}}] R [ μ P ] and se ^ [ μ ^ P ] \widehat{\operatorname{se}}[\widehat{\mu}_{\mathrm{P}}] se [ μ P ] in (6.1) and (6.2) for resulting μ ^ P \widehat{\mu}_{\mathrm{P}} μ P .
Proposed semiparametric estimator μ ^ i γ \widehat{\mu}_i^\gamma μ i γ . The parametric model is the same as that in μ ^ P \widehat{\mu}_{\mathrm{P}} μ P . We consider both γ = 0 \gamma=0 γ = 0 and 1 1 1 , corresponding to additive and multiplicative adjustment, respectively. As the dictionary, we utilize D i \mathcal{D}_i D i for i = 0 i=0 i = 0 , 1 1 1 or 2 2 2 in Section 5 with the Gaussian
kernel K ( z ) = ( 2 π ) − d / 2 exp ( − ∥ z ∥ 2 2 / 2 ) K(z) = (2\pi)^{-d/2}\exp\left(-\lVert z\rVert_2^2/2\right) K ( z ) = ( 2 π ) − d /2 exp ( − ∥ z ∥ 2 2 /2 ) . The bandwidths of these dictionaries are given in each of the subsections below. We set the sequence π k \pi_k π k forming the coefficients of ξ ^ \hat{\xi} ξ ^ to π k = 2 / ( k − 1 + 2 ) \pi_k = 2/(k-1+2) π k = 2/ ( k − 1 + 2 ) , which has already been introduced in § 4 . For each parameter γ \gamma γ and dictionary index i i i , we write the proposed estimators as μ ^ i γ ( I ) \hat{\mu}^{\gamma}_i(I) μ ^ i γ ( I ) for the number of iterations I I I . Hence the adjustment terms in μ ^ i γ ( I ) \hat{\mu}^{\gamma}_i(I) μ ^ i γ ( I ) are composed of I I I selected words. We compute the smallest divergence risk min I ∈ I R ^ [ μ ^ i γ ( I ) ] \min_{I\in\mathbb{I}}\widehat{R}[\hat{\mu}^{\gamma}_i(I)] min I ∈ I R [ μ ^ i γ ( I )] over the range I = { 1 , 2 , … , 300 } \mathbb{I}=\{1,2,\ldots,300\} I = { 1 , 2 , … , 300 } . We also calculate se ^ [ μ ^ i γ ( I ) ] \widehat{\operatorname{se}}[\hat{\mu}^{\gamma}_i(I)] se [ μ ^ i γ ( I )] at argmin I ∈ I R ^ [ μ ^ i γ ( I ) ] \operatorname*{argmin}_{I\in\mathbb{I}}\widehat{R}[\hat{\mu}^{\gamma}_i(I)] argmin I ∈ I R [ μ ^ i γ ( I )] .
μ ^ S γ = u − 1 ( m ( ⋅ , θ ^ M ) + m ( ⋅ , θ ^ M ) ⋅ ξ ^ H γ ) . \hat{\mu}^{\gamma}_{S}=u^{-1}\left(m(\cdot,\hat{\theta}_{M})+m(\cdot,\hat{\theta}_{M})\cdot\hat{\xi}_{H}^{\gamma}\right). μ ^ S γ = u − 1 ( m ( ⋅ , θ ^ M ) + m ( ⋅ , θ ^ M ) ⋅ ξ ^ H γ ) .
where m m m , θ ^ M \hat{\theta}_{M} θ ^ M and u − 1 u^{-1} u − 1 are the same as those in our μ ^ i γ \hat{\mu}^{\gamma}_i μ ^ i γ , and ξ ^ H \hat{\xi}_{H} ξ ^ H is (5.1) with the Gaussian kernel. We take bandwidths H S = { ( 0.01 ⋅ s ) 2 ⋅ I d ∣ s = 1 , … , 200 } \mathbb{H}_{S}=\left\{(0.01\cdot s)^2\cdot I_d\mid s=1,\ldots,200\right\} H S = { ( 0.01 ⋅ s ) 2 ⋅ I d ∣ s = 1 , … , 200 } and denote the resulting estimator by μ ^ S γ ( H ) \hat{\mu}^{\gamma}_{S}(H) μ ^ S γ ( H ) for a bandwidth H ∈ H S H\in\mathbb{H}_{S} H ∈ H S . We compute the divergence risk min H ∈ H S R ^ [ μ ^ S γ ( H ) ] \min_{H\in\mathbb{H}_{S}}\widehat{R}[\hat{\mu}^{\gamma}_{S}(H)] min H ∈ H S R [ μ ^ S γ ( H )] and its standard error se ^ [ μ ^ S γ ( H ) ] \widehat{\operatorname{se}}[\hat{\mu}^{\gamma}_{S}(H)] se [ μ ^ S γ ( H )] at argmin H ∈ H S R ^ [ μ ^ S γ ( H ) ] \operatorname*{argmin}_{H\in\mathbb{H}_{S}}\widehat{R}[\hat{\mu}^{\gamma}_{S}(H)] argmin H ∈ H S R [ μ ^ S γ ( H )] .
Nonparametric local linear estimator μ ^ L \hat{\mu}_{L} μ ^ L . We also compare the proposed estimators with the nonparametric local linear estimator in Fan et al. [10 ] . The link function is set to u − 1 u^{-1} u − 1 , the same as μ ^ i γ \hat{\mu}^{\gamma}_i μ ^ i γ . We use bandwidths H L = { ( 0.01 ⋅ s ) 2 ⋅ I d ∣ s = 1 , … , 250 } \mathbb{H}_{L}=\left\{(0.01\cdot s)^2\cdot I_d\mid s=1,\ldots,250\right\} H L = { ( 0.01 ⋅ s ) 2 ⋅ I d ∣ s = 1 , … , 250 } and the Gaussian kernel. The notation μ ^ L ( H ) \hat{\mu}_{L}(H) μ ^ L ( H ) denotes this estimator with a bandwidth H ∈ H L H\in\mathbb{H}_{L} H ∈ H L . We compute min H ∈ H L R ^ [ μ ^ L ( H ) ] \min_{H\in\mathbb{H}_{L}}\widehat{R}[\hat{\mu}_{L}(H)] min H ∈ H L R [ μ ^ L ( H )] and se ^ [ μ ^ L ( H ) ] \widehat{\operatorname{se}}[\hat{\mu}_{L}(H)] se [ μ ^ L ( H )] at argmin H ∈ H L R ^ [ μ ^ L ( H ) ] \operatorname*{argmin}_{H\in\mathbb{H}_{L}}\widehat{R}[\hat{\mu}_{L}(H)] argmin H ∈ H L R [ μ ^ L ( H )] .
GBRT μ ^ X \hat{\mu}_{X} μ ^ X via XGBoost. We also compare our estimators with the GBRT implemented in the package xgboost in R, see Chen et al. [7 ] and Chen and Guestrin [6 ] . The common settings for its arguments throughout this Section 6 are: base_score = ( 1 / n ) ∑ i = 1 n Y i =(1/n)\sum_{i=1}^{n}Y_i = ( 1/ n ) ∑ i = 1 n Y i ; min_child_weight = 0 =0 = 0 ; maxdepth ∈ { 1 , 2 , … , 6 } = M X \in\{1,2,\ldots,6\}=\mathcal{M}_{X} ∈ { 1 , 2 , … , 6 } = M X ; and tree_method = “exact”. Moreover, we set the parameter objective = “reg:squarederror” in normal regression; = “binary:logistic” in Bernoulli regression; and = “count:poisson” in Poisson regression. The parameters eta and lambda are: in normal regression, eta = 0.1 and lambda = 0; in
Bernoulli and Poisson regression, eta = 0.01 and lambda ∈ { 0 , 1 , 10 , 100 , 1000 } = Λ X \in\{0,1,10,100,1000\} = \Lambda_X ∈ { 0 , 1 , 10 , 100 , 1000 } = Λ X . All the other parameters not mentioned here are set to their default values. To denote the estimator with maxdepth = m m m and the number of iterations nrounds = I I I , we use the notation μ ^ X ( m , I ) \hat{\mu}_X(m,I) μ ^ X ( m , I ) for normal regression, and μ ^ X ( m , λ , I ) \hat{\mu}_X(m,\lambda,I) μ ^ X ( m , λ , I ) for lambda = λ \lambda λ in Bernoulli and Poisson regression. Its divergence risk min m ∈ M X , I ∈ I X R ^ [ μ ^ X ( m , I ) ] \min_{m\in\mathcal{M}_X,I\in\mathcal{I}_X}\widehat{\mathcal{R}}[\hat{\mu}_X(m,I)] min m ∈ M X , I ∈ I X R [ μ ^ X ( m , I )] or min m ∈ M X , λ ∈ Λ X , I ∈ I X R ^ [ μ ^ X ( m , λ , I ) ] \min_{m\in\mathcal{M}_X,\lambda\in\Lambda_X,I\in\mathcal{I}_X}\widehat{\mathcal{R}}[\hat{\mu}_X(m,\lambda,I)] min m ∈ M X , λ ∈ Λ X , I ∈ I X R [ μ ^ X ( m , λ , I )] is calculated, where the range I X \mathcal{I}_X I X is as follows: in normal regression, I X = { 1 , 2 , … , 20000 } \mathcal{I}_X = \{1,2,\ldots,20000\} I X = { 1 , 2 , … , 20000 } ; in Bernoulli and Poisson regression, we first set I X = { 1 , 2 , … , 5000 } \mathcal{I}_X = \{1,2,\ldots,5000\} I X = { 1 , 2 , … , 5000 } . If the minimizing I I I is between 4900 and 5000, we additionally search over { 5001 , 5002 , … , 10000 } \{5001,5002,\ldots,10000\} { 5001 , 5002 , … , 10000 } for the minimum value. If this minimizing I I I satisfies 9901 ≤ I ≤ 10000 9901 \le I \le10000 9901 ≤ I ≤ 10000 , then we again expand the search range to { 10001 , 10002 , … , 15000 } \{10001,10002,\ldots,15000\} { 10001 , 10002 , … , 15000 } . We also compute se ^ [ μ ^ X ( m , I ) ] \widehat{\operatorname{se}}[\hat{\mu}_X(m,I)] se [ μ ^ X ( m , I )] or se ^ [ μ ^ X ( m , λ , I ) ] \widehat{\operatorname{se}}[\hat{\mu}_X(m,\lambda,I)] se [ μ ^ X ( m , λ , I )] at the pair of searched parameters that minimizes the risk.
Normal regression
Here we consider normal regression in d = 3 d=3 d = 3 and the following five target regression functions:
μ 1 ( x ) = exp [ x 1 + x 2 + x 3 ] , \mu_1(x)=\exp[x_1+x_2+x_3], μ 1 ( x ) = exp [ x 1 + x 2 + x 3 ] ,
μ 2 ( x ) = e x 1 + sin ( π 2 ( x 2 + x 3 ) ) , \mu_2(x)=e^{x_1}+\sin\left(\frac{\pi}{2}(x_2+x_3)\right), μ 2 ( x ) = e x 1 + sin ( 2 π ( x 2 + x 3 ) ) ,
μ 3 ( x ) = x 1 2 + e x 2 + sin ( π x 3 ) , \mu_3(x)=x_1^2+e^{x_2}+\sin(\pi x_3), μ 3 ( x ) = x 1 2 + e x 2 + sin ( π x 3 ) ,
μ 4 ( x ) = 1 + x 1 + 2 x 2 + 3 x 3 + x 1 2 + e x 2 + sin ( π x 3 ) , \mu_4(x)=1+x_1+2x_2+3x_3+x_1^2+e^{x_2}+\sin(\pi x_3), μ 4 ( x ) = 1 + x 1 + 2 x 2 + 3 x 3 + x 1 2 + e x 2 + sin ( π x 3 ) ,
μ 5 ( x ) = ( 1 + x 1 + 2 x 2 + 3 x 3 ) ⋅ { x 1 2 + e x 2 + sin ( π x 3 ) } . \mu_5(x)=(1+x_1+2x_2+3x_3)\cdot\{x_1^2+e^{x_2}+\sin(\pi x_3)\}. μ 5 ( x ) = ( 1 + x 1 + 2 x 2 + 3 x 3 ) ⋅ { x 1 2 + e x 2 + sin ( π x 3 )} .
The sample size is n = 50 , 200 n=50,200 n = 50 , 200 .
We utilize the dictionaries D 0 D_0 D 0 , D 1 D_1 D 1 and D 2 D_2 D 2 for the proposed estimator μ ^ i \hat{\mu}_i μ ^ i , so that i = 0 , 1 , 2 i=0,1,2 i = 0 , 1 , 2 . The sets of bandwidths are given by H 0 = { s 2 ⋅ I d ∣ s 2 = 0.1 , 0.2 , … , 1.5 } \mathcal{H}_0=\{s^2\cdot I_d\mid s^2=0.1,0.2,\ldots,1.5\} H 0 = { s 2 ⋅ I d ∣ s 2 = 0.1 , 0.2 , … , 1.5 } for D 0 D_0 D 0 , H 1 = { 0.1 , 0.2 , … , 1.0 } \mathcal{H}_1=\{0.1,0.2,\ldots,1.0\} H 1 = { 0.1 , 0.2 , … , 1.0 } for D 1 D_1 D 1 and H 2 = { s 2 ⋅ I 2 ∣ s 2 = 0.1 , 0.2 , … , 1.0 } \mathcal{H}_2=\{s^2\cdot I_2\mid s^2=0.1,0.2,\ldots,1.0\} H 2 = { s 2 ⋅ I 2 ∣ s 2 = 0.1 , 0.2 , … , 1.0 } for D 2 D_2 D 2 .
The simulation results for these settings are presented in Table 6.1 . For each μ \mu μ and n n n , Table 6.1 lists the above-mentioned divergence risks as follows: in columns 3–5, min I ∈ I R ^ [ μ ^ i 0 ( I ) ] \min_{I\in\mathcal{I}}\widehat{\mathcal{R}}[\hat{\mu}_i^0(I)] min I ∈ I R [ μ ^ i 0 ( I )] for
Table 6.1. The minima of estimated divergence risks (× 10 3 \times10^{3} × 1 0 3 ) in low-dimensional normal regression.
γ = 0 \gamma= 0 γ = 0 , i = 0 , 1 , 2 i = 0,1,2 i = 0 , 1 , 2 ; in columns 6–8, min I ∈ I R ^ [ μ ^ i 1 ( I ) ] \min_{I\in\mathcal{I}} \widehat{\mathcal{R}}[\widehat{\mu}^{1}_{i}(I)] min I ∈ I R [ μ i 1 ( I )] for γ = 1 \gamma= 1 γ = 1 , i = 0 , 1 , 2 i = 0,1,2 i = 0 , 1 , 2 ; in column 9, R ^ [ μ ^ P ] \widehat{\mathcal{R}}[\widehat{\mu}_{P}] R [ μ P ] ; in columns 10 and 11, min H ∈ H S R ^ [ μ ^ S γ ( H ) ] \min_{H\in\mathcal{H}_{S}} \widehat{\mathcal{R}}[\widehat{\mu}^{\gamma}_{S}(H)] min H ∈ H S R [ μ S γ ( H )] for γ = 0 , 1 \gamma= 0,1 γ = 0 , 1 ; in column 12, min m ∈ M X , I ∈ I X R ^ [ μ ^ X ( m , I ) ] \min_{m\in\mathcal{M}_{X},I\in\mathcal{I}_{X}} \widehat{\mathcal{R}}[\widehat{\mu}_{X}(m,I)] min m ∈ M X , I ∈ I X R [ μ X ( m , I )] ; and in column 13, min H ∈ H L R ^ [ μ ^ L ( H ) ] \min_{H\in\mathcal{H}_{L}} \widehat{\mathcal{R}}[\widehat{\mu}_{L}(H)] min H ∈ H L R [ μ L ( H )] . Next to these risk values, the corresponding standard errors are given in square brackets. The parameters minimizing the risks are also reported beneath the corresponding risk values: in columns 3–5, argmin I ∈ I R ^ [ μ ^ i 0 ( I ) ] \operatorname{argmin}_{I\in\mathcal{I}} \widehat{\mathcal{R}}[\widehat{\mu}^{0}_{i}(I)] argmin I ∈ I R [ μ i 0 ( I )] for i = 0 , 1 , 2 i = 0,1,2 i = 0 , 1 , 2 ; in columns 6–8, argmin I ∈ I R ^ [ μ ^ i 1 ( I ) ] \operatorname{argmin}_{I\in\mathcal{I}} \widehat{\mathcal{R}}[\widehat{\mu}^{1}_{i}(I)] argmin I ∈ I R [ μ i 1 ( I )] for i = 0 , 1 , 2 i = 0,1,2 i = 0 , 1 , 2 ; in columns 10 and 11, argmin H ∈ H S R ^ [ μ ^ S γ ( H ) ] \operatorname{argmin}_{H\in\mathcal{H}_{S}} \widehat{\mathcal{R}}[\widehat{\mu}^{\gamma}_{S}(H)] argmin H ∈ H S R [ μ S γ ( H )] for γ = 0 , 1 \gamma= 0,1 γ = 0 , 1 ; in column 12, argmin m ∈ M X , I ∈ I X R ^ [ μ ^ X ( m , I ) ] \operatorname{argmin}_{m\in\mathcal{M}_{X},I\in\mathcal{I}_{X}} \widehat{\mathcal{R}}[\widehat{\mu}_{X}(m,I)] argmin m ∈ M X , I ∈ I X R [ μ X ( m , I )] ; and in column 13, argmin H ∈ H L R ^ [ μ ^ L ( H ) ] \operatorname{argmin}_{H\in\mathcal{H}_{L}} \widehat{\mathcal{R}}[\widehat{\mu}_{L}(H)] argmin H ∈ H L R [ μ L ( H )] . In each row, the row-wise minimum, second minimum and third minimum risk values are underlined with solid, dashed and dotted lines, respectively.
Observations from Table 6.1 are as follows:
6.a. Comparing μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ with the other estimators. As shown in each row, in almost all cases of ( μ , n ) (\mu,n) ( μ , n ) except ( μ , n ) = ( μ 1 , 50 ) , ( μ 1 , 200 ) (\mu,n) = (\mu_{1},50),(\mu_{1},200) ( μ , n ) = ( μ 1 , 50 ) , ( μ 1 , 200 ) , the risk values underlined with the solid lines are given by μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ with some adjustment type γ \gamma γ and a dictionary D i \mathcal{D}_{i} D i .
6.a.1. μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ vs μ ^ P \widehat{\mu}_{P} μ P . In columns 3–8 and column 9, the risks for μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ are smaller than that of μ ^ P \widehat{\mu}_{P} μ P for any γ \gamma γ and i i i in each ( μ , n ) (\mu,n) ( μ , n ) . This means that μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ adjusts μ ^ P \widehat{\mu}_{P} μ P well in these cases.
6.a.2. μ ^ 0 0 \widehat{\mu}^{0}_{0} μ 0 0 vs μ ^ S 0 \widehat{\mu}^{0}_{S} μ S 0 . Columns 3 and 10 indicate that μ ^ 0 0 \widehat{\mu}^{0}_{0} μ 0 0 yields a smaller risk value than μ ^ S 0 \widehat{\mu}^{0}_{S} μ S 0 in each ( μ , n ) (\mu,n) ( μ , n ) . That is, μ ^ P \widehat{\mu}_{P} μ P is improved by the adjustment constructed with the algorithm more effectively than by that in μ ^ S 0 \widehat{\mu}^{0}_{S} μ S 0 .
6.a.3. μ ^ 0 1 \widehat{\mu}^{1}_{0} μ 0 1 vs μ ^ S 1 \widehat{\mu}^{1}_{S} μ S 1 . Except when ( μ , n ) = ( μ 4 , 50 ) (\mu,n) = (\mu_{4},50) ( μ , n ) = ( μ 4 , 50 ) , similar results to those in the description 6.a.2. can be observed between μ ^ 0 1 \widehat{\mu}^{1}_{0} μ 0 1 (column 6) and μ ^ S 1 \widehat{\mu}^{1}_{S} μ S 1 (column 11).
6.a.4. μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ vs μ ^ X \widehat{\mu}_{X} μ X . According to columns 3–8 and column 12, several μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ with some γ \gamma γ , i i i have higher accuracy than μ ^ X \widehat{\mu}_{X} μ X . On the other hand, in ( μ , n ) = ( μ 3 , 50 ) , ( μ 3 , 200 ) , ( μ 4 , 200 ) (\mu,n) = (\mu_{3},50),(\mu_{3},200),(\mu_{4},200) ( μ , n ) = ( μ 3 , 50 ) , ( μ 3 , 200 ) , ( μ 4 , 200 ) , where μ \mu μ are additive functions of univariate ones, μ ^ X \widehat{\mu}_{X} μ X is the second or the third best among all the compared estimators.
6.a.5. μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ vs μ ^ L \widehat{\mu}_{L} μ L . In ( μ , n ) = ( μ 1 , 50 ) , ( μ 1 , 200 ) (\mu,n) = (\mu_{1},50),(\mu_{1},200) ( μ , n ) = ( μ 1 , 50 ) , ( μ 1 , 200 ) , μ ^ L \widehat{\mu}_{L} μ L is more accurate than μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ with any γ \gamma γ , i i i . One possible reason for this is as follows: considering the following quantity
Ratio ( μ , n ) = R ^ [ μ ^ P ] min H ∈ H L R ^ [ μ ^ L ( H ) ] \operatorname{Ratio}(\mu,n) = \frac{\widehat{\mathcal{R}}[\widehat{\mu}_{P}]}{\min_{H\in\mathcal{H}_{L}} \widehat{\mathcal{R}}[\widehat{\mu}_{L}(H)]} Ratio ( μ , n ) = min H ∈ H L R [ μ L ( H )] R [ μ P ]
the resulting Ratio ( μ 1 , 50 ) ≈ 7.7 \operatorname{Ratio}(\mu_1,50) \approx7.7 Ratio ( μ 1 , 50 ) ≈ 7.7 for the case ( μ , n ) = ( μ 1 , 50 ) (\mu,n)=(\mu_1,50) ( μ , n ) = ( μ 1 , 50 ) is much larger than the other Ratio ( μ i , 50 ) \operatorname{Ratio}(\mu_i,50) Ratio ( μ i , 50 ) for i = 2 , … , 5 i=2,\ldots,5 i = 2 , … , 5 . This is also seen when n = 200 n=200 n = 200 , among Ratio ( μ 1 , 200 ) ≈ 45.6 \operatorname{Ratio}(\mu_1,200) \approx45.6 Ratio ( μ 1 , 200 ) ≈ 45.6 and the other Ratio ( μ i , 200 ) \operatorname{Ratio}(\mu_i,200) Ratio ( μ i , 200 ) , i = 2 , … , 5 i=2,\ldots,5 i = 2 , … , 5 . Hence, the risks of μ ^ P \hat{\mu}_{P} μ ^ P are relatively larger than those of μ ^ L \hat{\mu}_{L} μ ^ L , and significant improvement in such μ ^ P \hat{\mu}_{P} μ ^ P cannot be provided by the adjustment in μ ^ i γ \hat{\mu}_{i}^{\gamma} μ ^ i γ in these cases.
6.b. μ ^ i 0 \hat{\mu}_{i}^{0} μ ^ i 0 vs μ ^ i 1 \hat{\mu}_{i}^{1} μ ^ i 1 . In each ( μ , n ) (\mu,n) ( μ , n ) , the smallest risk among μ ^ i γ \hat{\mu}_{i}^{\gamma} μ ^ i γ is achieved by μ ^ i 0 \hat{\mu}_{i}^{0} μ ^ i 0 with γ = 0 \gamma=0 γ = 0 and some i i i . That is, in these cases, the additive adjustment is more effective than the multiplicative adjustment. In particular, since μ 5 \mu_5 μ 5 is defined as the product of linear and nonlinear functions, one might intuitively expect that μ 5 \mu_5 μ 5 would be better estimated by μ ^ i 1 \hat{\mu}_{i}^{1} μ ^ i 1 than μ ^ i 0 \hat{\mu}_{i}^{0} μ ^ i 0 , but this does not occur in the last rows of Table 6.1 .
6.c. μ ^ 0 0 \hat{\mu}_{0}^{0} μ ^ 0 0 vs μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 vs μ ^ 2 0 \hat{\mu}_{2}^{0} μ ^ 2 0 . As seen in columns 3–5, μ ^ i 0 \hat{\mu}_{i}^{0} μ ^ i 0 with γ = 0 \gamma=0 γ = 0 and different i i i yield the best accuracy for different μ \mu μ among μ ^ 0 0 \hat{\mu}_{0}^{0} μ ^ 0 0 , μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 and μ ^ 2 0 \hat{\mu}_{2}^{0} μ ^ 2 0 . Intuitively, this can be explained by considering whether the variables in μ \mu μ match those in the words constructing the adjustment term in μ ^ i 0 \hat{\mu}_{i}^{0} μ ^ i 0 : for example, we focus on μ 3 \mu_3 μ 3 and μ 4 \mu_4 μ 4 . Since these targets are additive functions of univariate components, the adjustment for γ = 0 \gamma=0 γ = 0 seems to be more appropriately constructed from D 1 \mathcal{D}_1 D 1 of univariate words than D 0 \mathcal{D}_0 D 0 and D 2 \mathcal{D}_2 D 2 , which include bivariate or trivariate words. In fact, Table 6.1 shows that μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 estimates μ 3 \mu_3 μ 3 and μ 4 \mu_4 μ 4 better than μ ^ 0 0 \hat{\mu}_{0}^{0} μ ^ 0 0 and μ ^ 2 0 \hat{\mu}_{2}^{0} μ ^ 2 0 . Next for μ 2 \mu_2 μ 2 , which sums univariate and bivariate functions, it can be said that the adjustment term should be constructed from D 2 \mathcal{D}_2 D 2 , which has both univariate and bivariate words. Actually, μ ^ 2 0 \hat{\mu}_{2}^{0} μ ^ 2 0 has the best accuracy among μ ^ i 0 \hat{\mu}_{i}^{0} μ ^ i 0 in Table 6.1 . An analogous comparison of μ ^ 0 0 \hat{\mu}_{0}^{0} μ ^ 0 0 to μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 , μ ^ 2 0 \hat{\mu}_{2}^{0} μ ^ 2 0 can be made when μ = μ 1 \mu=\mu_1 μ = μ 1 , indicating that μ ^ 0 0 \hat{\mu}_{0}^{0} μ ^ 0 0 of trivariate words provides better estimation than μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 and μ ^ 2 0 \hat{\mu}_{2}^{0} μ ^ 2 0 . In this way, there seems to be some appropriate combination of γ \gamma γ and a dictionary for estimating μ \mu μ .
In addition to Table 6.1, Figures 6.1 and 6.2 present the curves of the estimated divergence risks for each estimator against the number of iterations I I I in μ ^ i γ \hat{\mu}_{i}^{\gamma} μ ^ i γ , when ( μ , n ) = ( μ 1 , 50 ) (\mu,n)=(\mu_1,50) ( μ , n ) = ( μ 1 , 50 ) and ( μ 3 , 50 ) (\mu_3,50) ( μ 3 , 50 ) . The risk curve for μ ^ 0 0 \hat{\mu}_{0}^{0} μ ^ 0 0 (respectively μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 , μ ^ 2 0 \hat{\mu}_{2}^{0} μ ^ 2 0 ) is drawn with the points “○” (respectively “●”, “⊕”). In contrast, the risk curve for μ ^ 0 1 \hat{\mu}_{0}^{1} μ ^ 0 1 (respectively μ ^ 1 1 \hat{\mu}_{1}^{1} μ ^ 1 1 , μ ^ 2 1 \hat{\mu}_{2}^{1} μ ^ 2 1 ) is depicted with the points “□” (respectively “■,” “⊞”). The horizontal dotted lines represent the results for the estimators other than μ ^ i γ \hat{\mu}^{\gamma}_{i} μ ^ i γ ; at both ends of these lines, the points “×” indicate the result for μ ^ P \hat{\mu}_{P} μ ^ P ; “⊗” for μ ^ S 0 \hat{\mu}^{0}_{S} μ ^ S 0 ; “⊠” for μ ^ S 1 \hat{\mu}^{1}_{S} μ ^ S 1 ; “▽” for μ ^ X \hat{\mu}_{X} μ ^ X ; and “△” for μ ^ L \hat{\mu}_{L} μ ^ L .
6.d. Figure 6.1 . The risk curves for μ ^ 0 0 \hat{\mu}^{0}_{0} μ ^ 0 0 , μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 , μ ^ 1 1 \hat{\mu}^{1}_{1} μ ^ 1 1 , μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 and μ ^ 2 1 \hat{\mu}^{1}_{2} μ ^ 2 1 decrease as I I I increases. On the other hand, the risk of μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 initially decreases with I I I , but increases from I ≈ 30 I \approx30 I ≈ 30 , eventually taking much the same risk values as μ ^ P \hat{\mu}_{P} μ ^ P for sufficiently large I I I .
Figure 6.1. The curves of estimated divergence risks for each estimator against the number of iterations for the proposed estimators μ ^ i γ \hat{\mu}^{\gamma}_{i} μ ^ i γ , when μ = μ 1 \mu= \mu_{1} μ = μ 1 and n = 50 n = 50 n = 50 in normal regression. ○: μ ^ 0 0 \hat{\mu}^{0}_{0} μ ^ 0 0 , ●: μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 , ⊕: μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 , □: μ ^ 0 1 \hat{\mu}^{1}_{0} μ ^ 0 1 , ■: μ ^ 1 1 \hat{\mu}^{1}_{1} μ ^ 1 1 , ⊞: μ ^ 2 1 \hat{\mu}^{1}_{2} μ ^ 2 1 , ×: μ ^ P \hat{\mu}_{P} μ ^ P , ⊗: μ ^ S 0 \hat{\mu}^{0}_{S} μ ^ S 0 , ⊠: μ ^ S 1 \hat{\mu}^{1}_{S} μ ^ S 1 , ▽: μ ^ X \hat{\mu}_{X} μ ^ X and △: μ ^ L \hat{\mu}_{L} μ ^ L .
6.e. Figure 6.2 . The risks of μ ^ i γ \hat{\mu}^{\gamma}_{i} μ ^ i γ with each γ \gamma γ and i i i decrease with I I I . In particular, the risk of μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 becomes smaller than that of μ ^ L \hat{\mu}_{L} μ ^ L around I ≈ 200 I \approx200 I ≈ 200 , while the risk of μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 does so before I ≈ 100 I \approx100 I ≈ 100 , and μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 always outperforms the other estimators after I ≈ 100 I \approx100 I ≈ 100 . This does not contradict the intuition in the description 6.c. that, when estimating the additive target μ = μ 3 \mu= \mu_{3} μ = μ 3 of univariate functions, the adjustment for γ = 0 \gamma= 0 γ = 0 is more appropriately constructed from D 1 \mathcal{D}_{1} D 1 of univariate words than D 2 \mathcal{D}_{2} D 2 including bivariate ones.
Figure 6.2. The curves of estimated divergence risks for each estimator against the number of iterations for the proposed estimators μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ , when μ = μ 3 \mu=\mu_3 μ = μ 3 and n = 50 n=50 n = 50 in normal regression. ◯ \bigcirc ◯ : μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 , ∙ \bullet ∙ : μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 , ⊕ \oplus ⊕ : μ ^ 2 0 \hat{\mu}_2^0 μ ^ 2 0 , □ \square □ : μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 , ■ \blacksquare ■ : μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 , ⊞ \boxplus ⊞ : μ ^ 2 1 \hat{\mu}_2^1 μ ^ 2 1 , × \times × : μ ^ P \hat{\mu}_{P} μ ^ P , ⊗ \otimes ⊗ : μ ^ S 0 \hat{\mu}_{S}^{0} μ ^ S 0 , ⊠ \boxtimes ⊠ : μ ^ S 1 \hat{\mu}_{S}^{1} μ ^ S 1 , ▽ \triangledown ▽ : μ ^ X \hat{\mu}_{X} μ ^ X and △ \triangle △ : μ ^ L \hat{\mu}_{L} μ ^ L .
6.f. Figure 6.1 vs Figure 6.2 . In Figure 6.1 , μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 improves μ ^ P \hat{\mu}_{P} μ ^ P less than the other proposed estimators μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ for each I I I . In contrast, Figure 6.2 shows that μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 always outperforms the other estimators when I > 100 I>100 I > 100 . As described in 6.c., this difference in the behavior of μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 between μ = μ 1 \mu=\mu_1 μ = μ 1 and μ = μ 3 \mu=\mu_3 μ = μ 3 may be explained by considering whether the variables in μ \mu μ match those in the words.
In normal regression, it can be claimed that the proposed estimators with dictionaries D 0 \mathcal{D}_0 D 0 , D 1 \mathcal{D}_1 D 1 and D 2 \mathcal{D}_2 D 2 have good estimation accuracy in the simulated cases.
Bernoulli and Poisson regression
For Bernoulli and Poisson regression, we consider the following dimension d d d , mean function μ \mu μ , and sample size n n n :
d = 3 d=3 d = 3 , μ ( x ) = p 1 ( x ) = 0.5 + 0.4 cos ( π 3 ( x 1 + x 2 + x 3 ) ) \mu(x)=p_1(x)=0.5+0.4\cos\left(\frac{\pi}{3}(x_1+x_2+x_3)\right) μ ( x ) = p 1 ( x ) = 0.5 + 0.4 cos ( 3 π ( x 1 + x 2 + x 3 ) ) and n = 50 , 200 n=50,200 n = 50 , 200 ;
in Poisson regression,
d = 3 d=3 d = 3 , μ ( x ) = λ 1 ( x ) = exp [ 2 sin ( π 12 ( x 1 + x 2 + x 3 ) − π 2 ) + 4 ] \mu(x)=\lambda_1(x)=\exp\left[2\sin\left(\frac{\pi}{12}(x_1+x_2+x_3)-\frac{\pi}{2}\right)+4\right] μ ( x ) = λ 1 ( x ) = exp [ 2 sin ( 12 π ( x 1 + x 2 + x 3 ) − 2 π ) + 4 ] and n = 50 , 200 n=50,200 n = 50 , 200 ;
We utilize the dictionaries D 0 \mathcal{D}_0 D 0 and D 1 \mathcal{D}_1 D 1 for the proposed estimator μ ^ i γ \widehat{\mu}^{\gamma}_i μ i γ (that is, i = 0 , 1 i=0,1 i = 0 , 1 ). We apply several sets of bandwidths defined for h min = 0.1 , 0.2 , … , 0.7 h_{\min}=0.1,0.2,\ldots,0.7 h m i n = 0.1 , 0.2 , … , 0.7 :
H 0 , h min = { s 2 ⋅ I d ∣ s 2 = h min , h min + 0.1 , … , 1.5 } \mathbb{H}_{0,h_{\min}}=\left\{s^2\cdot I_d\mid s^2=h_{\min},h_{\min}+0.1,\ldots,1.5\right\} H 0 , h m i n = { s 2 ⋅ I d ∣ s 2 = h m i n , h m i n + 0.1 , … , 1.5 }
of d × d d\times d d × d scalar bandwidth matrices for D 0 \mathcal{D}_0 D 0 and H 1 , h min = { s 2 ∣ s 2 = h min , h min + 0.1 , … , 1.0 } \mathbb{H}_{1,h_{\min}}=\left\{s^2\mid s^2=h_{\min},h_{\min}+0.1,\ldots,1.0\right\} H 1 , h m i n = { s 2 ∣ s 2 = h m i n , h m i n + 0.1 , … , 1.0 } of positive real numbers for D 1 \mathcal{D}_1 D 1 . For these sets, h min < h min ′ h_{\min}<h'_{\min} h m i n < h m i n ′ implies H i , h min ⊃ H i , h min ′ \mathbb{H}_{i,h_{\min}}\supset\mathbb{H}_{i,h'_{\min}} H i , h m i n ⊃ H i , h m i n ′ . Hence, larger h min h_{\min} h m i n restricts these sets to larger bandwidths. As will be discussed below, such a restriction on bandwidths is necessary for stable estimation with the proposed estimators. To emphasize the dependence on h min h_{\min} h m i n , we write the proposed estimators as μ ^ i γ ( h min , I ) \widehat{\mu}^{\gamma}_i(h_{\min},I) μ i γ ( h m i n , I ) from D i \mathcal{D}_i D i with H i , h min \mathbb{H}_{i,h_{\min}} H i , h m i n . For each h min = 0.1 , 0.2 , … , 0.7 h_{\min}=0.1,0.2,\ldots,0.7 h m i n = 0.1 , 0.2 , … , 0.7 , we compute min I ∈ I R ^ [ μ ^ i γ ( h min , I ) ] \min_{I\in\mathcal{I}}\widehat{\mathcal{R}}\left[\widehat{\mu}^{\gamma}_i(h_{\min},I)\right] min I ∈ I R [ μ i γ ( h m i n , I ) ] and its standard error.
The results are summarized in Table 6.2 , much as in Table 6.1 . We point out the main differences between Table 6.2 and Table 6.1 : for the proposed estimators, the first, second and third smallest values in { min I ∈ I R ^ [ μ ^ i γ ( h min , I ) ] } h min = 0.1 , 0.2 , … , 0.7 \left\{\min_{I\in\mathcal{I}}\widehat{\mathcal{R}}\left[\widehat{\mu}^{\gamma}_i(h_{\min},I)\right]\right\}_{h_{\min}=0.1,0.2,\ldots,0.7} { min I ∈ I R [ μ i γ ( h m i n , I ) ] } h m i n = 0.1 , 0.2 , … , 0.7 are reported in columns 3–5 for μ ^ 0 0 \widehat{\mu}^{0}_0 μ 0 0 ; in columns 6–8 for μ ^ 1 0 \widehat{\mu}^{0}_1 μ 1 0 ; in columns 9–11 for μ ^ 0 1 \widehat{\mu}^{1}_0 μ 0 1 ; and in columns 12–14 for μ ^ 1 1 \widehat{\mu}^{1}_1 μ 1 1 . Below each risk, the corresponding h min h_{\min} h m i n is also given together with I I I minimizing it: for example, the entries in rows 2–3 and columns 3–5 indicate that, in ( μ , n ) = ( p 1 , 50 ) (\mu,n)=(p_1,50) ( μ , n ) = ( p 1 , 50 ) , min I ∈ I R ^ [ μ ^ 0 0 ( 0.5 , I ) ] \min_{I\in\mathcal{I}}\widehat{\mathcal{R}}\left[\widehat{\mu}^{0}_0(0.5,I)\right] min I ∈ I R [ μ 0 0 ( 0.5 , I ) ] , min I ∈ I R ^ [ μ ^ 0 0 ( 0.6 , I ) ] \min_{I\in\mathcal{I}}\widehat{\mathcal{R}}\left[\widehat{\mu}^{0}_0(0.6,I)\right] min I ∈ I R [ μ 0 0 ( 0.6 , I ) ] and min I ∈ I R ^ [ μ ^ 0 0 ( 0.7 , I ) ] \min_{I\in\mathcal{I}}\widehat{\mathcal{R}}\left[\widehat{\mu}^{0}_0(0.7,I)\right] min I ∈ I R [ μ 0 0 ( 0.7 , I ) ] are smaller than min I ∈ I R ^ [ μ ^ 0 0 ( h min , I ) ] \min_{I\in\mathcal{I}}\widehat{\mathcal{R}}\left[\widehat{\mu}^{0}_0(h_{\min},I)\right] min I ∈ I R [ μ 0 0 ( h m i n , I ) ] with the other h min h_{\min} h m i n . Columns 15–19 report the results of the estimators other than μ ^ i γ \widehat{\mu}^{\gamma}_i μ i γ . For μ ^ X \widehat{\mu}_{X} μ X in column 18, m a x d e p t h = m \mathit{maxdepth}=m maxdepth = m , l a m b d a = λ \mathit{lambda}=\lambda lambda = λ and n r o u n d s = I \mathit{nrounds}=I nrounds = I minimizing the risk are given as in [ m , λ , I ] [m,\lambda,I] [ m , λ , I ] .
Table 6.2. The minima of estimated divergence risks (× 10 1 \times10^{1} × 1 0 1 for λ 2 \lambda_{2} λ 2 ; × 10 3 \times10^{3} × 1 0 3 for others) in low-dimensional Bernoulli and Poisson regression.
Recognitions from Table 6.2 are as follows:
6.g. By D i , h min \mathcal{D}_{i,h_{\min}} D i , h m i n , we denote the dictionary D i \mathcal{D}_i D i with the set of bandwidths H i , h min \mathbb{H}_{i,h_{\min}} H i , h m i n . In Section 6.1.1 on normal regression, we used μ ^ i γ \widehat{\mu}^{\gamma}_i μ i γ constructed only from D i , 0.1 \mathcal{D}_{i,0.1} D i , 0.1 with h min = 0.1 h_{\min}=0.1 h m i n = 0.1 , and such
μ ^ i γ \widehat{\mu}_i^\gamma μ i γ outperforms the other estimators in almost all simulated cases. On the other hand, in Bernoulli and Poisson regression, μ ^ i γ \widehat{\mu}_i^\gamma μ i γ from D i , 0.1 \mathcal{D}_{i,0.1} D i , 0.1 does not always have the best accuracy among the compared estimators, and the restriction for the bandwidth set improves the accuracy of μ ^ i γ \widehat{\mu}_i^\gamma μ i γ in some simulated cases: for example, focusing on the case ( μ , n ) = ( λ 1 , 200 ) (\mu,n)=(\lambda_1,200) ( μ , n ) = ( λ 1 , 200 ) and the estimator μ ^ 0 1 \widehat{\mu}_0^1 μ 0 1 , we can see from Table 6.2 that
min I ∈ I R ^ [ μ ^ 0 1 ( 0.7 , I ) ] = 21.6 ± 0.4 < 23 ± 0.4 = min H ∈ H L R ^ [ μ ^ L ( H ) ] (19) \min_{I\in\mathbb{I}}\widehat{R}\left[\widehat{\mu}_0^1(0.7,I)\right]=21.6\pm0.4<23\pm0.4=\min_{H\in\mathbb{H}_{L}}\widehat{R}\left[\widehat{\mu}_{L}(H)\right] \tag*{(19)} I ∈ I min R [ μ 0 1 ( 0.7 , I ) ] = 21.6 ± 0.4 < 23 ± 0.4 = H ∈ H L min R [ μ L ( H ) ] (19)
for μ ^ 0 1 \widehat{\mu}_0^1 μ 0 1 with D i , 0.7 \mathcal{D}_{i,0.7} D i , 0.7 , while
min I ∈ I R ^ [ μ ^ 0 1 ( h min , I ) ] > min H ∈ H L R ^ [ μ ^ L ( H ) ] (20) \min_{I\in\mathbb{I}}\widehat{R}\left[\widehat{\mu}_0^1(h_{\min},I)\right]>\min_{H\in\mathbb{H}_{L}}\widehat{R}\left[\widehat{\mu}_{L}(H)\right] \tag*{(20)} I ∈ I min R [ μ 0 1 ( h m i n , I ) ] > H ∈ H L min R [ μ L ( H ) ] (20)
for μ ^ 0 1 \widehat{\mu}_0^1 μ 0 1 from D i , h min \mathcal{D}_{i,h_{\min}} D i , h m i n with h min ≤ 0.5 h_{\min}\leq0.5 h m i n ≤ 0.5 .
We describe this in further detail. First, h min > 0.1 h_{\min}>0.1 h m i n > 0.1 implies D i , 0.1 ⊃ D i , h min \mathcal{D}_{i,0.1}\supset\mathcal{D}_{i,h_{\min}} D i , 0.1 ⊃ D i , h m i n , so that D i , 0.1 \mathcal{D}_{i,0.1} D i , 0.1 is a thicker dictionary than D i , h min \mathcal{D}_{i,h_{\min}} D i , h m i n . However, the above results (6.4) and (6.5) mean that the estimated “divergence risk” does not become smaller, even if the “empirical risk” in constructing μ ^ i γ \widehat{\mu}_i^\gamma μ i γ becomes smaller by using a thicker dictionary. More specifically, there is a difference between construction of μ ^ i γ ( 0.1 , I ) \widehat{\mu}_i^\gamma(0.1,I) μ i γ ( 0.1 , I ) and μ ^ i γ ( h min , I ) \widehat{\mu}_i^\gamma(h_{\min},I) μ i γ ( h m i n , I ) , in that μ ^ i γ ( 0.1 , I ) \widehat{\mu}_i^\gamma(0.1,I) μ i γ ( 0.1 , I ) may select words B ♯ B_{\sharp} B ♯ such that B ♯ ∉ D i , h min B_{\sharp}\notin\mathcal{D}_{i,h_{\min}} B ♯ ∈ / D i , h m i n but B ♯ ∈ D i , 0.1 B_{\sharp}\in\mathcal{D}_{i,0.1} B ♯ ∈ D i , 0.1 . In ⟨ S t e p 2 ⟩ \langle Step\ 2\rangle ⟨ S t e p 2 ⟩ of the algorithm in § 3 for μ ^ i γ \widehat{\mu}_i^\gamma μ i γ , the following inequality for the “empirical risk”
inf B ˉ ∈ D i , 0.1 L ^ ( u − 1 ( T ( θ ˉ , C ( π k ∣ B ˉ , ξ k − 1 ) ) ) ) ≤ inf B ˉ ∈ D i , h min L ^ ( u − 1 ( T ( θ ˉ , C ( π k ∣ B ˉ , ξ k − 1 ) ) ) ) , \inf_{\bar{B}\in\mathcal{D}_{i,0.1}}\widehat{L}\left(u^{-1}\left(T\left(\bar{\theta},C\left(\pi_k\mid\bar{B},\xi_{k-1}\right)\right)\right)\right)\leq\inf_{\bar{B}\in\mathcal{D}_{i,h_{\min}}}\widehat{L}\left(u^{-1}\left(T\left(\bar{\theta},C\left(\pi_k\mid\bar{B},\xi_{k-1}\right)\right)\right)\right), B ˉ ∈ D i , 0.1 inf L ( u − 1 ( T ( θ ˉ , C ( π k ∣ B ˉ , ξ k − 1 ) ) ) ) ≤ B ˉ ∈ D i , h m i n inf L ( u − 1 ( T ( θ ˉ , C ( π k ∣ B ˉ , ξ k − 1 ) ) ) ) ,
holds for k = 1 , … , I k=1,\ldots,I k = 1 , … , I . However, if the above B ♯ B_{\sharp} B ♯ is selected at some iteration k k k in ⟨ S t e p 2 ⟩ \langle Step\ 2\rangle ⟨ S t e p 2 ⟩ , the resulting forms of μ ^ i γ ( 0.1 , I ) \widehat{\mu}_i^\gamma(0.1,I) μ i γ ( 0.1 , I ) and μ ^ i γ ( h min , I ) \widehat{\mu}_i^\gamma(h_{\min},I) μ i γ ( h m i n , I ) differ essentially, and the inequality of the “divergence risk”
R ^ [ μ ^ i γ ( 0.1 , I ) ] < R ^ [ μ ^ i γ ( h min , I ) ] \widehat{R}\left[\widehat{\mu}_i^\gamma(0.1,I)\right]<\widehat{R}\left[\widehat{\mu}_i^\gamma(h_{\min},I)\right] R [ μ i γ ( 0.1 , I ) ] < R [ μ i γ ( h m i n , I ) ]
does not always hold. That is, the divergence risk is not always reduced with a thicker dictionary.
From the above observations, when constructing the proposed estimators in Bernoulli and Poisson regression, it may be preferable to utilize D 0 \mathcal{D}_0 D 0 and D 1 \mathcal{D}_1 D 1 with larger bandwidths than those in normal regression.
6.h. According to columns 3–14 of Table 6.2 , in almost all cases except ( μ , n ) = ( λ 1 , 50 ) (\mu,n)=(\lambda_1,50) ( μ , n ) = ( λ 1 , 50 ) , the risks indicated by the solid lines are given by μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ with some γ , i \gamma,i γ , i . On the other hand, there are also some γ , i \gamma,i γ , i for which μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ does not have a smaller risk than other estimators:
6.h.1. μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ vs μ ^ P \hat{\mu}_{P} μ ^ P . The proposed estimators μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 , μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 in ( μ , n ) = ( λ 1 , 50 ) (\mu,n)=(\lambda_1,50) ( μ , n ) = ( λ 1 , 50 ) , and μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 , μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 , μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 in ( μ , n ) = ( p 2 , 150 ) (\mu,n)=(p_2,150) ( μ , n ) = ( p 2 , 150 ) are not more accurate than μ ^ P \hat{\mu}_{P} μ ^ P .
6.h.2. μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 vs μ ^ S 0 \hat{\mu}_S^0 μ ^ S 0 . In ( μ , n ) = ( λ 1 , 50 ) (\mu,n)=(\lambda_1,50) ( μ , n ) = ( λ 1 , 50 ) , ( p 2 , 150 ) (p_2,150) ( p 2 , 150 ) and ( λ 2 , 50 ) (\lambda_2,50) ( λ 2 , 50 ) , the risks of μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 are not smaller than those of μ ^ S 0 \hat{\mu}_S^0 μ ^ S 0 . On the other hand, the risks of μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 are smaller than those of μ ^ S 0 \hat{\mu}_S^0 μ ^ S 0 in ( μ , n ) = ( p 1 , 50 ) (\mu,n)=(p_1,50) ( μ , n ) = ( p 1 , 50 ) , ( p 1 , 200 ) (p_1,200) ( p 1 , 200 ) and ( λ 1 , 200 ) (\lambda_1,200) ( λ 1 , 200 ) .
6.h.3. μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 vs μ ^ S 1 \hat{\mu}_S^1 μ ^ S 1 . In ( μ , n ) = ( λ 1 , 50 ) (\mu,n)=(\lambda_1,50) ( μ , n ) = ( λ 1 , 50 ) , ( p 2 , 150 ) (p_2,150) ( p 2 , 150 ) and ( λ 2 , 50 ) (\lambda_2,50) ( λ 2 , 50 ) , μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 does not have smaller risks than μ ^ S 1 \hat{\mu}_S^1 μ ^ S 1 . On the other hand, μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 has smaller risks than μ ^ S 1 \hat{\mu}_S^1 μ ^ S 1 in ( μ , n ) = ( p 1 , 50 ) (\mu,n)=(p_1,50) ( μ , n ) = ( p 1 , 50 ) , ( p 1 , 200 ) (p_1,200) ( p 1 , 200 ) and ( λ 1 , 200 ) (\lambda_1,200) ( λ 1 , 200 ) .
6.h.4. μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ vs μ ^ X , μ ^ L \hat{\mu}_X,\hat{\mu}_L μ ^ X , μ ^ L . In ( μ , n ) = ( λ 1 , 50 ) (\mu,n)=(\lambda_1,50) ( μ , n ) = ( λ 1 , 50 ) , μ ^ X \hat{\mu}_X μ ^ X and μ ^ L \hat{\mu}_L μ ^ L have the best and the second best accuracy among the compared estimators. On the other hand, in almost all cases except ( μ , n ) = ( λ 1 , 50 ) (\mu,n)=(\lambda_1,50) ( μ , n ) = ( λ 1 , 50 ) , μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ with some γ , i \gamma,i γ , i has higher accuracy than μ ^ X \hat{\mu}_X μ ^ X and μ ^ L \hat{\mu}_L μ ^ L .
When ( μ , n ) = ( p 1 , 50 ) (\mu,n)=(p_1,50) ( μ , n ) = ( p 1 , 50 ) , ( p 1 , 200 ) (p_1,200) ( p 1 , 200 ) , ( λ 1 , 50 ) (\lambda_1,50) ( λ 1 , 50 ) and ( λ 2 , 200 ) (\lambda_2,200) ( λ 2 , 200 ) , Figure 6.3 and Figure 6.4 display risk curves for each estimator in exactly the same manner as in Figure 6.1 and Figure 6.2 . In particular, for the proposed estimators, Figure 6.3 and Figure 6.4 present the results of R ^ [ μ ^ i γ ( h min ∗ ( γ , i ) , I ) ] \widehat{R}[\hat{\mu}_i^\gamma(h_{\min}^*(\gamma,i),I)] R [ μ ^ i γ ( h m i n ∗ ( γ , i ) , I )] with h min ∗ ( γ , i ) = argmin { min I ∈ I R ^ [ μ ^ i γ ( h min , I ) ] ∣ h min = 0.1 , 0.2 , … , 0.7 } h_{\min}^*(\gamma,i)=\operatorname*{argmin}\{\min_{I\in\mathcal{I}}\widehat{R}[\hat{\mu}_i^\gamma(h_{\min},I)]\mid h_{\min}=0.1,0.2,\ldots,0.7\} h m i n ∗ ( γ , i ) = argmin { min I ∈ I R [ μ ^ i γ ( h m i n , I )] ∣ h m i n = 0.1 , 0.2 , … , 0.7 } : that is, R ^ [ μ ^ i γ ( h min , I ) ] \widehat{R}[\hat{\mu}_i^\gamma(h_{\min},I)] R [ μ ^ i γ ( h m i n , I )] only with the h min h_{\min} h m i n that yields the best accuracy among min I ∈ I R ^ [ μ ^ i γ ( h min , I ) ] \min_{I\in\mathcal{I}}\widehat{R}[\hat{\mu}_i^\gamma(h_{\min},I)] min I ∈ I R [ μ ^ i γ ( h m i n , I )] for the simulated h min h_{\min} h m i n .
Figure 6.3. The curves of estimated divergence risks for each estimator against the number of iterations for the proposed estimators μ ^ i \hat{\mu}_i μ ^ i , when μ = p 1 \mu= p_1 μ = p 1 in Bernoulli regression. (a) n = 50 n = 50 n = 50 ; (b) n = 200 n = 200 n = 200 . ○: μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 , ●: μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 , □: μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 , ■: μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 , ×: μ ^ P \hat{\mu}_P μ ^ P , ⊗: μ ^ S 0 \hat{\mu}_S^0 μ ^ S 0 , ⊠: μ ^ S 1 \hat{\mu}_S^1 μ ^ S 1 , ▽: μ ^ X \hat{\mu}_X μ ^ X and △: μ ^ L \hat{\mu}_L μ ^ L .
Figure 6.4. The curves of estimated divergence risks for each estimator against the number of iterations for the proposed estimators μ ^ i \hat{\mu}_i μ ^ i , when μ = λ 1 \mu= \lambda_1 μ = λ 1 in Poisson regression. (a) n = 50 n = 50 n = 50 ; (b) n = 200 n = 200 n = 200 . ○: μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 , ●: μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 , □: μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 , ■: μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 , ×: μ ^ P \hat{\mu}_P μ ^ P , ⊗: μ ^ S 0 \hat{\mu}_S^0 μ ^ S 0 , ⊠: μ ^ S 1 \hat{\mu}_S^1 μ ^ S 1 , ▽: μ ^ X \hat{\mu}_X μ ^ X and △: μ ^ L \hat{\mu}_L μ ^ L .
6.i. Figure 6.3 .
6.i.1. It can be seen that panel (a) contains only three rather than four risk curves for μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ . This is because the curves for μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 and μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 overlap for almost all I I I .
6.i.2. In panel (a), in ( μ , n ) = ( p 1 , 50 ) (\mu,n)=(p_1,50) ( μ , n ) = ( p 1 , 50 ) , the risk of each μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ initially takes larger values than that of μ ^ P \hat{\mu}_P μ ^ P for sufficiently small I I I , and then decreases with I I I . In particular, the risk
of μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 is smaller than that of μ ^ L \hat{\mu}_L μ ^ L before I ≈ 50 I \approx50 I ≈ 50 , and then becomes the smallest among the estimators. Similar behavior can be seen in panel (b) for n = 200 n = 200 n = 200 .
6.i.3. Comparing the curves of μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ between n = 50 n = 50 n = 50 and n = 200 n = 200 n = 200 for each γ , i \gamma, i γ , i , we can see that the risk in n = 200 n = 200 n = 200 is smaller than that in n = 50 n = 50 n = 50 for almost all I I I . This means that the estimation accuracy of the proposed estimators improves as the sample size increases.
6.j. Figure 6.4 .
6.j.1. In panel (a), the curves for μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 and μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 overlap for almost all I I I , and similarly the curves for μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 and μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 also overlap. Hence, the proposed estimators do not differ substantially between γ = 0 \gamma= 0 γ = 0 and 1. Much the same behavior is observed in panel (b) for n = 200 n = 200 n = 200 .
6.j.2. As observed in panel (a) for ( μ , n ) = ( λ 1 , 50 ) (\mu, n) = (\lambda_1, 50) ( μ , n ) = ( λ 1 , 50 ) , μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 and μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 do not have smaller risk values than μ ^ P \hat{\mu}_P μ ^ P . On the other hand, both μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 and μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 improve μ ^ P \hat{\mu}_P μ ^ P . However, after μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 and μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 yield much the same risk values as μ ^ S 0 \hat{\mu}_S^0 μ ^ S 0 , μ ^ S 1 \hat{\mu}_S^1 μ ^ S 1 around I ≈ 30 I \approx30 I ≈ 30 , they increase with I I I . In addition, the risks of μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 and μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 are higher than those of μ ^ X \hat{\mu}_X μ ^ X and μ ^ L \hat{\mu}_L μ ^ L for any I I I .
6.j.3. In contrast to the case n = 50 n = 50 n = 50 , panel (b) shows that both μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 and μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 slightly improve μ ^ P \hat{\mu}_P μ ^ P in n = 200 n = 200 n = 200 . Moreover, μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 and μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 have the smallest risk around I ≈ 50 I \approx50 I ≈ 50 among the estimators.
In normal regression, μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ outperforms μ ^ P \hat{\mu}_P μ ^ P and μ ^ S γ \hat{\mu}_S^\gamma μ ^ S γ for each γ , i \gamma, i γ , i . On the other hand, in Bernoulli and Poisson regression, although μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ with some γ , i \gamma, i γ , i underperforms μ ^ P \hat{\mu}_P μ ^ P and μ ^ S γ \hat{\mu}_S^\gamma μ ^ S γ , there are also some γ , i \gamma, i γ , i for which μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ has the best accuracy among the compared estimators. Consequently, as in normal regression, there also seems to be some appropriate combination of γ \gamma γ and dictionary for estimating μ \mu μ in Bernoulli and Poisson regression.
Higher dimension
This subsection addresses higher-dimensional settings in which the target μ \mu μ is an additive function composed of univariate ones. We consider both cases in d = 10 d = 10 d = 10 and in a high-dimension low-sample-size setting n / d = 0.5 n/d = 0.5 n / d = 0.5 . In detail, we put η ( x ) = x 1 2 + x 2 + e x 3 + sin ( π x 4 ) \eta(x) = x_1^2 + x_2 + e^{x_3} + \sin(\pi x_4) η ( x ) = x 1 2 + x 2 + e x 3 + sin ( π x 4 ) , which is defined on R d \mathbb{R}^d R d but depends only on the first four coordinates x 1 , x 2 , x 3 , x 4 x_1, x_2, x_3, x_4 x 1 , x 2 , x 3 , x 4 . We then define μ \mu μ and set ( d , n ) (d,n) ( d , n ) as follows:
μ = μ d = η \mu= \mu_d = \eta μ = μ d = η for ( d , n ) = ( 10 , 50 ) , ( 100 , 50 ) (d,n) = (10,50),(100,50) ( d , n ) = ( 10 , 50 ) , ( 100 , 50 ) in normal regression;
μ = p d = ( 1 + e − η ) − 1 \mu= p_d = (1 + e^{-\eta})^{-1} μ = p d = ( 1 + e − η ) − 1 for ( d , n ) = ( 10 , 150 ) , ( 300 , 150 ) (d,n) = (10,150),(300,150) ( d , n ) = ( 10 , 150 ) , ( 300 , 150 ) in Bernoulli regression;
and μ = λ d = e η \mu= \lambda_d = e^\eta μ = λ d = e η for ( d , n ) = ( 10 , 50 ) , ( 100 , 50 ) (d,n) = (10,50),(100,50) ( d , n ) = ( 10 , 50 ) , ( 100 , 50 ) in Poisson regression.
These target functions can each be expressed as u − 1 ( η ) u^{-1}(\eta) u − 1 ( η ) in the three regression settings.
We estimate this μ \mu μ with the following estimators:
Parametric estimator μ ^ P \widehat{\mu}_{P} μ P . In d = 10 d = 10 d = 10 , we use u − 1 ( [ 1 x T ] θ ^ M ) u^{-1}\left([1\ x^{\mathsf{T}}]\widehat{\theta}_{M}\right) u − 1 ( [ 1 x T ] θ M ) with the MLE θ ^ M \widehat{\theta}_{M} θ M of (6.3). Here we also utilize the LASSO-based parametric regression estimator: u − 1 ( [ 1 x T ] θ ^ L ) u^{-1}\left([1\ x^{\mathsf{T}}]\widehat{\theta}_{L}\right) u − 1 ( [ 1 x T ] θ L ) , where θ ^ L \widehat{\theta}_{L} θ L denotes the LASSO estimator of the parameter θ \theta θ . We write the corresponding parametric estimators as μ ^ P ( θ ^ M ) \widehat{\mu}_{P}(\widehat{\theta}_{M}) μ P ( θ M ) and μ ^ P ( θ ^ L ) \widehat{\mu}_{P}(\widehat{\theta}_{L}) μ P ( θ L ) , respectively. In the case n / d = 0.5 n/d = 0.5 n / d = 0.5 , we use only μ ^ P ( θ ^ L ) \widehat{\mu}_{P}(\widehat{\theta}_{L}) μ P ( θ L ) . We implement θ ^ L \widehat{\theta}_{L} θ L with the package glmnet in R, where the regularization parameter is selected according to the default cross-validation procedure in the function cv.glmnet, see Section “Cross-validation” in Hastie et al. [13 ] . Under this setting, we compute the divergence risks of the resulting parametric estimators.
Proposed semiparametric estimator μ ^ i γ \widehat{\mu}^{\gamma}_{i} μ i γ . We fix i = 1 i = 1 i = 1 and γ = 0 \gamma= 0 γ = 0 . That is, the dictionary D 1 \mathcal{D}_1 D 1 is utilized and the adjustment is additive, so that μ ^ 1 0 \widehat{\mu}^{0}_{1} μ 1 0 is an additive function composed of univariate words. In d = 10 d = 10 d = 10 , we consider μ ^ 1 0 \widehat{\mu}^{0}_{1} μ 1 0 derived from the above two parametric estimators μ ^ P ( θ ^ M ) \widehat{\mu}_{P}(\widehat{\theta}_{M}) μ P ( θ M ) and μ ^ P ( θ ^ L ) \widehat{\mu}_{P}(\widehat{\theta}_{L}) μ P ( θ L ) . We write them as μ ^ 1 0 ( θ ^ M ) = u − 1 ( m ( ⋅ , θ ^ M ) + ξ ^ M ) \widehat{\mu}^{0}_{1}(\widehat{\theta}_{M}) = u^{-1}\left(m(\cdot,\widehat{\theta}_{M}) + \widehat{\xi}_{M}\right) μ 1 0 ( θ M ) = u − 1 ( m ( ⋅ , θ M ) + ξ M ) and μ ^ 1 0 ( θ ^ L ) = u − 1 ( m ( ⋅ , θ ^ L ) + ξ ^ L ) \widehat{\mu}^{0}_{1}(\widehat{\theta}_{L}) = u^{-1}\left(m(\cdot,\widehat{\theta}_{L}) + \widehat{\xi}_{L}\right) μ 1 0 ( θ L ) = u − 1 ( m ( ⋅ , θ L ) + ξ L ) , where ξ ^ M \widehat{\xi}_{M} ξ M and ξ ^ L \widehat{\xi}_{L} ξ L are the adjustment terms resulting from the corresponding parametric estimators. All remaining parameter settings and the calculation of divergence risks are identical to those in § 6.1 .
Besides the estimation accuracy, we investigate practical behavior of the word selection in constructing the adjustment terms of the μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 . Since these ξ ^ M \hat{\xi}_{\mathrm{M}} ξ ^ M and ξ ^ L \hat{\xi}_{\mathrm{L}} ξ ^ L depend on the parametric estimators utilized, we also examine the behavior of θ ^ M \hat{\theta}_{\mathrm{M}} θ ^ M and θ ^ L \hat{\theta}_{\mathrm{L}} θ ^ L . Specifically, in addition to the divergence risk R ^ \hat{R} R ^ , we compute the quantities θ ^ M ∗ \hat{\theta}_{\mathrm{M}}^* θ ^ M ∗ , θ ^ L ∗ \hat{\theta}_{\mathrm{L}}^* θ ^ L ∗ and C s ( θ ^ M ) C_s\left(\hat{\theta}_{\mathrm{M}}\right) C s ( θ ^ M ) , C s ( θ ^ L ) C_s\left(\hat{\theta}_{\mathrm{L}}\right) C s ( θ ^ L ) below, based on the observations ( Y 1 [ r ] , X 1 [ r ] ) , … , ( Y n [ r ] , X n [ r ] ) \left(Y_1^{[r]},X_1^{[r]}\right),\ldots,\left(Y_n^{[r]},X_n^{[r]}\right) ( Y 1 [ r ] , X 1 [ r ] ) , … , ( Y n [ r ] , X n [ r ] ) for r = 1 , … , R r=1,\ldots,R r = 1 , … , R at the beginning of Section 6:
θ ^ M ∗ = 1 R ∑ r = 1 R θ ^ M [ r ] , θ ^ L ∗ = 1 R ∑ r = 1 R θ ^ L [ r ] , \hat{\theta}_{\mathrm{M}}^*=\frac{1}{R}\sum_{r=1}^{R}\hat{\theta}_{\mathrm{M}}^{[r]},\qquad\hat{\theta}_{\mathrm{L}}^*=\frac{1}{R}\sum_{r=1}^{R}\hat{\theta}_{\mathrm{L}}^{[r]}, θ ^ M ∗ = R 1 r = 1 ∑ R θ ^ M [ r ] , θ ^ L ∗ = R 1 r = 1 ∑ R θ ^ L [ r ] ,
where θ ^ M [ r ] \hat{\theta}_{\mathrm{M}}^{[r]} θ ^ M [ r ] and θ ^ L [ r ] \hat{\theta}_{\mathrm{L}}^{[r]} θ ^ L [ r ] are respectively the MLE and the LASSO estimator in the r r r -th repetition of the simulation. For s = 1 , 2 , … , d s=1,2,\ldots,d s = 1 , 2 , … , d ,
C s ( θ ^ M ) = 1 R ∑ r = 1 R [ ∑ k = 1 I I { S M [ r ] ( k ) = s } ] , C s ( θ ^ L ) = 1 R ∑ r = 1 R [ ∑ k = 1 I I { S L [ r ] ( k ) = s } ] , C_s\left(\hat{\theta}_{\mathrm{M}}\right)=\frac{1}{R}\sum_{r=1}^{R}\left[\sum_{k=1}^{I}\mathbb{I}\left\{S_{\mathrm{M}}^{[r]}(k)=s\right\}\right],\qquad C_s\left(\hat{\theta}_{\mathrm{L}}\right)=\frac{1}{R}\sum_{r=1}^{R}\left[\sum_{k=1}^{I}\mathbb{I}\left\{S_{\mathrm{L}}^{[r]}(k)=s\right\}\right], C s ( θ ^ M ) = R 1 r = 1 ∑ R [ k = 1 ∑ I I { S M [ r ] ( k ) = s } ] , C s ( θ ^ L ) = R 1 r = 1 ∑ R [ k = 1 ∑ I I { S L [ r ] ( k ) = s } ] ,
where I \mathbb{I} I is the indicator function, and S M [ r ] ( k ) S_{\mathrm{M}}^{[r]}(k) S M [ r ] ( k ) , S L [ r ] ( k ) S_{\mathrm{L}}^{[r]}(k) S L [ r ] ( k ) are defined as follows: when the number of repetitions for the simulation is r r r , S M [ r ] ( k ) S_{\mathrm{M}}^{[r]}(k) S M [ r ] ( k ) and S L [ r ] ( k ) S_{\mathrm{L}}^{[r]}(k) S L [ r ] ( k ) denote the coordinate indices corresponding to the univariate word selected in the k k k -th iteration of ⟨ S t e p 2 ⟩ \langle\mathit{Step\ 2}\rangle ⟨ Step 2 ⟩ in the algorithm for ξ ^ M \hat{\xi}_{\mathrm{M}} ξ ^ M and ξ ^ L \hat{\xi}_{\mathrm{L}} ξ ^ L , respectively. Thus, C s C_s C s measures the frequency with which words corresponding to the s s s -th explanatory variable x s x_s x s are selected. We also calculate their standard errors as in (6.2):
se ^ [ θ ^ M ] = 1 R 2 ∑ r = 1 R { θ ^ M [ r ] − θ ^ M ∗ } 2 , se ^ [ θ ^ L ] = 1 R 2 ∑ r = 1 R { θ ^ L [ r ] − θ ^ L ∗ } 2 , \widehat{\operatorname{se}}\left[\hat{\theta}_{\mathrm{M}}\right]=\sqrt{\frac{1}{R^2}\sum_{r=1}^{R}\left\{\hat{\theta}_{\mathrm{M}}^{[r]}-\hat{\theta}_{\mathrm{M}}^*\right\}^2},\qquad\widehat{\operatorname{se}}\left[\hat{\theta}_{\mathrm{L}}\right]=\sqrt{\frac{1}{R^2}\sum_{r=1}^{R}\left\{\hat{\theta}_{\mathrm{L}}^{[r]}-\hat{\theta}_{\mathrm{L}}^*\right\}^2}, se [ θ ^ M ] = R 2 1 r = 1 ∑ R { θ ^ M [ r ] − θ ^ M ∗ } 2 , se [ θ ^ L ] = R 2 1 r = 1 ∑ R { θ ^ L [ r ] − θ ^ L ∗ } 2 ,
se ^ [ C s ( θ ^ M ) ] = 1 R 2 ∑ r = 1 R { ∑ k = 1 I I { S M [ r ] ( k ) = s } − C s ( θ ^ M ) } 2 , \widehat{\operatorname{se}}\left[C_s\left(\hat{\theta}_{\mathrm{M}}\right)\right]=\sqrt{\frac{1}{R^2}\sum_{r=1}^{R}\left\{\sum_{k=1}^{I}\mathbb{I}\left\{S_{\mathrm{M}}^{[r]}(k)=s\right\}-C_s\left(\hat{\theta}_{\mathrm{M}}\right)\right\}^2}, se [ C s ( θ ^ M ) ] = R 2 1 r = 1 ∑ R { k = 1 ∑ I I { S M [ r ] ( k ) = s } − C s ( θ ^ M ) } 2 ,
se ^ [ C s ( θ ^ L ) ] = 1 R 2 ∑ r = 1 R { ∑ k = 1 I I { S L [ r ] ( k ) = s } − C s ( θ ^ L ) } 2 . \widehat{\operatorname{se}}\left[C_s\left(\hat{\theta}_{\mathrm{L}}\right)\right]=\sqrt{\frac{1}{R^2}\sum_{r=1}^{R}\left\{\sum_{k=1}^{I}\mathbb{I}\left\{S_{\mathrm{L}}^{[r]}(k)=s\right\}-C_s\left(\hat{\theta}_{\mathrm{L}}\right)\right\}^2}. se [ C s ( θ ^ L ) ] = R 2 1 r = 1 ∑ R { k = 1 ∑ I I { S L [ r ] ( k ) = s } − C s ( θ ^ L ) } 2 .
When comparing the results of θ ^ M \hat{\theta}_{\mathrm{M}} θ ^ M and θ ^ L \hat{\theta}_{\mathrm{L}} θ ^ L , we will refer to
θ ∗ = arg min θ ∈ R d + 1 D U ( μ , u − 1 ( [ 1 ⋅ T ] θ ) ) . (21) \theta^*=\underset{\theta\in\mathbb{R}^{d+1}}{\arg\min}\;D_U\left(\mu,u^{-1}\left(\left[1\ \cdot^{\mathsf{T}}\right]\theta\right)\right). \tag*{(21)} θ ∗ = θ ∈ R d + 1 arg min D U ( μ , u − 1 ( [ 1 ⋅ T ] θ ) ) . (21)
Under the above settings, several components of θ ∗ = [ θ 0 ∗ θ 1 ∗ θ 2 ∗ ⋯ θ d ∗ ] T \theta^*=\left[\theta_0^*\ \theta_1^*\ \theta_2^*\ \cdots\ \theta_d^*\right]^{\mathsf{T}} θ ∗ = [ θ 0 ∗ θ 1 ∗ θ 2 ∗ ⋯ θ d ∗ ] T can be obtained explicitly:
Lemma 6.1. If μ = u − 1 ( η ) \mu=u^{-1}(\eta) μ = u − 1 ( η ) as defined above and X ∼ U ( [ − 1 , 1 ] d ) \boldsymbol{X}\sim U\left([-1,1]^{d}\right) X ∼ U ( [ − 1 , 1 ] d ) , then the following components of θ ∗ \theta^{*} θ ∗ in (21) satisfy
θ 1 ∗ = θ 5 ∗ = θ 6 ∗ = ⋯ = θ d ∗ = 0. \theta^{*}_{1}=\theta^{*}_{5}=\theta^{*}_{6}=\cdots=\theta^{*}_{d}=0. θ 1 ∗ = θ 5 ∗ = θ 6 ∗ = ⋯ = θ d ∗ = 0. If u = id u=\operatorname{id} u = id additionally, then
θ ∗ = [ 1 / 3 + sinh ( 1 ) 0 1 3 / e 3 / π 0 d − 4 ] ≈ [ 1.51 0 1 1.10 0.95 0 d − 4 ] . (22) \theta^{*}=
\begin{bmatrix}
1/3+\sinh(1)\\
0\\
1\\
3/e\\
3/\pi\\
\boldsymbol{0}_{d-4}
\end{bmatrix}
\approx
\begin{bmatrix}
1.51\\
0\\
1\\
1.10\\
0.95\\
\boldsymbol{0}_{d-4}
\end{bmatrix}. \tag*{(22)} θ ∗ = 1/3 + sinh ( 1 ) 0 1 3/ e 3/ π 0 d − 4 ≈ 1.51 0 1 1.10 0.95 0 d − 4 . (22)
Normal regression
The simulation results for normal regression are presented in Table 6.3 , Table 6.4 and Figure 6.5.
Table 6.3. The minima of estimated divergence risks (× 10 3 \times10^{3} × 1 0 3 ) in higher-dimensional normal regression.
Table 6.4. The results for θ ^ M ∗ \widehat{\theta}^{*}_{\mathrm{M}} θ M ∗ , θ ^ L ∗ \widehat{\theta}^{*}_{\mathrm{L}} θ L ∗ , C s ( θ ^ M ) C_s\left(\widehat{\theta}_{\mathrm{M}}\right) C s ( θ M ) and C s ( θ ^ L ) C_s\left(\widehat{\theta}_{\mathrm{L}}\right) C s ( θ L ) in normal regression.
Table 6.3 lists the resulting divergence risks in each d d d , corresponding standard errors and the minimizing parameters in the same manner as in Table 6.1 : in column 2 for μ ^ 1 0 ( θ ^ M ) \hat{\mu}^{0}_{1}\left(\hat{\theta}_{M}\right) μ ^ 1 0 ( θ ^ M ) ; in column 3 for μ ^ P ( θ ^ M ) \hat{\mu}_{P}\left(\hat{\theta}_{M}\right) μ ^ P ( θ ^ M ) ; in column 4 for μ ^ 1 0 ( θ ^ L ) \hat{\mu}^{0}_{1}\left(\hat{\theta}_{L}\right) μ ^ 1 0 ( θ ^ L ) ; in column 5 for μ ^ P ( θ ^ L ) \hat{\mu}_{P}\left(\hat{\theta}_{L}\right) μ ^ P ( θ ^ L ) ; and in column 6 for μ ^ X \hat{\mu}_{X} μ ^ X . Table 6.4 summarizes the estimation results of the parameter θ \theta θ for each d d d and for each component s = 1 , 2 , … , d s=1,2,\ldots,d s = 1 , 2 , … , d : with θ M ∗ \theta^{*}_{M} θ M ∗ in row 1 and with θ L ∗ \theta^{*}_{L} θ L ∗ in row 2. In addition, Table 6.4 shows C s ( θ ^ M ) C_{s}\left(\hat{\theta}_{M}\right) C s ( θ ^ M ) in row 3 and C s ( θ ^ L ) C_{s}\left(\hat{\theta}_{L}\right) C s ( θ ^ L ) in row 4 for each s s s . In particular, column 5 for s = 5 , 6 , … , d s=5,6,\ldots,d s = 5 , 6 , … , d in Table 6.4 displays max s = 5 , 6 , … , d ∣ θ M , s ∗ ∣ \max_{s=5,6,\ldots,d}|\theta^{*}_{M,s}| max s = 5 , 6 , … , d ∣ θ M , s ∗ ∣ , max s = 5 , 6 , … , d ∣ θ L , s ∗ ∣ \max_{s=5,6,\ldots,d}|\theta^{*}_{L,s}| max s = 5 , 6 , … , d ∣ θ L , s ∗ ∣ , min s = 5 , 6 , … , d C s ( θ ^ M ) \min_{s=5,6,\ldots,d}C_{s}\left(\hat{\theta}_{M}\right) min s = 5 , 6 , … , d C s ( θ ^ M ) and max s = 5 , 6 , … , d C s ( θ ^ L ) \max_{s=5,6,\ldots,d}C_{s}\left(\hat{\theta}_{L}\right) max s = 5 , 6 , … , d C s ( θ ^ L ) , where the ‘minimum’ of C s ( θ ^ M ) C_{s}\left(\hat{\theta}_{M}\right) C s ( θ ^ M ) among s = 5 , 6 , … , d s=5,6,\ldots,d s = 5 , 6 , … , d is displayed unlike the other ‘maxima’, for the comparison below. Next to these resulting values in Table 6.4 , their standard errors are given in square brackets. In addition to d = 10 d=10 d = 10 , Table 6.4 summarizes θ ^ L ∗ \widehat{\theta}^{*}_{\mathrm{L}} θ L ∗ and C s ( θ ^ L ) C_s\left(\widehat{\theta}_{\mathrm{L}}\right) C s ( θ L ) in d = 100 d=100 d = 100 . Figure 6.5 depicts the results of both values ∣ θ ^ M s ∗ ∣ \lvert\widehat{\theta}^{*}_{\mathrm{Ms}}\rvert ∣ θ Ms ∗ ∣ and ∣ θ ^ L s ∗ ∣ \lvert\widehat{\theta}^{*}_{\mathrm{Ls}}\rvert ∣ θ Ls ∗ ∣ for s = 5 , 6 , … , d s=5,6,\ldots,d s = 5 , 6 , … , d as the points “●” and “■,” respectively. Figure 6.5 also illustrates the following intervals [ ∣ θ ˉ s ∣ − se ^ [ θ ˉ s ] , ∣ θ ˉ s ∣ + se ^ [ θ ˉ s ] ] \left[\left|\bar{\theta}_s\right|-\widehat{\operatorname{se}}\left[\bar{\theta}_s\right],\left|\bar{\theta}_s\right|+\widehat{\operatorname{se}}\left[\bar{\theta}_s\right]\right] [ θ ˉ s − se [ θ ˉ s ] , θ ˉ s + se [ θ ˉ s ] ] above and below each θ ˉ s = θ ^ M s ∗ , θ ^ L s ∗ \bar{\theta}_s=\widehat{\theta}^{*}_{\mathrm{Ms}},\widehat{\theta}^{*}_{\mathrm{Ls}} θ ˉ s = θ Ms ∗ , θ Ls ∗ for s = 5 , 6 , … , d s=5,6,\ldots,d s = 5 , 6 , … , d .
Figure 6.5. The results of θ ^ M s ∗ \widehat{\theta}^{*}_{\mathrm{Ms}} θ Ms ∗ (●) and θ ^ L s ∗ \widehat{\theta}^{*}_{\mathrm{Ls}} θ Ls ∗ (■) for s = 5 , 6 , … , d s=5,6,\ldots,d s = 5 , 6 , … , d in normal regression with d = 10 d=10 d = 10 .
Findings from Table 6.3 are as follows:
6.k. d = 10 d=10 d = 10 :
6.k.1. Estimation accuracy. As seen in row 1 of Table 6.3 , resulting μ ^ l 0 ( θ ^ L ) \widehat{\mu}^{0}_{\mathrm{l}}\left(\widehat{\theta}_{\mathrm{L}}\right) μ l 0 ( θ L ) and μ ^ l 0 ( θ ^ M ) \widehat{\mu}^{0}_{\mathrm{l}}\left(\widehat{\theta}_{\mathrm{M}}\right) μ l 0 ( θ M ) have respectively the smallest and second smallest risk values among the compared estimators. We further detail the behavior of μ ^ 1 0 ( θ ^ L ) \hat{\mu}_1^0(\hat{\theta}_L) μ ^ 1 0 ( θ ^ L ) and μ ^ 1 0 ( θ ^ M ) \hat{\mu}_1^0(\hat{\theta}_M) μ ^ 1 0 ( θ ^ M ) :
6.k.1.1. θ ^ M 7 ∗ \hat{\theta}_{M7}^* θ ^ M 7 ∗ vs θ ^ L 7 ∗ \hat{\theta}_{L7}^* θ ^ L 7 ∗ . In Figure 6.5 , 0 ∈ [ θ ^ M 7 ∗ − s e ^ [ θ ^ M 7 ∗ ] , θ ^ M 7 ∗ + s e ^ [ θ ^ M 7 ∗ ] ] 0 \in\left[\hat{\theta}_{M7}^*-\widehat{\mathrm{se}}\left[\hat{\theta}_{M7}^*\right],\hat{\theta}_{M7}^*+\widehat{\mathrm{se}}\left[\hat{\theta}_{M7}^*\right]\right] 0 ∈ [ θ ^ M 7 ∗ − se [ θ ^ M 7 ∗ ] , θ ^ M 7 ∗ + se [ θ ^ M 7 ∗ ] ] holds, whereas 0 ∉ [ ∣ θ ^ L 7 ∗ ∣ − s e ^ [ θ ^ L 7 ∗ ] , ∣ θ ^ L 7 ∗ ∣ + s e ^ [ θ ^ L 7 ∗ ] ] 0 \notin\left[\left|\hat{\theta}_{L7}^*\right|-\widehat{\mathrm{se}}\left[\hat{\theta}_{L7}^*\right],\left|\hat{\theta}_{L7}^*\right|+\widehat{\mathrm{se}}\left[\hat{\theta}_{L7}^*\right]\right] 0 ∈ / [ θ ^ L 7 ∗ − se [ θ ^ L 7 ∗ ] , θ ^ L 7 ∗ + se [ θ ^ L 7 ∗ ] ] . However, based on the values indicated by the points and the lengths of the intervals in Figure 6.5 , it is seen that the mean squared error of θ ^ L s ∗ \hat{\theta}_{Ls}^* θ ^ L s ∗ is smaller than that of θ ^ M s ∗ \hat{\theta}_{Ms}^* θ ^ M s ∗ for each s = 5 , 6 , … , 10 s=5,6,\ldots,10 s = 5 , 6 , … , 10 .
This means that θ L ∗ \theta_L^* θ L ∗ estimates the coefficients of the variables x 5 , … , x 10 x_5,\ldots,x_{10} x 5 , … , x 10 to be closer to 0 than θ ^ M ∗ \hat{\theta}_M^* θ ^ M ∗ . Hence, compared with μ ^ P ( θ ^ M ) \hat{\mu}_P\left(\hat{\theta}_M\right) μ ^ P ( θ ^ M ) , μ ^ P ( θ ^ L ) \hat{\mu}_P\left(\hat{\theta}_L\right) μ ^ P ( θ ^ L ) better captures the underlying fact that μ \mu μ does not depend on x 5 , x 6 , … , x 10 x_5,x_6,\ldots,x_{10} x 5 , x 6 , … , x 10 .
6.k.1.2. ξ ^ M \hat{\xi}_M ξ ^ M vs ξ ^ L \hat{\xi}_L ξ ^ L . From the entries in rows 3, 4 and column 8 of Table 6.4 , we can observe that max s = 5 , 6 , … , 10 C s ( θ ^ L ) = 2.7 ± 0.1 < 7.0 ± 0.2 = min s = 5 , 6 , … , d C s ( θ ^ M ) \max_{s=5,6,\ldots,10} C_s\left(\hat{\theta}_L\right)=2.7\pm0.1<7.0\pm0.2=\min_{s=5,6,\ldots,d} C_s\left(\hat{\theta}_M\right) max s = 5 , 6 , … , 10 C s ( θ ^ L ) = 2.7 ± 0.1 < 7.0 ± 0.2 = min s = 5 , 6 , … , d C s ( θ ^ M ) . Therefore, in the adjustment by μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 , ξ ^ L \hat{\xi}_L ξ ^ L selects univariate words for x 5 , x 6 , … , x 10 x_5,x_6,\ldots,x_{10} x 5 , x 6 , … , x 10 fewer times than ξ ^ M \hat{\xi}_M ξ ^ M . One possible factor may be considered from the comparison in the description 6.k.1.1., which indicates that ξ ^ L \hat{\xi}_L ξ ^ L needs to select words for x 5 , x 6 , … , x 10 x_5,x_6,\ldots,x_{10} x 5 , x 6 , … , x 10 less often than ξ ^ M \hat{\xi}_M ξ ^ M .
On the other hand, as seen from rows 3 and 4 of Table 6.4 , ∑ s = 1 4 C s ( θ ^ L ) ≈ 283.8 > 257.1 ≈ ∑ s = 1 4 C s ( θ ^ M ) \sum_{s=1}^{4} C_s\left(\hat{\theta}_L\right)\approx283.8>257.1\approx\sum_{s=1}^{4}C_s\left(\hat{\theta}_M\right) ∑ s = 1 4 C s ( θ ^ L ) ≈ 283.8 > 257.1 ≈ ∑ s = 1 4 C s ( θ ^ M ) , which means that ξ ^ L \hat{\xi}_L ξ ^ L more often selects words for the variables x 1 , x 2 , x 3 , x 4 x_1,x_2,x_3,x_4 x 1 , x 2 , x 3 , x 4 on which the target μ \mu μ essentially depends, compared with ξ ^ M \hat{\xi}_M ξ ^ M .
6.k.2. Correcting variable selection in μ ^ P ( θ ^ L ) \hat{\mu}_P\left(\hat{\theta}_L\right) μ ^ P ( θ ^ L ) . As indicated in Table 6.4 , the resulting estimates with θ ^ L s ∗ \hat{\theta}_{Ls}^* θ ^ L s ∗ are close to 0 when s = 1 s=1 s = 1 , and similarly when s = 5 , 6 , … , 10 s=5,6,\ldots,10 s = 5 , 6 , … , 10 .
On the other hand, the resulting C s ( θ ^ L ) C_s\left(\hat{\theta}_L\right) C s ( θ ^ L ) are not similar between s = 1 s=1 s = 1 and s = 5 , 6 , … , 10 s=5,6,\ldots,10 s = 5 , 6 , … , 10 : in fact, C 1 ( θ ^ L ) C_1\left(\hat{\theta}_L\right) C 1 ( θ ^ L ) takes a much larger value than max s = 5 , 6 , … , 10 C s ( θ ^ L ) \max_{s=5,6,\ldots,10}C_s\left(\hat{\theta}_L\right) max s = 5 , 6 , … , 10 C s ( θ ^ L ) . These smaller C s ( θ ^ L ) C_s\left(\hat{\theta}_L\right) C s ( θ ^ L ) for s = 5 , 6 , … , 10 s=5,6,\ldots,10 s = 5 , 6 , … , 10 are thought to be caused by μ ^ P ( θ ^ L ) \hat{\mu}_P\left(\hat{\theta}_L\right) μ ^ P ( θ ^ L ) , which correctly excludes variables x 5 , x 6 , … , x 10 x_5,x_6,\ldots,x_{10} x 5 , x 6 , … , x 10 from the parametric model as in the description 6.k.1.1., suggesting that these variables are not important for variable selection. However, the result θ ^ L 1 ∗ ≈ 0 \hat{\theta}_{L1}^*\approx0 θ ^ L 1 ∗ ≈ 0 in Table 6.4 means that μ ^ P ( θ ^ L ) \hat{\mu}_P\left(\hat{\theta}_L\right) μ ^ P ( θ ^ L ) also discards the variable x 1 x_1 x 1 , on which the target μ \mu μ actually depends. To correct this, it seems that C 1 ( θ ^ L ) C_1\left(\hat{\theta}_L\right) C 1 ( θ ^ L ) takes a larger value than max s = 5 , 6 , … , 10 C s ( θ ^ L ) \max_{s=5,6,\ldots,10}C_s\left(\hat{\theta}_L\right) max s = 5 , 6 , … , 10 C s ( θ ^ L ) . In this way, it is expected that actual importance of the variables discarded at the stage of μ ^ P ( θ ^ L ) \widehat{\mu}_{P}\left(\widehat{\theta}_{L}\right) μ P ( θ L ) can be evaluated, by considering how many words for such variables are included in the adjustment term of μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{L}\right) μ 1 0 ( θ L ) .
6.1. n / d = 0.5 n/d = 0.5 n / d = 0.5 . In Table 6.3 , the smallest risk is given by μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{L}\right) μ 1 0 ( θ L ) among the compared estimators. Moreover, as shown in rows 5 and 6 of Table 6.4 , θ ^ L s ∗ ≈ 0 \widehat{\theta}_{Ls}^{*} \approx0 θ L s ∗ ≈ 0 when s = 1 s = 1 s = 1 and when s = 5 , 6 , … , 100 s = 5,6,\ldots,100 s = 5 , 6 , … , 100 , whereas max s = 5 , 6 , … , 100 C s ( θ ^ L ) < C 1 ( θ ^ L ) \max_{s=5,6,\ldots,100} C_{s}\left(\widehat{\theta}_{L}\right) < C_{1}\left(\widehat{\theta}_{L}\right) max s = 5 , 6 , … , 100 C s ( θ L ) < C 1 ( θ L ) . Therefore, we can draw the same conclusion as in the description 6.k.2. in d = 10 d = 10 d = 10 : it is expected that actual importance of the variables discarded at the stage of μ ^ P ( θ ^ L ) \widehat{\mu}_{P}\left(\widehat{\theta}_{L}\right) μ P ( θ L ) can be evaluated by considering how many words for such variables are included in the adjustment term of μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{L}\right) μ 1 0 ( θ L ) . In this way, even in the high-dimension low-sample-size setting here, μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{L}\right) μ 1 0 ( θ L ) estimates μ \mu μ better than the other estimators compared, and also corrects variable selection in μ ^ P ( θ ^ L ) \widehat{\mu}_{P}\left(\widehat{\theta}_{L}\right) μ P ( θ L ) .
Besides Table 6.3 , Figure 6.6 displays the curves of the estimated divergence risks for each estimator against the number of iterations I I I for μ ^ 1 0 \widehat{\mu}_{1}^{0} μ 1 0 , when μ = μ 10 \mu= \mu_{10} μ = μ 10 . The risk curves for μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{M}\right) μ 1 0 ( θ M ) and μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{L}\right) μ 1 0 ( θ L ) are plotted with the points “●” and “■,” respectively. On the other hand, the horizontal dotted lines are the results for the estimators other than μ ^ i ? \widehat{\mu}_{i}^{?} μ i ? ; at both ends of these lines, the points “×” indicate the result for μ ^ P ( θ ^ M ) \widehat{\mu}_{P}\left(\widehat{\theta}_{M}\right) μ P ( θ M ) ; “⊠” for μ ^ P ( θ ^ L ) \widehat{\mu}_{P}\left(\widehat{\theta}_{L}\right) μ P ( θ L ) ; and “▽” for μ ^ X \widehat{\mu}_{X} μ X .
6.m. Figure 6.6.
6.m.1. Both resulting risk curves for μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{M}\right) μ 1 0 ( θ M ) (●) and μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{L}\right) μ 1 0 ( θ L ) (■) decrease with I I I .
6.m.2. Focusing on the parametric estimation, the estimated divergence risk for μ ^ P ( θ ^ L ) \widehat{\mu}_{P}\left(\widehat{\theta}_{L}\right) μ P ( θ L ) (⊠) is smaller than that for μ ^ P ( θ ^ M ) \widehat{\mu}_{P}\left(\widehat{\theta}_{M}\right) μ P ( θ M ) (×). However, after adjusting them, μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{L}\right) μ 1 0 ( θ L ) has smaller risk values than μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{M}\right) μ 1 0 ( θ M ) for almost all I I I . One possible reason for this may be the behavior of θ ^ L \widehat{\theta}_{L} θ L and θ ^ M \widehat{\theta}_{M} θ M as in the description 6.k.1.1.
Bernoulli and Poisson regression
As in § 6.1.2 , when constructing μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{M}\right) μ 1 0 ( θ M ) and μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{L}\right) μ 1 0 ( θ L ) , we apply several sets of bandwidths: H 1 , h min = { s 2 ∣ s 2 = h min , h min + 0.1 , … , 1.0 } \mathbb{H}_{1,h_{\min}} = \{s^{2} \mid s^{2} = h_{\min}, h_{\min} + 0.1,\ldots,1.0\} H 1 , h m i n = { s 2 ∣ s 2 = h m i n , h m i n + 0.1 , … , 1.0 } for h min = 0.1 , 0.2 , … , 0.7 h_{\min} = 0.1,0.2,\ldots,0.7 h m i n = 0.1 , 0.2 , … , 0.7 .
Figure 6. The curves of estimated divergence risks for each estimator against the number of iterations for the proposed estimators μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ , when μ = μ 10 \mu= \mu_{10} μ = μ 10 in normal regression. ●: μ ^ 1 0 ( θ ^ M ) \hat{\mu}_1^0(\hat{\theta}_{M}) μ ^ 1 0 ( θ ^ M ) , ■: μ ^ 1 0 ( θ ^ L ) \hat{\mu}_1^0(\hat{\theta}_{L}) μ ^ 1 0 ( θ ^ L ) , ×: μ ^ P ( θ ^ M ) \hat{\mu}_{P}(\hat{\theta}_{M}) μ ^ P ( θ ^ M ) , ⊠: μ ^ P ( θ ^ L ) \hat{\mu}_{P}(\hat{\theta}_{L}) μ ^ P ( θ ^ L ) and ▽: μ ^ X \hat{\mu}_{X} μ ^ X .
Table 6.5 summarizes the resulting minima of divergence risks in a similar manner to that in Table 6.3 . Table 6.5 includes the results for μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 derived from specific three h min h_{\min} h m i n , just as in Table 6.2 of § 6.1.2 . In addition to Table 6.5 , Figure 6.9 and Figure 6.10 present resulting risk curves of μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 in much the same manner as in Figure 6.6, when μ = p 10 \mu= p_{10} μ = p 10 and λ 10 \lambda_{10} λ 10 , respectively. These Figure 6.9 and Figure 6.10 contain the curves for μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 only derived from one h min h_{\min} h m i n chosen just as in Figure 6.3 and Figure 6.4 of § 6.1.2 . Moreover, Figure 6.7 and Figure 6.8 depict the resulting θ ^ \hat{\theta} θ ^ in exactly the same way as in Figure 6.5 , and Table 6.6 lists the resulting θ ^ \hat{\theta} θ ^ and C s C_s C s similarly to Table 6.4 , where the result of μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 is presented only for the same h min h_{\min} h m i n as in Figure 6.9 and Figure 6.10 .
Table 6.5. The minima of estimated divergence risks (× 10 3 \times10^3 × 1 0 3 ) for higher-dimensional Bernoulli and Poisson regression.
Table 6.6. The results for θ ^ M ∗ \widehat{\theta}_{M}^{*} θ M ∗ , θ ^ L ∗ \widehat{\theta}_{L}^{*} θ L ∗ , C s ( θ ^ M ) C_s\left(\widehat{\theta}_{M}\right) C s ( θ M ) and C s ( θ ^ L ) C_s\left(\widehat{\theta}_{L}\right) C s ( θ L ) in Bernoulli and Poisson regression.
Figure 6.7. The results of θ ^ M s ∗ \hat{\theta}_{M_s}^{*} θ ^ M s ∗ (●) and θ ^ L s ∗ \hat{\theta}_{L_s}^{*} θ ^ L s ∗ (■) for s = 5 , 6 , … , d s=5,6,\ldots,d s = 5 , 6 , … , d in Bernoulli regression with d = 10 d=10 d = 10 .
Figure 6.8. The results of θ ^ M s ∗ \hat{\theta}_{\mathrm{M}s}^{*} θ ^ M s ∗ (●) and θ ^ L s ∗ \hat{\theta}_{\mathrm{L}s}^{*} θ ^ L s ∗ (■) for s = 5 , 6 , … , d s=5,6,\ldots,d s = 5 , 6 , … , d in Poisson regression with d = 10 d=10 d = 10 .
Figure 6.9. The curves of estimated divergence risks for each estimator against the number of iterations for the proposed estimators μ ^ i γ \widehat{\mu}_{i}^{\gamma} μ i γ , when μ = p 10 \mu= p_{10} μ = p 10 in Bernoulli regression. ●: μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{M}}\right) μ 1 0 ( θ M ) , ■: μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{L}}\right) μ 1 0 ( θ L ) , ×: μ ^ P 0 ( θ ^ M ) \widehat{\mu}_{\mathrm{P}}^{0}\left(\widehat{\theta}_{\mathrm{M}}\right) μ P 0 ( θ M ) , ⊠: μ ^ P 0 ( θ ^ L ) \widehat{\mu}_{\mathrm{P}}^{0}\left(\widehat{\theta}_{\mathrm{L}}\right) μ P 0 ( θ L ) and ▽: μ ^ X \widehat{\mu}_{\mathrm{X}} μ X .
Figure 6.10. The curves of estimated divergence risks for each estimator against the number of iterations for the proposed estimators μ ^ i γ \hat{\mu}_i^\gamma μ ^ i γ , when μ = λ 10 \mu= \lambda_{10} μ = λ 10 in Poisson regression. ●: μ ^ 1 0 ( θ ^ M ) \hat{\mu}_1^0(\hat{\theta}_M) μ ^ 1 0 ( θ ^ M ) , ■: μ ^ 1 0 ( θ ^ L ) \hat{\mu}_1^0(\hat{\theta}_L) μ ^ 1 0 ( θ ^ L ) , ×: μ ^ P ( θ ^ M ) \hat{\mu}_P(\hat{\theta}_M) μ ^ P ( θ ^ M ) , ⊠: μ ^ P ( θ ^ L ) \hat{\mu}_P(\hat{\theta}_L) μ ^ P ( θ ^ L ) and ▽: μ ^ X \hat{\mu}_X μ ^ X .
d = 10 d = 10 d = 10 :
Estimation accuracy
As seen in Table 6.5 , when both μ = p 10 , λ 10 \mu= p_{10}, \lambda_{10} μ = p 10 , λ 10 , μ ^ 1 0 ( θ ^ L ) \hat{\mu}_1^0(\hat{\theta}_{L}) μ ^ 1 0 ( θ ^ L ) has the smallest risk value among the compared estimators, just as it does for normal regression in § 6.2.1 . Moreover, the behavior of each θ ^ M \hat{\theta}_{M} θ ^ M , θ ^ L \hat{\theta}_{L} θ ^ L , C s ( θ ^ M ) C_s(\hat{\theta}_{M}) C s ( θ ^ M ) and
C s ( θ ^ L ) C_s\left(\hat{\theta}_{L}\right) C s ( θ ^ L ) here is also similar to the results in § 6.2.1 : in fact, Figure 6.7 (μ = p 10 \mu=p_{10} μ = p 10 ) and 6.8 (μ = λ 10 \mu=\lambda_{10} μ = λ 10 ) show that μ ^ P ( θ ^ L ) \hat{\mu}_{P}\left(\hat{\theta}_{L}\right) μ ^ P ( θ ^ L ) captures the underlying dependence of μ \mu μ on x 5 , x 6 , … , x 10 x_5,x_6,\ldots,x_{10} x 5 , x 6 , … , x 10 better than μ ^ P ( θ ^ M ) \hat{\mu}_{P}\left(\hat{\theta}_{M}\right) μ ^ P ( θ ^ M ) ; and as observed in column 8 of Table 6.6 , max s = 5 , 6 , … , 10 C s ( θ ^ L ) < min s = 5 , 6 , … , d C s ( θ ^ M ) \max_{s=5,6,\ldots,10}C_s\left(\hat{\theta}_{L}\right)<\min_{s=5,6,\ldots,d}C_s\left(\hat{\theta}_{M}\right) max s = 5 , 6 , … , 10 C s ( θ ^ L ) < min s = 5 , 6 , … , d C s ( θ ^ M ) when μ = p 10 , λ 10 \mu=p_{10},\lambda_{10} μ = p 10 , λ 10 ; as recognized from column 4–7 of Table 6.6 , ∑ s = 1 4 C s ( θ ^ L ) ≈ 231.6 > 182.3 ≈ ∑ s = 1 4 C s ( θ ^ M ) \sum_{s=1}^{4}C_s\left(\hat{\theta}_{L}\right)\approx231.6>182.3\approx\sum_{s=1}^{4}C_s\left(\hat{\theta}_{M}\right) ∑ s = 1 4 C s ( θ ^ L ) ≈ 231.6 > 182.3 ≈ ∑ s = 1 4 C s ( θ ^ M ) when μ = p 10 \mu=p_{10} μ = p 10 and ∑ s = 1 4 C s ( θ ^ L ) ≈ 240.8 > 220.1 ≈ ∑ s = 1 4 C s ( θ ^ M ) \sum_{s=1}^{4}C_s\left(\hat{\theta}_{L}\right)\approx240.8>220.1\approx\sum_{s=1}^{4}C_s\left(\hat{\theta}_{M}\right) ∑ s = 1 4 C s ( θ ^ L ) ≈ 240.8 > 220.1 ≈ ∑ s = 1 4 C s ( θ ^ M ) when μ = λ 10 \mu=\lambda_{10} μ = λ 10 .
On the other hand, focusing on the case μ = p 10 \mu=p_{10} μ = p 10 , we can see from Table 6.5 that μ ^ 1 0 ( θ ^ L ) \hat{\mu}_{1}^{0}\left(\hat{\theta}_{L}\right) μ ^ 1 0 ( θ ^ L ) from the dictionary with H 1 , 0.1 \mathbb{H}_{1,0.1} H 1 , 0.1 yields a larger risk value than that with H 1 , 0.2 \mathbb{H}_{1,0.2} H 1 , 0.2 of larger bandwidths, which is not analogous to the normal regression setting in § 6.2.1 . It is thought that μ ^ 1 0 ( θ ^ L ) \hat{\mu}_{1}^{0}\left(\hat{\theta}_{L}\right) μ ^ 1 0 ( θ ^ L ) in this case behaves similarly to the description 6.g. in § 6.1.2 .
Correcting variable selection in μ ^ P ( θ ^ L ) \hat{\mu}_{P}\left(\hat{\theta}_{L}\right) μ ^ P ( θ ^ L )
As in the description 6.k.2. for normal regression, it is expected also in the Bernoulli and Poisson regression settings here that actual importance of the variables overlooked in the phase of μ ^ P ( θ ^ L ) \hat{\mu}_{P}\left(\hat{\theta}_{L}\right) μ ^ P ( θ ^ L ) can be evaluated, by considering how many words for such variables appear in the adjustment term of μ ^ 1 0 ( θ ^ L ) \hat{\mu}_{1}^{0}\left(\hat{\theta}_{L}\right) μ ^ 1 0 ( θ ^ L ) : in fact, as observed from Figure 6.7 and Figure 6.8 , θ ^ L s ∗ ≈ 0 \hat{\theta}_{L_s}^{*}\approx0 θ ^ L s ∗ ≈ 0 when s = 1 s=1 s = 1 and when s = 5 , 6 , … , 10 s=5,6,\ldots,10 s = 5 , 6 , … , 10 , while max s = 5 , 6 , … , 10 C s ( θ ^ L ) < C 1 ( θ ^ L ) \max_{s=5,6,\ldots,10}C_s\left(\hat{\theta}_{L}\right)<C_1\left(\hat{\theta}_{L}\right) max s = 5 , 6 , … , 10 C s ( θ ^ L ) < C 1 ( θ ^ L ) in Table 6.6 .
On the other hand, compared with the Poisson regression setting, the difference between the resulting values of C 1 ( θ ^ L ) C_{1}\left(\hat{\theta}_{\mathrm{L}}\right) C 1 ( θ ^ L ) and max s = 5 , 6 , … , 10 C s ( θ ^ L ) \max_{s=5,6,\ldots,10}C_{s}\left(\hat{\theta}_{\mathrm{L}}\right) max s = 5 , 6 , … , 10 C s ( θ ^ L ) seems less distinct in the Bernoulli regression setting.
n / d = 0.5 n/d=0.5 n / d = 0.5
First we focus on the case μ = λ 100 \mu=\lambda_{100} μ = λ 100 . Row 8 of Table 6.5 shows that the smallest risk is given by μ ^ 1 0 ( θ ^ L ) \hat{\mu}_{1}^{0}\left(\hat{\theta}_{\mathrm{L}}\right) μ ^ 1 0 ( θ ^ L ) . Moreover, as seen from Table 6.6 , θ ^ L s ∗ ≈ 0 \hat{\theta}_{\mathrm{L}s}^{*}\approx0 θ ^ L s ∗ ≈ 0 when s = 1 s=1 s = 1 and when s = 5 , 6 , … , 100 s=5,6,\ldots,100 s = 5 , 6 , … , 100 , while max s = 5 , 6 , … , 100 C s ( θ ^ L ) < C 1 ( θ ^ L ) \max_{s=5,6,\ldots,100}C_{s}\left(\hat{\theta}_{\mathrm{L}}\right)<C_{1}\left(\hat{\theta}_{\mathrm{L}}\right) max s = 5 , 6 , … , 100 C s ( θ ^ L ) < C 1 ( θ ^ L ) . Therefore, just as in the description 6.1. for normal regression, even in the high-dimension low-sample-size Poisson regression setting here, μ ^ 1 0 ( θ ^ L ) \hat{\mu}_{1}^{0}\left(\hat{\theta}_{\mathrm{L}}\right) μ ^ 1 0 ( θ ^ L ) estimates μ \mu μ better than the other estimators compared, and also corrects variable selection in μ ^ P ( θ ^ L ) \hat{\mu}_{\mathrm{P}}\left(\hat{\theta}_{\mathrm{L}}\right) μ ^ P ( θ ^ L ) .
On the other hand, as shown in the result for μ = p 300 \mu=p_{300} μ = p 300 in Table 6.5 , the risk of μ ^ 1 0 ( θ ^ L ) \hat{\mu}_{1}^{0}\left(\hat{\theta}_{\mathrm{L}}\right) μ ^ 1 0 ( θ ^ L ) is larger than that of μ ^ X \hat{\mu}_{\mathrm{X}} μ ^ X , which is not found in the other cases simulated above. In addition, as observed in Table 6.6 that θ ^ L s ≈ 0 \hat{\theta}_{\mathrm{L}s}\approx0 θ ^ L s ≈ 0 when s = 1 s=1 s = 1 and when s = 5 , 6 , … , 300 s=5,6,\ldots,300 s = 5 , 6 , … , 300 , but C 1 ( θ ^ L ) < max s = 5 , 6 , … , 300 C s ( θ ^ L ) C_{1}\left(\hat{\theta}_{\mathrm{L}}\right)<\max_{s=5,6,\ldots,300}C_{s}\left(\hat{\theta}_{\mathrm{L}}\right) C 1 ( θ ^ L ) < max s = 5 , 6 , … , 300 C s ( θ ^ L ) : indeed, the coefficient of the variable x 1 x_{1} x 1 on which the target μ \mu μ actually and essentially depends is estimated as θ ^ L 1 = − 0.0002 ± 0.0004 \hat{\theta}_{\mathrm{L}1}=-0.0002\pm0.0004 θ ^ L 1 = − 0.0002 ± 0.0004 , suggesting that x 1 x_{1} x 1 is not important for variable selection. However, the inequality C 1 ( θ ^ L ) < max s = 5 , 6 , … , 300 C s ( θ ^ L ) C_{1}\left(\hat{\theta}_{\mathrm{L}}\right)<\max_{s=5,6,\ldots,300}C_{s}\left(\hat{\theta}_{\mathrm{L}}\right) C 1 ( θ ^ L ) < max s = 5 , 6 , … , 300 C s ( θ ^ L ) indicates that the adjustment term in μ ^ 1 0 ( θ ^ L ) \hat{\mu}_{1}^{0}\left(\hat{\theta}_{\mathrm{L}}\right) μ ^ 1 0 ( θ ^ L ) includes fewer words for x 1 x_{1} x 1 . Hence, the adjustment does not seem to work well in this Bernoulli regression setting.
6.p. Figure 6.9:
6.p.1. The risk for μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{M}}\right) μ 1 0 ( θ M ) (●) initially takes larger values than μ ^ P ( θ ^ M ) \widehat{\mu}_{\mathrm{P}}\left(\widehat{\theta}_{\mathrm{M}}\right) μ P ( θ M ) (×) for sufficiently small I I I , and then decreases with I I I . Similar behavior is observed between μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{L}}\right) μ 1 0 ( θ L ) (■) and μ ^ P ( θ ^ L ) \widehat{\mu}_{\mathrm{P}}\left(\widehat{\theta}_{\mathrm{L}}\right) μ P ( θ L ) (⊠).
6.p.2. Analogous results to the description 6.m.2 for normal regression are found even in the Bernoulli regression setting here: the risk for μ ^ P ( θ ^ L ) \widehat{\mu}_{\mathrm{P}}\left(\widehat{\theta}_{\mathrm{L}}\right) μ P ( θ L ) (⊠) is larger than that for μ ^ P ( θ ^ M ) \widehat{\mu}_{\mathrm{P}}\left(\widehat{\theta}_{\mathrm{M}}\right) μ P ( θ M ) (×), while μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{L}}\right) μ 1 0 ( θ L ) has smaller risk values than μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{M}}\right) μ 1 0 ( θ M ) for almost all I I I .
6.q. Figure 6.10: Results much the same as those described in 6.m. are also seen in the Poisson regression setting here: both risk curves for μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{M}}\right) μ 1 0 ( θ M ) and μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{L}}\right) μ 1 0 ( θ L ) decrease for almost all I I I . In addition, μ ^ 1 0 ( θ ^ L ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{L}}\right) μ 1 0 ( θ L ) has smaller risk values than μ ^ 1 0 ( θ ^ M ) \widehat{\mu}_{1}^{0}\left(\widehat{\theta}_{\mathrm{M}}\right) μ 1 0 ( θ M ) for almost all I I I , even though μ ^ P ( θ ^ L ) \widehat{\mu}_{\mathrm{P}}\left(\widehat{\theta}_{\mathrm{L}}\right) μ P ( θ L ) initially underperforms μ ^ P ( θ ^ M ) \widehat{\mu}_{\mathrm{P}}\left(\widehat{\theta}_{\mathrm{M}}\right) μ P ( θ M ) .
Real data applications
In this section, we apply the proposed estimator μ ^ \hat{\mu} μ ^ in § 3 to several data sets to evaluate its practical performance, focusing specifically on predictive performance. Five-fold cross-validation (CV) is conducted to compare several estimators in both normal and Bernoulli regression settings.
More specifically, we randomly divide the data set X = { ( Y 1 , X 1 ) , … , ( Y n , X n ) } \mathcal{X} = \{(Y_1, X_1), \ldots, (Y_n, X_n)\} X = {( Y 1 , X 1 ) , … , ( Y n , X n )} into K = 5 K = 5 K = 5 sets S 1 [ t ] , S 2 [ t ] , … , S K [ t ] S_1^{[t]}, S_2^{[t]}, \ldots, S_K^{[t]} S 1 [ t ] , S 2 [ t ] , … , S K [ t ] of almost equal size, which is repeated T = 20 T = 20 T = 20 times for t = 1 , 2 , … , T t = 1, 2, \ldots, T t = 1 , 2 , … , T . Then we construct an estimator μ ˉ t ( − k ) \bar{\mu}_t^{(-k)} μ ˉ t ( − k ) from the observations outside S k [ t ] S_k^{[t]} S k [ t ] for k = 1 , 2 , … , K k = 1, 2, \ldots, K k = 1 , 2 , … , K , and calculate the following CV error in each regression setting:
CV [ μ ˉ ] = 1 2 ⋅ 1 T ∑ t = 1 T 1 K ∑ k = 1 K 1 ∣ S k [ t ] ∣ ∑ ( Y , X ) ∈ S k [ t ] { Y − μ ˉ t ( − k ) ( X ) } 2 (23) \operatorname{CV}[\bar{\mu}] = \frac{1}{2}\cdot\frac{1}{T}\sum_{t=1}^{T}\frac{1}{K}\sum_{k=1}^{K}\frac{1}{|S_k^{[t]}|}\sum_{(Y,X)\in S_k^{[t]}}\left\{Y-\bar{\mu}_t^{(-k)}(X)\right\}^{2} \tag*{(23)} CV [ μ ˉ ] = 2 1 ⋅ T 1 t = 1 ∑ T K 1 k = 1 ∑ K ∣ S k [ t ] ∣ 1 ( Y , X ) ∈ S k [ t ] ∑ { Y − μ ˉ t ( − k ) ( X ) } 2 (23)
in normal regression;
CV [ μ ˉ ] = 1 T ∑ t = 1 T 1 K ∑ k = 1 K 1 ∣ S k [ t ] ∣ ∑ ( Y , X ) ∈ S k [ t ] [ Y − I { μ ˉ t ( − k ) ( X ) ≥ 0.5 } ] 2 (24) \operatorname{CV}[\bar{\mu}] = \frac{1}{T}\sum_{t=1}^{T}\frac{1}{K}\sum_{k=1}^{K}\frac{1}{\lvert S_k^{[t]}\rvert}\sum_{(Y,\mathbf{X})\in S_k^{[t]}}\left[Y-\mathbb{I}\left\{\bar{\mu}_{t}^{(-k)}(\mathbf{X})\ge0.5\right\}\right]^2 \tag*{(24)} CV [ μ ˉ ] = T 1 t = 1 ∑ T K 1 k = 1 ∑ K ∣ S k [ t ] ∣ 1 ( Y , X ) ∈ S k [ t ] ∑ [ Y − I { μ ˉ t ( − k ) ( X ) ≥ 0.5 } ] 2 (24)
in Bernoulli regression, representing the prediction error in assigning the label 1 to an input X \mathbf{X} X if μ ˉ t ( − k ) ( X ) ≥ 0.5 \bar{\mu}_{t}^{(-k)}(\mathbf{X})\ge0.5 μ ˉ t ( − k ) ( X ) ≥ 0.5 and 0 otherwise. The settings and notation for the estimators in this section are as follows:
Parametric estimator μ ^ P \hat{\mu}_{P} μ ^ P . The parametric model u − 1 ( m ( ⋅ , θ ) ) u^{-1}(m(\cdot,\boldsymbol{\theta})) u − 1 ( m ( ⋅ , θ )) here is exactly the same as that introduced just before § 6.1 , where the hyperplane is used. As the estimator θ ^ \hat{\boldsymbol{\theta}} θ ^ , we use the MLE if n > d n>d n > d and the LASSO estimator otherwise. The LASSO estimator is implemented just as in § 6.2 , by using the package glmnet in R. Under this setting, we compute CV [ μ ^ P ] \operatorname{CV}[\hat{\mu}_{P}] CV [ μ ^ P ] in (7.1) and (7.2).
Proposed semiparametric estimator μ ^ ^ i \hat{\hat{\mu}}_{i} μ ^ ^ i . We apply the same settings (for the link function u u u , adjustment type γ \gamma γ , dictionaries D i \mathcal{D}_{i} D i and sequence π k \pi_{k} π k ) and notation as in § 6 . For each I = 1 , 2 , … , 300 I=1,2,\ldots,300 I = 1 , 2 , … , 300 , we compute CV [ μ ^ ^ i ( I ) ] \operatorname{CV}[\hat{\hat{\mu}}_{i}(I)] CV [ μ ^ ^ i ( I )] in normal regression, and CV [ μ ^ ^ i ( h min , I ) ] \operatorname{CV}[\hat{\hat{\mu}}_{i}(h_{\min},I)] CV [ μ ^ ^ i ( h m i n , I )] in Bernoulli regression.
GBRT μ ^ X \hat{\mu}_{X} μ ^ X via XGBoost. Its implementation settings are identical to those in § 6 . Using the same notation as in § 6 , we compute CV [ μ ^ X ( m , I ) ] \operatorname{CV}[\hat{\mu}_{X}(m,I)] CV [ μ ^ X ( m , I )] for m ∈ { 1 , 2 , … , 6 } = M X m\in\{1,2,\ldots,6\}=\mathcal{M}_{X} m ∈ { 1 , 2 , … , 6 } = M X and { 1 , 2 , … , 20000 } = I X \{1,2,\ldots,20000\}=\mathcal{I}_{X} { 1 , 2 , … , 20000 } = I X in normal regression and CV [ μ ^ X ( m , λ , I ) ] \operatorname{CV}[\hat{\mu}_{X}(m,\lambda,I)] CV [ μ ^ X ( m , λ , I )] for m ∈ M X m\in\mathcal{M}_{X} m ∈ M X , λ ∈ { 0 , 1 , 10 , 100 , 1000 } = Λ X \lambda\in\{0,1,10,100,1000\}=\Lambda_{X} λ ∈ { 0 , 1 , 10 , 100 , 1000 } = Λ X and the searched I ∈ I X I\in\mathcal{I}_{X} I ∈ I X in Bernoulli regression.
Normal regression
Low dimension
The first data set concerns compressive strength of concrete, studied in Yeh [31 ] , and is available at https://archive.ics.uci.edu/dataset/165/concrete+compressive+strength . This data set contains information on eight variables for n = 1030 n=1030 n = 1030 concrete samples, such as their ingredients: cement (k g / m 3 \mathrm{kg}/\mathrm{m}^{3} kg / m 3 ), blast furnace slag (k g / m 3 \mathrm{kg}/\mathrm{m}^{3} kg / m 3 ), fly ash (k g / m 3 \mathrm{kg}/\mathrm{m}^{3} kg / m 3 ), water (k g / m 3 \mathrm{kg}/\mathrm{m}^{3} kg / m 3 ), superplasticizer (k g / m 3 \mathrm{kg}/\mathrm{m}^{3} kg / m 3 ), coarse aggregate (k g / m 3 \mathrm{kg}/\mathrm{m}^{3} kg / m 3 ), fine aggregate (k g / m 3 \mathrm{kg}/\mathrm{m}^{3} kg / m 3 ), age of testing (days). The data set also records the resulting compressive strength (MPa) for each sample. As in Yeh [31 ] , we predict the compressive strength, using the above d = 8 d = 8 d = 8 variables as inputs.
We use the proposed estimators μ ^ i γ \hat{\mu}^{\gamma}_{i} μ ^ i γ with i = 0 , 1 , 2 i = 0, 1, 2 i = 0 , 1 , 2 and γ = 0 , 1 \gamma= 0, 1 γ = 0 , 1 . The resulting curves of the prediction error (7.1) are presented in Figure 7.1 . In this figure, the curve for μ ^ 0 0 \hat{\mu}^{0}_{0} μ ^ 0 0 (respectively μ ^ 1 0 , μ ^ 2 0 \hat{\mu}^{0}_{1}, \hat{\mu}^{0}_{2} μ ^ 1 0 , μ ^ 2 0 ) is drawn with the points “○” (respectively “●,” “⊕”). In contrast, the curve for μ ^ 0 1 \hat{\mu}^{1}_{0} μ ^ 0 1 (respectively μ ^ 1 1 , μ ^ 2 1 \hat{\mu}^{1}_{1}, \hat{\mu}^{1}_{2} μ ^ 1 1 , μ ^ 2 1 ) is depicted with the points “□” (respectively “■,” “⊞”). In addition, the curve with ▼ is the result of μ ^ X ( m ∗ , I ) \hat{\mu}_{X}(m^{*}, I) μ ^ X ( m ∗ , I ) for 1 ≤ I ≤ 300 1 \le I \le300 1 ≤ I ≤ 300 , where m ∗ m^{*} m ∗ achieves min m ∈ M X , I ∈ I X CV [ μ ^ X ( m , I ) ] \min_{m \in\mathcal{M}_{X}, I \in\mathcal{I}_{X}} \operatorname{CV}[\hat{\mu}_{X}(m, I)] min m ∈ M X , I ∈ I X CV [ μ ^ X ( m , I )] . The horizontal dotted line with the points “×” represents the result for μ ^ P \hat{\mu}_{P} μ ^ P .
Figure 7.1. The curves for prediction errors of each estimator for the concrete data set. ○: μ ^ 0 0 \hat{\mu}^{0}_{0} μ ^ 0 0 , ●: μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 , ⊕: μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 , □: μ ^ 0 1 \hat{\mu}^{1}_{0} μ ^ 0 1 , ■: μ ^ 1 1 \hat{\mu}^{1}_{1} μ ^ 1 1 , ⊞: μ ^ 2 1 \hat{\mu}^{1}_{2} μ ^ 2 1 , ×: μ ^ P \hat{\mu}_{P} μ ^ P and ▼: μ ^ X \hat{\mu}_{X} μ ^ X .
Observations from Figure 7.1 are as follows:
The resulting CV errors of μ ^ i γ \hat{\mu}^{\gamma}_{i} μ ^ i γ with any γ , i \gamma, i γ , i are smaller than that of μ ^ P \hat{\mu}_{P} μ ^ P (×) for almost any I I I , indicating that the proposed estimators adjust μ ^ P \hat{\mu}_{P} μ ^ P well.
Among the proposed estimators and μ ^ X \hat{\mu}_{X} μ ^ X (▼), μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 (⊕) has the smallest CV error for any I I I .
7.b. We describe results of applying μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 to the concrete data set. The algorithm of the adjustment selects the following words:
7.b.1. Univariate words are selected 59 times out of 300 total iterations. Among them, words corresponding to the variable “Age” are chosen 55 times.
7.b.2. Bivariate words are selected 241 times, of which the two-dimensional variable including “Age” is the most frequently selected and accounts for 193 times.
7.b.3. Hence, the proposed estimator μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 adjusts the parametric estimator μ ^ P \hat{\mu}_{P} μ ^ P by incorporating many words corresponding to the variable “Age”. This suggests that words related to the variable “Age” partly contribute to predicting the variable “compressive strength”.
7.c. Note that μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 is constructed by optimizing the Bregman divergence, rather than in terms of prediction. However, the above descriptions confirm that the resulting μ ^ 2 0 \hat{\mu}^{0}_{2} μ ^ 2 0 improves the prediction accuracy for μ ^ P \hat{\mu}_{P} μ ^ P .
Higher dimension
The second data set is a riboflavin data set from the package hdi in R. It measures the riboflavin (vitamin B2 _2 2 ) production rate of a bacterium called “Bacillus subtilis” across n = 71 n = 71 n = 71 strains, along with the expression levels of 4088 genes. More details can be found in [3 ] . The objective is to predict the production rate from a much larger number of d = 4088 d = 4088 d = 4088 variables than the sample size n n n , and to select important genes for this prediction.
We use μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 and μ ^ 1 1 \hat{\mu}^{1}_{1} μ ^ 1 1 . The results of the prediction error (7.1) are illustrated in Figure 7.2 , in exactly the same manner as in Figure 7.1 . To confirm the result of variable selection during the nonparametric adjustment, we examine the quantity C s C_s C s for s = 1 , 2 , … , d s = 1, 2, \ldots, d s = 1 , 2 , … , d , which was defined in § 6.2 . This C s C_s C s counts the frequency with which words corresponding to the s s s -th explanatory variable x s x_s x s are selected in the construction.
Figure 7.2. The curves for prediction errors of each estimator for the riboflavin data set. ●: μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 , ■: μ ^ 1 1 \hat{\mu}^{1}_{1} μ ^ 1 1 , ×: μ ^ P \hat{\mu}_{P} μ ^ P and ▼: μ ^ X \hat{\mu}_{X} μ ^ X .
7.d. Findings from Figure 7.2 are as follows:
The resulting CV error of μ ^ 1 1 \hat{\mu}^{1}_{1} μ ^ 1 1 (■) increases drastically at I = 137 , 252 I = 137, 252 I = 137 , 252 and fails to improve μ ^ P \hat{\mu}_{P} μ ^ P for any I I I .
Focusing on I = 137 I = 137 I = 137 , we examine this drastic increase. We now use the notation introduced in (23) . We write the proposed estimator μ ^ 1 1 \hat{\mu}^{1}_{1} μ ^ 1 1 as μ ^ t ( − k ) \hat{\mu}^{(-k)}_{t} μ ^ t ( − k ) for a repetition t t t among the 20 CV runs and for an index k k k determining the subsets S k [ t ] S^{[t]}_{k} S k [ t ] , k = 1 , 2 , … , 5 k = 1, 2, \ldots, 5 k = 1 , 2 , … , 5 .
In a certain repetition t 0 t_{0} t 0 and index k 0 k_{0} k 0 of the 5-fold CV, the word selected at I = 137 I = 137 I = 137 takes an extreme value for an input X ∗ \mathbf{X}_{*} X ∗ in the test data S k 0 [ t 0 ] S^{[t_{0}]}_{k_{0}} S k 0 [ t 0 ] . Because this word is included in the adjustment term, μ ^ t 0 ( − k 0 ) ( X ∗ ) \hat{\mu}^{(-k_{0})}_{t_{0}}(\mathbf{X}_{*}) μ ^ t 0 ( − k 0 ) ( X ∗ ) takes a value far from the test data Y ∗ \mathbf{Y}_{*} Y ∗ associated with X ∗ \mathbf{X}_{*} X ∗ , which causes the drastic increase in the CV error in Figure 7.3 .
Figure 7.3. The curves for the empirical risks of μ ^ 1 \hat{\mu}_1 μ ^ 1 (■) and μ ^ P \hat{\mu}_P μ ^ P (×) for the riboflavin data set.
We also check for any problem in constructing μ ^ t 0 ( − k 0 ) \hat{\mu}^{(-k_{0})}_{t_{0}} μ ^ t 0 ( − k 0 ) via the algorithm in § 3 . Figure 7.3 presents the empirical risk of μ ^ t 0 ( − k 0 ) \hat{\mu}^{(-k_{0})}_{t_{0}} μ ^ t 0 ( − k 0 ) against each iteration I I I . As shown in Figure 7.3 , the empirical risk of μ ^ t 0 ( − k 0 ) \hat{\mu}^{(-k_{0})}_{t_{0}} μ ^ t 0 ( − k 0 ) is smaller than that of μ ^ P \hat{\mu}_{P} μ ^ P and tends to decrease with I I I . That is, μ ^ t 0 ( − k 0 ) \hat{\mu}^{(-k_{0})}_{t_{0}} μ ^ t 0 ( − k 0 ) does not seem to exhibit problematic behavior in its construction.
7.d.2. For this data set, μ ^ P \hat{\mu}_P μ ^ P (×) achieves a smaller CV error than μ ^ X \hat{\mu}_X μ ^ X (▼) for any I I I .
7.d.3. This μ ^ P \hat{\mu}_P μ ^ P is adjusted by the proposed estimator μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 (●), which yields the smallest CV error among the compared estimators after I ≈ 50 I \approx50 I ≈ 50 .
7.e. We apply μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 to the riboflavin data set. We focus on variable selection, by investigating how many elements are contained in following sets of the explanatory variables x s x_s x s , s = 1 , 2 , … , 4088 s = 1, 2, \ldots, 4088 s = 1 , 2 , … , 4088 :
7.e.1. First, we see that ∣ { x s ∣ θ ^ s = 0 and C s = 0 } ∣ = 3901 \left|\left\{x_s \mid\hat{\theta}_s = 0 \text{ and } C_s = 0\right\}\right| = 3901 { x s ∣ θ ^ s = 0 and C s = 0 } = 3901 , where the set on the left-hand side is the intersection of the following two sets: the variables not selected by the LASSO estimator in μ ^ P \hat{\mu}_P μ ^ P , and the variables not selected in the adjustment. In other words, about 95% of the total 4088 variables are selected at neither the parametric nor the nonparametric stage.
7.e.2. The total number of variables selected in the entire procedure of μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 is 187, which, compared with results in Section 5.1.1 of Bühlmann et al. [3 ] , is larger than that in the FWER-controlling procedure (53). On the other hand, this 187 is smaller than that in the FDR-controlling procedure (375) reported therein. These 187 variables might be moderate to yield the prediction performance in the description 7.d.3..
7.e.3. Other results are that ∣ { x s ∣ θ ^ s ≠ 0 } ∣ = 28 \left|\left\{x_s\mid\hat{\theta}_s\ne0\right\}\right|=28 { x s ∣ θ ^ s = 0 } = 28 , whereas ∣ { x s ∣ θ ^ s = 0 and C s ≠ 0 } ∣ = 159 \left|\left\{x_s\mid\hat{\theta}_s=0\text{ and }C_s\ne0\right\}\right|=159 { x s ∣ θ ^ s = 0 and C s = 0 } = 159 .
That is, while the parametric regression selects only 28 variables, about 5 times more ones are newly selected in the nonparametric adjustment.
7.e.3.1. The above set { x s ∣ θ ^ s ≠ 0 } \left\{x_s\mid\hat{\theta}_s\ne0\right\} { x s ∣ θ ^ s = 0 } includes the variables x 624 x_{624} x 624 , x 2564 x_{2564} x 2564 and x 4003 x_{4003} x 4003 , which correspond to the genes labeled ‘LYSC_at’, ‘YOAB_at’ and ‘YXLD_at’, respectively. These three variables were all the selected ones by methods in Sections 3.2.3 and 3.3.1 of Bühlmann et al. [3 ] .
7.e.3.2. In this application, the variables with the top four values of C s C_s C s are x 1479 x_{1479} x 1479 (labeled ‘YCDI_at’, C 1479 = 16 C_{1479}=16 C 1479 = 16 ), x 285 x_{285} x 285 (‘DEGS_at’, C 285 = 11 C_{285}=11 C 285 = 11 ), x 2020 x_{2020} x 2020 (‘YHDK_at’, C 2020 = 9 C_{2020}=9 C 2020 = 9 ) and x 24 x_{24} x 24 (‘ADDA_at’, C 24 = 8 C_{24}=8 C 24 = 8 ). In addition, their coefficients are all estimated as θ ^ s = 0 \hat{\theta}_s=0 θ ^ s = 0 in μ ^ P \hat{\mu}_{P} μ ^ P , so that these four belong to { x s ∣ θ ^ s = 0 and C s ≠ 0 } \left\{x_s\mid\hat{\theta}_s=0\text{ and }C_s\ne0\right\} { x s ∣ θ ^ s = 0 and C s = 0 } of the 159 variables.
7.e.3.3. On the other hand, compared with C 1479 = 16 C_{1479}=16 C 1479 = 16 , more than 80% of the 159 variables in { x s ∣ θ ^ s = 0 and C s ≠ 0 } \left\{x_s\mid\hat{\theta}_s=0\text{ and }C_s\ne0\right\} { x s ∣ θ ^ s = 0 and C s = 0 } have relatively small C s C_s C s : C s = 1 C_s=1 C s = 1 or C s = 2 C_s=2 C s = 2 .
7.e.4. According to the description 7.e.1., some variables are not selected at any stage of μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 , suggesting that they are not important for this application. As described in 7.e.3., there are also variables that are not selected by μ ^ P \hat{\mu}_P μ ^ P but are selected relatively often in the adjustment, such as x 1479 x_{1479} x 1479 (gene ‘YCDI_at’). This result indicates that some variables not selected by μ ^ P \hat{\mu}_P μ ^ P may also be important, as in the simulation results 6.k.2. and 6.l. in § 6.2.1 .
7.f. These are results of applying the estimators to this high-dimension low-sample-size data set. The proposed estimator μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 adjusts μ ^ P \hat{\mu}_P μ ^ P . This adjustment seems to yield its better performance in predicting the riboflavin production rate than the other estimators (the description 7.d.3.). At the same time, it can be claimed that μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 selects some genes potentially important for the prediction, among the huge number of genes that μ ^ P \hat{\mu}_P μ ^ P does not select.
Bernoulli regression
Low dimension
To confirm the predictive performance of μ ^ \widehat{\mu} μ in Bernoulli regression, we use a data set on banknote authentication from https://archive.ics.uci.edu/dataset/267/banknote+authentication. It includes n = 1372 n=1372 n = 1372 observations on five variables: four features extracted from images of banknote-like specimens, and a binary variable indicating whether each specimen is genuine (0) or forged (1). The objective is to predict this binary indicator using these features as d = 4 d=4 d = 4 explanatory variables.
We use the proposed estimators μ ^ i γ \widehat{\mu}_i^\gamma μ i γ with i = 0 , 1 i=0,1 i = 0 , 1 and γ = 0 , 1 \gamma=0,1 γ = 0 , 1 and h min = 0.1 , 0.3 , 0.5 , 0.7 h_{\min}=0.1,0.3,0.5,0.7 h m i n = 0.1 , 0.3 , 0.5 , 0.7 . Resulting curves of the prediction error (24) are depicted in Figure 7.4 in much the same manner as in Figure 7.1 . Among { μ ^ i γ ( 0.1 , I ) , μ ^ i γ ( 0.3 , I ) , μ ^ i γ ( 0.5 , I ) , μ ^ i γ ( 0.7 , I ) } \{\widehat{\mu}_i^\gamma(0.1,I),\widehat{\mu}_i^\gamma(0.3,I),\widehat{\mu}_i^\gamma(0.5,I),\widehat{\mu}_i^\gamma(0.7,I)\} { μ i γ ( 0.1 , I ) , μ i γ ( 0.3 , I ) , μ i γ ( 0.5 , I ) , μ i γ ( 0.7 , I )} , we draw only one curve of μ ^ i γ ( h min ∗ ( γ , i ) , I ) \widehat{\mu}_i^\gamma(h_{\min}^*(\gamma,i),I) μ i γ ( h m i n ∗ ( γ , i ) , I ) for each γ , i \gamma,i γ , i with the following h min ∗ ( γ , i ) h_{\min}^*(\gamma,i) h m i n ∗ ( γ , i ) : first, we compare the CV results among
Figure 7.4. The curves for prediction errors of each estimator for the banknote authentication data set. ∘ \circ ∘ : μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 , ∙ \bullet ∙ : μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 , ⊕ \oplus ⊕ : μ ^ 2 0 \hat{\mu}_2^0 μ ^ 2 0 , □ \square □ : μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 , ■ \blacksquare ■ : μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 , ⊞ \boxplus ⊞ : μ ^ 2 1 \hat{\mu}_2^1 μ ^ 2 1 , × \times × : μ ^ P \hat{\mu}_{P} μ ^ P and ▼ \blacktriangledown ▼ : μ ^ X \hat{\mu}_{X} μ ^ X .
{ CV [ μ ^ i γ ( 0.1 , I ) ] , CV [ μ ^ i γ ( 0.3 , I ) ] , CV [ μ ^ i γ ( 0.5 , I ) ] , CV [ μ ^ i γ ( 0.7 , I ) ] } \{\operatorname{CV}[\widehat{\mu}_i^\gamma(0.1,I)],\operatorname{CV}[\widehat{\mu}_i^\gamma(0.3,I)],\operatorname{CV}[\widehat{\mu}_i^\gamma(0.5,I)],\operatorname{CV}[\widehat{\mu}_i^\gamma(0.7,I)]\} { CV [ μ i γ ( 0.1 , I )] , CV [ μ i γ ( 0.3 , I )] , CV [ μ i γ ( 0.5 , I )] , CV [ μ i γ ( 0.7 , I )]}
for each I I I . Then we define h min ∗ ( γ , i ) h_{\min}^*(\gamma,i) h m i n ∗ ( γ , i ) as the largest h min h_{\min} h m i n that most frequently minimizes CV [ μ ^ i γ ( h min , I ) ] \operatorname{CV}[\widehat{\mu}_i^\gamma(h_{\min},I)] CV [ μ i γ ( h m i n , I )] through 1 ≤ I ≤ 300 1\le I\le300 1 ≤ I ≤ 300 .
In addition, when applying an estimator μ ˉ \bar{\mu} μ ˉ to the data set, we check the following numbers of mislabeled cases:
F p [ μ ˉ ] = ∑ i = 1 n [ Y i − I { μ ˉ ( X i ) ≥ p } ] 2 , (25) F_p[\bar{\mu}]=\sum_{i=1}^{n}[Y_i-\mathbb{I}\{\bar{\mu}(X_i)\ge p\}]^2, \tag*{(25)} F p [ μ ˉ ] = i = 1 ∑ n [ Y i − I { μ ˉ ( X i ) ≥ p } ] 2 , (25)
F P p [ μ ˉ ] = ∑ i ∈ { j ∣ Y j = 0 } [ Y i − I { μ ˉ ( X i ) ≥ p } ] 2 , (26) FP_p[\bar{\mu}]=\sum_{i\in\{j\mid Y_j=0\}}[Y_i-\mathbb{I}\{\bar{\mu}(X_i)\ge p\}]^2, \tag*{(26)} F P p [ μ ˉ ] = i ∈ { j ∣ Y j = 0 } ∑ [ Y i − I { μ ˉ ( X i ) ≥ p } ] 2 , (26)
F N p [ μ ˉ ] = F p [ μ ˉ ] − F P p [ μ ˉ ] , (27) FN_p[\bar{\mu}]=F_p[\bar{\mu}]-FP_p[\bar{\mu}], \tag*{(27)} F N p [ μ ˉ ] = F p [ μ ˉ ] − F P p [ μ ˉ ] , (27)
for thresholds p = 0.1 , 0.3 , 0.5 , 0.7 , 0.9 p=0.1,0.3,0.5,0.7,0.9 p = 0.1 , 0.3 , 0.5 , 0.7 , 0.9 .
7.g. Recognitions from Figure 7.4 are as follows:
7.g.1. The semiparametric estimators μ ^ 0 1 \hat{\mu}_0^1 μ ^ 0 1 (□ \square □ ) and μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 (■ \blacksquare ■ ) with γ = 1 \gamma= 1 γ = 1 have almost the same CV error as μ ^ P \hat{\mu}_{P} μ ^ P for any I I I .
7.g.2. The resulting CV errors of μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 (∘ \circ ∘ ) and μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 (∙ \bullet ∙ ) with γ = 0 \gamma= 0 γ = 0 are smaller than those of the other estimators for any I I I . In particular, μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 yields slightly smaller CV errors than μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 for almost all I I I .
7.h. We now describe results of applying μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 to the banknote authentication data set. The results for mislabeling (25) –(27) of μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 and μ ^ P \hat{\mu}_{P} μ ^ P are summarized in Table 7.1 .
Table 7.1. The results of F p [ μ ^ P ] F_p[\hat{\mu}_P] F p [ μ ^ P ] and F p [ μ ^ 0 0 ] F_p[\hat{\mu}_0^0] F p [ μ ^ 0 0 ] for the banknote authentication data set. For each threshold p p p , rows 2 and 4 present F p [ μ ^ P ] F_p[\hat{\mu}_P] F p [ μ ^ P ] and F p [ μ ^ 0 0 ] F_p[\hat{\mu}_0^0] F p [ μ ^ 0 0 ] , respectively. Below them, F P p \mathrm{FP}_p FP p and F N p \mathrm{FN}_p FN p for each estimator are given in parentheses as ( F P p , F N p ) (\mathrm{FP}_p,\mathrm{FN}_p) ( FP p , FN p ) .
7.h.1. In each column of Table 7.1 , F p [ μ ^ P ] > F p [ μ ^ 0 0 ] F_p[\hat{\mu}_{P}] > F_p[\hat{\mu}_0^0] F p [ μ ^ P ] > F p [ μ ^ 0 0 ] holds for any searched p p p . Hence, μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 fits the data better than μ ^ P \hat{\mu}_{P} μ ^ P . In particular, as observed in row 2, no input is mislabeled by μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 for any p p p , suggesting that μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 fits the data well.
7.h.2. In banknote authentication, if a forged banknote is incorrectly classified as genuine, it may be allowed to enter circulation. Thus it is important for estimators to avoid such false positives. As shown in Table 7.1 , the false positives resulting from μ ^ P \hat{\mu}_P μ ^ P are correctly adjusted by μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 . This may highlight the practical value of using the semiparametric estimator alongside the parametric one.
7.i. The proposed estimator μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 is constructed by minimizing Bregman divergence, rather than in terms of prediction or labeling. Nevertheless, as indicated by the above descriptions, the resulting μ ^ 0 0 \hat{\mu}_0^0 μ ^ 0 0 improves the performance of μ ^ P \hat{\mu}_P μ ^ P well.
Higher dimension
The final data set is a colon cancer data set from the package rda in R. This data set comprises 40 colon tumor samples (labeled 1) and 22 normal colon tissue samples (labeled 0), with 2000 variables of microarray expression for each sample. We predict whether samples are normal or not, by using the much larger number of d = 2000 d = 2000 d = 2000 variables than n = 62 n = 62 n = 62 , and simultaneously select important genes for this prediction.
We use the proposed estimators μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 and μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 and h min = 0.1 , 0.2 , … , 0.7 h_{\min} = 0.1, 0.2, \ldots, 0.7 h m i n = 0.1 , 0.2 , … , 0.7 . Resulting curves of the prediction error (24) are shown in Figure 7.5 , in much the same manner as in Figure 7.4 .
Figure 7.5. The curves for prediction errors of each estimator for the colon cancer data set. ●: μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 , ■: μ ^ 1 1 \hat{\mu}_{1}^{1} μ ^ 1 1 , ×: μ ^ P \hat{\mu}_{P} μ ^ P and ▼: μ ^ X \hat{\mu}_{X} μ ^ X .
7.j. Observations from Figure 7.5 are as follows:
7.j.1. The resulting CV error for μ ^ 1 1 \hat{\mu}_1^1 μ ^ 1 1 is smaller than that for μ ^ P \hat{\mu}_P μ ^ P before I ≈ 100 I \approx100 I ≈ 100 , while these two CV errors are close to each other after I ≈ 100 I \approx100 I ≈ 100 .
7.j.2. The semiparametric estimator μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 has the smallest CV error among those compared for any I I I .
7.k. We apply μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 to the colon cancer data set. The results for mislabeling (7.3)–(7.5) of μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 and μ ^ P \hat{\mu}_{P} μ ^ P are listed in Table 7.2 . It is observed from each column of Table 7.2 that μ ^ 1 0 \hat{\mu}_{1}^{0} μ ^ 1 0 fits the data better than μ ^ P \hat{\mu}_{P} μ ^ P for any threshold p p p .
Table 7.2. The results of F p [ μ ^ P ] F_p[\hat{\mu}_{P}] F p [ μ ^ P ] and F p [ μ ^ 1 0 ] F_p[\hat{\mu}_{1}^{0}] F p [ μ ^ 1 0 ] for the colon cancer data set. For each threshold p p p , rows 2 and 4 present F p [ μ ^ P ] F_p[\hat{\mu}_{P}] F p [ μ ^ P ] and F p [ μ ^ 1 0 ] F_p[\hat{\mu}_{1}^{0}] F p [ μ ^ 1 0 ] , respectively. Below them, F P p \mathrm{FP}_p FP p and F N p \mathrm{FN}_p FN p for each estimator are given in parentheses as ( F P p , F N p ) (\mathrm{FP}_p,\mathrm{FN}_p) ( FP p , FN p ) .
7.1. Next, we focus on variable selection of μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 , by examining the following counts of explanatory variables x s x_s x s , s = 1 , 2 , … , 2000 s = 1, 2, \ldots, 2000 s = 1 , 2 , … , 2000 as in Section 7.1.2:
7.1.1. We observe ∣ { x s ∣ θ ^ s = 0 and C s = 0 } ∣ = 1905 \left|\left\{x_s \mid\hat{\theta}_s = 0 \text{ and } C_s = 0\right\}\right| = 1905 { x s ∣ θ ^ s = 0 and C s = 0 } = 1905 , indicating that approximately 95% of the total 2000 variables are selected in neither the parametric nor the nonparametric phase.
7.1.2. The number of variables selected in the entire procedure of μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 is 95, which is larger than that by PAM (19) in Section 5 of Guo et al. [12 ] and, on the other hand, is smaller than that by SCRDA (143) reported therein. These 95 variables might be moderate to yield the prediction performance as in the description 7.j.2. .
7.1.3. Other results are that ∣ { x s ∣ θ ^ s ≠ 0 } ∣ = 4 \left|\left\{x_s \mid\hat{\theta}_s \ne0\right\}\right| = 4 { x s ∣ θ ^ s = 0 } = 4 , whereas ∣ { x s ∣ θ ^ s = 0 and C s ≠ 0 } ∣ = 91 \left|\left\{x_s \mid\hat{\theta}_s = 0 \text{ and } C_s \ne0\right\}\right| = 91 { x s ∣ θ ^ s = 0 and C s = 0 } = 91 . While the parametric regression selects only 4 variables, the nonparametric adjustment newly selects about twenty times as many.
7.1.3.1. The top four values of C s C_s C s are achieved by the following variables: x 897 x_{897} x 897 (C 897 = 36 C_{897} = 36 C 897 = 36 ), x 1635 x_{1635} x 1635 (C 1635 = 19 C_{1635} = 19 C 1635 = 19 ), x 765 x_{765} x 765 (C 765 = 16 C_{765} = 16 C 765 = 16 ) and x 513 x_{513} x 513 (C 513 = 8 C_{513} = 8 C 513 = 8 ), all of whose coefficients are estimated as θ ^ s = 0 \hat{\theta}_s = 0 θ ^ s = 0 in μ ^ P \hat{\mu}_{P} μ ^ P . Thus, these four are included in { x s ∣ θ ^ s = 0 and C s ≠ 0 } \left\{x_s \mid\hat{\theta}_s = 0 \text{ and } C_s \ne0\right\} { x s ∣ θ ^ s = 0 and C s = 0 } of the 91 variables.
7.1.3.2. On the other hand, compared with C 897 = 36 C_{897} = 36 C 897 = 36 , over 80% out of the 91 variables in { x s ∣ θ ^ s = 0 and C s ≠ 0 } \left\{x_s \mid\hat{\theta}_s = 0 \text{ and } C_s \ne0\right\} { x s ∣ θ ^ s = 0 and C s = 0 } have relatively small C s C_s C s : 1 ≤ C s ≤ 4 1 \le C_s \le4 1 ≤ C s ≤ 4 .
7.1.4. According to the description 7.1.1. , some variables are completely ignored by μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 , which suggests that they may not be important for this application. On the other hand, as described in 7.1.3. , some variables overlooked by μ ^ P \hat{\mu}_{P} μ ^ P are selected relatively often during the adjustment. This result indicates that some variables not selected by μ ^ P \hat{\mu}_{P} μ ^ P might nevertheless play an important role.
7.m. In the simulations for high-dimensional Bernoulli regression of Section 6.2.2 (specifically, the description 6.o. ), the resulting μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 had lower estimation accuracy than the compared estimator. In addition, the simulated C s C_s C s did not seem to differ clearly among the variables { x s ∣ θ ^ s = 0 and C s ≠ 0 } \left\{x_s \mid\hat{\theta}_s = 0 \text{ and } C_s \ne0\right\} { x s ∣ θ ^ s = 0 and C s = 0 } . On the other hand, we obtain some positive results for μ ^ 1 0 \hat{\mu}^{0}_{1} μ ^ 1 0 in the present application: actually, it can be said that μ ^ 1 0 \hat{\mu}_1^0 μ ^ 1 0 predicts the labels for colon samples more effectively than the other estimators (7.j.2. and 7.k.). It can also be claimed that differences in C s C_s C s , or differences in importance, occur among the variables { x s ∣ θ ^ s = 0 and C s ≠ 0 } \left\{x_s \mid\hat{\theta}_s=0 \text{ and } C_s \ne0\right\} { x s ∣ θ ^ s = 0 and C s = 0 } (7.e.3.2. and 7.l.3.2.).
Concluding remarks
We conclude this work with a summary and remarks.
We have examined semiparametric regression estimators that combine a parametric crude guess with its nonparametric adjustment, via the stagewise empirical risk minimization algorithm. They are mainly determined by the following triplet ( m ( ⋅ , θ ^ ) , U , D ) \left(m(\cdot,\hat{\theta}), U, D\right) ( m ( ⋅ , θ ^ ) , U , D ) (§ 3 ):
A parametric regression estimator m ( ⋅ , θ ^ ) m(\cdot,\hat{\theta}) m ( ⋅ , θ ^ ) with a model function m ( ⋅ , θ ) m(\cdot,\theta) m ( ⋅ , θ ) and an estimator θ ^ \hat{\theta} θ ^ for the finite-dimensional parameter θ \theta θ . This was used as the parametric crude guess, in which the model was allowed to be misspecified.
A strictly convex function U U U associated with the Bregman divergence, and a dictionary D D D . These were used in the stagewise empirical risk minimization algorithm to adjust the initial m ( ⋅ , θ ^ ) m(\cdot,\hat{\theta}) m ( ⋅ , θ ^ ) nonparametrically. Different choices of U U U and D D D yielded various forms of the proposed estimators.
In the theoretical study, we derived a non-asymptotic error bound for the proposed estimators (Theorem 4.1 in § 4 ). According to Theorem 4.1, the divergence risk for the proposed estimators can be bounded above only by the sum of a bias term and asymptotically vanishing terms.
Although previous studies by Klemelä [17 ] , Naito and Eguchi [24 ] , Nishida and Naito [26 ] and Yoshida and Naito [33 ] developed similar risk bounds, they did not consider data-dependent dictionaries in their theoretical analyses. The theoretical
evaluation of the proposed estimators with such dictionaries is a contribution of this study.
Proofs
Proof of Lemma 4.1
The proof of Lemma 4.1 is divided into the following two parts.
Lemma A.1. For any Λ ~ ∈ W \widetilde{\Lambda} \in\mathcal{W} Λ ∈ W , it follows that
L ^ ( μ k + 1 ( θ 1 ) ) − L ^ ( μ ( ⋅ ; Λ ~ , θ 2 ) ) \widehat{L}\left(\mu_{k+1}(\theta_1)\right)-\widehat{L}\left(\mu\left(\cdot;\widetilde{\Lambda},\theta_2\right)\right) L ( μ k + 1 ( θ 1 ) ) − L ( μ ( ⋅ ; Λ , θ 2 ) )
≤ ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 2 { A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) } + π k + 1 ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + π k + 1 ε (28) \le(1-\pi_{k+1})\left\{\widehat{L}(\mu_k(\theta_1))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)\right\}
+\pi_{k+1}^{2}\left\{A_1(\theta_1,\theta_2)+\lVert\theta_1-\theta_2\rVert_2 A_3(\theta_1,\theta_2)\right\}
+\pi_{k+1}\lVert\theta_1-\theta_2\rVert_2 A_2(\theta_1,\theta_2)+\pi_{k+1}\varepsilon \tag*{(28)} ≤ ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 2 { A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) } + π k + 1 ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + π k + 1 ε (28)
for k ≥ 0 k \ge0 k ≥ 0 , where A 1 A_1 A 1 , A 2 A_2 A 2 , A 3 A_3 A 3 are in Lemma 4.1.
Lemma A.2. If the following inequality
L ^ ( μ ℓ + 1 ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) \widehat{L}(\mu_{\ell+1}(\theta_1))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right) L ( μ ℓ + 1 ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) ≤ ( 1 − π ℓ + 1 ) { L ^ ( μ ℓ ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π ℓ + 1 2 a + π ℓ + 1 b , ℓ ≥ 0 (29) \le(1-\pi_{\ell+1})\left\{\widehat{L}(\mu_{\ell}(\theta_1))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)\right\}+\pi_{\ell+1}^{2}a+\pi_{\ell+1}b,\quad\ell\ge0 \tag*{(29)} ≤ ( 1 − π ℓ + 1 ) { L ( μ ℓ ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π ℓ + 1 2 a + π ℓ + 1 b , ℓ ≥ 0 (29) holds for some Λ ~ ∈ W \widetilde{\Lambda}\in\mathcal{W} Λ ∈ W , a ≥ 0 a\ge0 a ≥ 0 and b ≥ 0 b\ge0 b ≥ 0 , then
L ^ ( μ k ( θ 1 ) ) ≤ L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) + τ 2 k + τ a + b , k ∈ N . (30) \widehat{L}(\mu_k(\theta_1))\le\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)+\frac{\tau^2}{k+\tau}a+b,\quad k\in\mathbb{N}. \tag*{(30)} L ( μ k ( θ 1 )) ≤ L ( μ ( ⋅ , Λ , θ 2 ) ) + k + τ τ 2 a + b , k ∈ N . (30)
Proof of Lemma A.1
We pick Λ ~ = [ λ ~ B ] B ∈ D ∈ W \widetilde{\Lambda}=\left[\widetilde{\lambda}_B\right]_{B\in\mathcal{D}}\in\mathcal{W} Λ = [ λ B ] B ∈ D ∈ W . From the definitions of μ k + 1 \mu_{k+1} μ k + 1 , ξ k + 1 \xi_{k+1} ξ k + 1 , selected I I I words B 1 , … , B I B_1,\ldots,B_I B 1 , … , B I through algorithm in Section 3 and the operator C \mathcal{C} C and T T T , it follows that
L ^ ( μ k + 1 ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) = L ^ ( u − 1 ( T ( θ 1 , ξ k + 1 ) ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) = L ^ ( u − 1 ( T ( θ 1 , C ( π k + 1 ∣ B k + 1 , ξ k ) ) ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) ≤ { inf B ~ ∈ D L ^ ( u − 1 ( T ( θ 1 , C ( π k + 1 ∣ B ~ , ξ k ) ) ) ) + π k + 1 ε } − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) = ∑ B ∈ D λ ~ B [ inf B ~ ∈ D L ^ ( u − 1 ( T ( θ 1 , C ( π k + 1 ∣ B ~ , ξ k ) ) ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) ] + π k + 1 ε ≤ ∑ B ∈ D λ ~ B [ L ^ ( u − 1 ( T ( θ 1 , C ( π k + 1 ∣ B , ξ k ) ) ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) ] + π k + 1 ε = ∑ B ∈ D λ ~ B { L ^ ( u − 1 ( C ( π k + 1 ∣ T ( θ 1 , B ) , T ( θ 1 , ξ k ) ) ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 ε (31) \begin{aligned}
\widehat{L}(\mu_{k+1}(\theta_1))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right) \\
&=\widehat{L}\left(u^{-1}\left(T\left(\theta_1,\xi_{k+1}\right)\right)\right)-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right) \\
&=\widehat{L}\left(u^{-1}\left(T\left(\theta_1,\mathcal{C}\left(\pi_{k+1}\mid B_{k+1},\xi_k\right)\right)\right)\right)-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right) \\
&\le\left\{\inf_{\widetilde{B}\in\mathcal{D}}\widehat{L}\left(u^{-1}\left(T\left(\theta_1,\mathcal{C}\left(\pi_{k+1}\mid\widetilde{B},\xi_k\right)\right)\right)\right)+\pi_{k+1}\varepsilon\right\}-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right) \\
&=\sum_{B\in\mathcal{D}}\widetilde{\lambda}_B\left[\inf_{\widetilde{B}\in\mathcal{D}}\widehat{L}\left(u^{-1}\left(T\left(\theta_1,\mathcal{C}\left(\pi_{k+1}\mid\widetilde{B},\xi_k\right)\right)\right)\right)-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)\right]+\pi_{k+1}\varepsilon\\
&\le\sum_{B\in\mathcal{D}}\widetilde{\lambda}_B\left[\widehat{L}\left(u^{-1}\left(T\left(\theta_1,\mathcal{C}\left(\pi_{k+1}\mid B,\xi_k\right)\right)\right)\right)-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)\right]+\pi_{k+1}\varepsilon\\
&=\sum_{B\in\mathcal{D}}\widetilde{\lambda}_B\left\{\widehat{L}\left(u^{-1}\left(\mathcal{C}\left(\pi_{k+1}\mid T(\theta_1,B),T(\theta_1,\xi_k)\right)\right)\right)-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)\right\}+\pi_{k+1}\varepsilon \tag*{(31)}
\end{aligned} L ( μ k + 1 ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) = L ( u − 1 ( T ( θ 1 , ξ k + 1 ) ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) = L ( u − 1 ( T ( θ 1 , C ( π k + 1 ∣ B k + 1 , ξ k ) ) ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) ≤ { B ∈ D inf L ( u − 1 ( T ( θ 1 , C ( π k + 1 ∣ B , ξ k ) ) ) ) + π k + 1 ε } − L ( μ ( ⋅ , Λ , θ 2 ) ) = B ∈ D ∑ λ B [ B ∈ D inf L ( u − 1 ( T ( θ 1 , C ( π k + 1 ∣ B , ξ k ) ) ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) ] + π k + 1 ε ≤ B ∈ D ∑ λ B [ L ( u − 1 ( T ( θ 1 , C ( π k + 1 ∣ B , ξ k ) ) ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) ] + π k + 1 ε = B ∈ D ∑ λ B { L ( u − 1 ( C ( π k + 1 ∣ T ( θ 1 , B ) , T ( θ 1 , ξ k ) ) ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε (31)
Then by definition of L ^ \widehat{L} L , we see that
L ^ ( u − 1 ( C ( π k + 1 ∣ T ( θ 1 , B ) , T ( θ 1 , ξ k ) ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) = 1 n ∑ i = 1 n { U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − Y i C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) } − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) = 1 n ∑ i = 1 n [ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − Y i { π k + 1 T ( θ 1 , B ) ( X i ) + ( 1 − π k + 1 ) T ( θ 1 , ξ k ) ( X i ) } ] − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) = 1 n ∑ i = 1 n [ ( 1 − π k + 1 ) { U ∗ ( T ( θ 1 , ξ k ) ( X i ) ) − Y i T ( θ 1 , ξ k ) ( X i ) } + U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − ( 1 − π k + 1 ) U ∗ ( T ( θ 1 , ξ k ) ( X i ) ) − Y i π k + 1 T ( θ 1 , B ) ( X i ) ] − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) = ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + 1 n ∑ i = 1 n [ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − ( 1 − π k + 1 ) U ∗ ( T ( θ 1 , ξ k ) ( X i ) ) − Y i π k + 1 T ( θ 1 , B ) ( X i ) − π k + 1 L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) ] = ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + 1 n ∑ i = 1 n [ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − ( 1 − π k + 1 ) U ∗ ( T ( θ 1 , ξ k ) ( X i ) ) − Y i π k + 1 T ( θ 1 , B ) ( X i ) − π k + 1 { U ∗ ( u ( μ ( X i , Λ ~ , θ 2 ) ) ) − Y i u ( μ ( X i , Λ ~ , θ 2 ) ) } ] = ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + 1 n ∑ i = 1 n [ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) \begin{aligned}
&\widehat{L}(u^{-1}(C(\pi_{k+1}\mid T(\theta_1,B),T(\theta_1,\xi_k)))-\widehat{L}(\mu(\cdot,\widetilde{\Lambda},\theta_2))\\
&=\frac{1}{n}\sum_{i=1}^{n}\{U^{*}(C(\pi_{k+1}\mid T(\theta_1,B)(X_i),T(\theta_1,\xi_k)(X_i)))\\
&\qquad-Y_iC(\pi_{k+1}\mid T(\theta_1,B)(X_i),T(\theta_1,\xi_k)(X_i))\}-\widehat{L}(\mu(\cdot,\widetilde{\Lambda},\theta_2))\\
&=\frac{1}{n}\sum_{i=1}^{n}[U^{*}(C(\pi_{k+1}\mid T(\theta_1,B)(X_i),T(\theta_1,\xi_k)(X_i)))\\
&\qquad-Y_i\{\pi_{k+1}T(\theta_1,B)(X_i)+(1-\pi_{k+1})T(\theta_1,\xi_k)(X_i)\}]-\widehat{L}(\mu(\cdot,\widetilde{\Lambda},\theta_2))\\
&=\frac{1}{n}\sum_{i=1}^{n}[(1-\pi_{k+1})\{U^{*}(T(\theta_1,\xi_k)(X_i))-Y_iT(\theta_1,\xi_k)(X_i)\}\\
&\qquad+U^{*}(C(\pi_{k+1}\mid T(\theta_1,B)(X_i),T(\theta_1,\xi_k)(X_i)))-(1-\pi_{k+1})U^{*}(T(\theta_1,\xi_k)(X_i))\\
&\qquad-Y_i\pi_{k+1}T(\theta_1,B)(X_i)]-\widehat{L}(\mu(\cdot,\widetilde{\Lambda},\theta_2))\\
&=(1-\pi_{k+1})\{\widehat{L}(\mu_k(\theta_1))-\widehat{L}(\mu(\cdot,\widetilde{\Lambda},\theta_2))\}\\
&\qquad+\frac{1}{n}\sum_{i=1}^{n}[U^{*}(C(\pi_{k+1}\mid T(\theta_1,B)(X_i),T(\theta_1,\xi_k)(X_i)))-(1-\pi_{k+1})U^{*}(T(\theta_1,\xi_k)(X_i))\\
&\qquad-Y_i\pi_{k+1}T(\theta_1,B)(X_i)-\pi_{k+1}\widehat{L}(\mu(\cdot,\widetilde{\Lambda},\theta_2))]\\
&=(1-\pi_{k+1})\{\widehat{L}(\mu_k(\theta_1))-\widehat{L}(\mu(\cdot,\widetilde{\Lambda},\theta_2))\}\\
&\qquad+\frac{1}{n}\sum_{i=1}^{n}[U^{*}(C(\pi_{k+1}\mid T(\theta_1,B)(X_i),T(\theta_1,\xi_k)(X_i)))-(1-\pi_{k+1})U^{*}(T(\theta_1,\xi_k)(X_i))\\
&\qquad-Y_i\pi_{k+1}T(\theta_1,B)(X_i)-\pi_{k+1}\{U^{*}(u(\mu(X_i,\widetilde{\Lambda},\theta_2)))-Y_i u(\mu(X_i,\widetilde{\Lambda},\theta_2))\}]\\
&=(1-\pi_{k+1})\{\widehat{L}(\mu_k(\theta_1))-\widehat{L}(\mu(\cdot,\widetilde{\Lambda},\theta_2))\}\\
&\qquad+\frac{1}{n}\sum_{i=1}^{n}[U^{*}(C(\pi_{k+1}\mid T(\theta_1,B)(X_i),T(\theta_1,\xi_k)(X_i)))
\end{aligned} L ( u − 1 ( C ( π k + 1 ∣ T ( θ 1 , B ) , T ( θ 1 , ξ k ))) − L ( μ ( ⋅ , Λ , θ 2 )) = n 1 i = 1 ∑ n { U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ))) − Y i C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ))} − L ( μ ( ⋅ , Λ , θ 2 )) = n 1 i = 1 ∑ n [ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ))) − Y i { π k + 1 T ( θ 1 , B ) ( X i ) + ( 1 − π k + 1 ) T ( θ 1 , ξ k ) ( X i )}] − L ( μ ( ⋅ , Λ , θ 2 )) = n 1 i = 1 ∑ n [( 1 − π k + 1 ) { U ∗ ( T ( θ 1 , ξ k ) ( X i )) − Y i T ( θ 1 , ξ k ) ( X i )} + U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ))) − ( 1 − π k + 1 ) U ∗ ( T ( θ 1 , ξ k ) ( X i )) − Y i π k + 1 T ( θ 1 , B ) ( X i )] − L ( μ ( ⋅ , Λ , θ 2 )) = ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ))} + n 1 i = 1 ∑ n [ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ))) − ( 1 − π k + 1 ) U ∗ ( T ( θ 1 , ξ k ) ( X i )) − Y i π k + 1 T ( θ 1 , B ) ( X i ) − π k + 1 L ( μ ( ⋅ , Λ , θ 2 ))] = ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ))} + n 1 i = 1 ∑ n [ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ))) − ( 1 − π k + 1 ) U ∗ ( T ( θ 1 , ξ k ) ( X i )) − Y i π k + 1 T ( θ 1 , B ) ( X i ) − π k + 1 { U ∗ ( u ( μ ( X i , Λ , θ 2 ))) − Y i u ( μ ( X i , Λ , θ 2 ))}] = ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ))} + n 1 i = 1 ∑ n [ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i )))
− C ( π k + 1 ∣ U ∗ ( u ( μ ( X i , Λ ~ , θ 2 ) ) , U ∗ ( T ( θ 1 , ξ k ) ( X i ) ) ) ) - C\left(\pi_{k+1}\mid U^*\left(u\left(\mu\left(X_i,\widetilde{\Lambda},\theta_2\right)\right),U^*\left(T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right)\right) − C ( π k + 1 ∣ U ∗ ( u ( μ ( X i , Λ , θ 2 ) ) , U ∗ ( T ( θ 1 , ξ k ) ( X i ) ) ) )
− π k + 1 Y i { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ ~ , θ 2 ) ) } . (32) -\pi_{k+1}Y_i\left\{T\left(\theta_1,B\right)\left(X_i\right)-u\left(\mu\left(X_i,\widetilde{\Lambda},\theta_2\right)\right)\right\}. \tag*{(32)} − π k + 1 Y i { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ , θ 2 ) ) } . (32)
We shall further bound (A.5) from above. The convexity of U ∗ U^* U ∗ implies the following inequalities
C ( π k + 1 ∣ U ∗ ( a ) , U ∗ ( b ) ) ≥ U ∗ ( C ( π k + 1 ∣ a , b ) ) and U ∗ ( b ) − U ∗ ( a ) ≤ u − 1 ( b ) ( b − a ) for a , b ∈ R , C\left(\pi_{k+1}\mid U^*(a),U^*(b)\right)\ge U^*\left(C\left(\pi_{k+1}\mid a,b\right)\right)\text{ and }U^*(b)-U^*(a)\le u^{-1}(b)(b-a)\text{ for }a,b\in\mathbb{R}, C ( π k + 1 ∣ U ∗ ( a ) , U ∗ ( b ) ) ≥ U ∗ ( C ( π k + 1 ∣ a , b ) ) and U ∗ ( b ) − U ∗ ( a ) ≤ u − 1 ( b ) ( b − a ) for a , b ∈ R ,
so that
U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − C ( π k + 1 ∣ U ∗ ( u ( μ ( X i , Λ ~ , θ 2 ) ) ) , U ∗ ( T ( θ 1 , ξ k ) ( X i ) ) ) ≤ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − U ∗ ( C ( π k + 1 ∣ u ( μ ( X i , Λ ~ , θ 2 ) ) , T ( θ 1 , ξ k ) ( X i ) ) ) ≤ u − 1 ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) × π k + 1 { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ ~ , θ 2 ) ) } . (33) \begin{aligned}
&U^*\left(C\left(\pi_{k+1}\mid T\left(\theta_1,B\right)\left(X_i\right),T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right)\\
&\quad-C\left(\pi_{k+1}\mid U^*\left(u\left(\mu\left(X_i,\widetilde{\Lambda},\theta_2\right)\right)\right),U^*\left(T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right)\\
&\le U^*\left(C\left(\pi_{k+1}\mid T\left(\theta_1,B\right)\left(X_i\right),T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right)\\
&\quad-U^*\left(C\left(\pi_{k+1}\mid u\left(\mu\left(X_i,\widetilde{\Lambda},\theta_2\right)\right),T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right)\\
&\le u^{-1}\left(C\left(\pi_{k+1}\mid T\left(\theta_1,B\right)\left(X_i\right),T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right)\\
&\quad\times\pi_{k+1}\left\{T\left(\theta_1,B\right)\left(X_i\right)-u\left(\mu\left(X_i,\widetilde{\Lambda},\theta_2\right)\right)\right\}. \tag*{(33)}
\end{aligned} U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − C ( π k + 1 ∣ U ∗ ( u ( μ ( X i , Λ , θ 2 ) ) ) , U ∗ ( T ( θ 1 , ξ k ) ( X i ) ) ) ≤ U ∗ ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) − U ∗ ( C ( π k + 1 ∣ u ( μ ( X i , Λ , θ 2 ) ) , T ( θ 1 , ξ k ) ( X i ) ) ) ≤ u − 1 ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) × π k + 1 { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ , θ 2 ) ) } . (33)
In addition, by the mean value theorem, there exists a k ( X i , θ 1 , B ) ∈ ( 0 , 1 ) a_k\left(X_i,\theta_1,B\right)\in(0,1) a k ( X i , θ 1 , B ) ∈ ( 0 , 1 ) for i = 1 , … , n i=1,\ldots,n i = 1 , … , n such that
u − 1 ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) = u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) + ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) × { C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) − T ( θ 1 , ξ k ) ( X i ) } = u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) + π k + 1 ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) × { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } . (34) \begin{aligned}
u^{-1}\left(C\left(\pi_{k+1}\mid T\left(\theta_1,B\right)\left(X_i\right),T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right)
&=u^{-1}\left(T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\\
&\quad+\left(U^*\right)''\left(b_k\left(X_i,\theta_1,B\right)\right)\\
&\quad\times\left\{C\left(\pi_{k+1}\mid T\left(\theta_1,B\right)\left(X_i\right),T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)-T\left(\theta_1,\xi_k\right)\left(X_i\right)\right\}\\
&=u^{-1}\left(T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)+\pi_{k+1}\left(U^*\right)''\left(b_k\left(X_i,\theta_1,B\right)\right)\\
&\quad\times\left\{T\left(\theta_1,B\right)\left(X_i\right)-T\left(\theta_1,\xi_k\right)\left(X_i\right)\right\}. \tag*{(34)}
\end{aligned} u − 1 ( C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) ) = u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) + ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) × { C ( π k + 1 ∣ T ( θ 1 , B ) ( X i ) , T ( θ 1 , ξ k ) ( X i ) ) − T ( θ 1 , ξ k ) ( X i ) } = u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) + π k + 1 ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) × { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } . (34)
where we have used a simplified notation
b k ( X i , θ 1 , B ) = T ( θ 1 , ξ k ) ( X i ) + a k ( X i , θ 1 , B ) π k + 1 { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } = T ( θ 1 , C ( π k + 1 a k ( X i , θ 1 , B ) ∣ B , ξ k ) ) . \begin{aligned}
b_k\left(X_i,\theta_1,B\right)
&=T\left(\theta_1,\xi_k\right)\left(X_i\right)+a_k\left(X_i,\theta_1,B\right)\pi_{k+1}\left\{T\left(\theta_1,B\right)\left(X_i\right)-T\left(\theta_1,\xi_k\right)\left(X_i\right)\right\}\\
&=T\left(\theta_1,C\left(\pi_{k+1}a_k\left(X_i,\theta_1,B\right)\mid B,\xi_k\right)\right).
\end{aligned} b k ( X i , θ 1 , B ) = T ( θ 1 , ξ k ) ( X i ) + a k ( X i , θ 1 , B ) π k + 1 { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } = T ( θ 1 , C ( π k + 1 a k ( X i , θ 1 , B ) ∣ B , ξ k ) ) .
Moreover, the common term in (A.5) and (A.6) can be expressed as
T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ ~ , θ 2 ) ) = T ( θ 1 , B ) ( X i ) − T ( θ 2 , ∑ B ~ ∈ D λ ~ B ~ T ( θ 2 , B ~ ) ) ( X i ) = { m ( X i , θ 1 ) + m ( X i , θ 1 ) γ B ( X i ) } − { m ( X i , θ 2 ) + m ( X i , θ 2 ) γ ∑ B ~ ∈ D λ ~ B ~ T ( θ 2 , B ~ ) ( X i ) } + m ( X i , θ 2 ) γ B ( X i ) − m ( X i , θ 2 ) γ B ( X i ) = m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) + m ( X i , θ 2 ) γ { B ( X i ) − ∑ B ~ ∈ D λ ~ B ~ B ~ ( X i ) } . (35) \begin{aligned}
T(\theta_{1},B)(X_i)-u\left(\mu\left(X_i,\widetilde{\Lambda},\theta_{2}\right)\right)
&=T(\theta_{1},B)(X_i)-T\left(\theta_{2},\sum_{\widetilde{B}\in D}\widetilde{\lambda}_{\widetilde{B}}T(\theta_{2},\widetilde{B})\right)(X_i)\\
&=\{m(X_i,\theta_{1})+m(X_i,\theta_{1})^\gamma B(X_i)\}\\
&\quad-\left\{m(X_i,\theta_{2})+m(X_i,\theta_{2})^\gamma\sum_{\widetilde{B}\in D}\widetilde{\lambda}_{\widetilde{B}}T(\theta_{2},\widetilde{B})(X_i)\right\}\\
&\quad+m(X_i,\theta_{2})^\gamma B(X_i)-m(X_i,\theta_{2})^\gamma B(X_i)\\
&=m(X_i,\theta_{1})-m(X_i,\theta_{2})+\{m(X_i,\theta_{1})^\gamma-m(X_i,\theta_{2})^\gamma\}\cdot B(X_i)\\
&\quad+m(X_i,\theta_{2})^\gamma\left\{B(X_i)-\sum_{\widetilde{B}\in D}\widetilde{\lambda}_{\widetilde{B}}\widetilde{B}(X_i)\right\}. \tag*{(35)}
\end{aligned} T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ , θ 2 ) ) = T ( θ 1 , B ) ( X i ) − T θ 2 , B ∈ D ∑ λ B T ( θ 2 , B ) ( X i ) = { m ( X i , θ 1 ) + m ( X i , θ 1 ) γ B ( X i )} − ⎩ ⎨ ⎧ m ( X i , θ 2 ) + m ( X i , θ 2 ) γ B ∈ D ∑ λ B T ( θ 2 , B ) ( X i ) ⎭ ⎬ ⎫ + m ( X i , θ 2 ) γ B ( X i ) − m ( X i , θ 2 ) γ B ( X i ) = m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) + m ( X i , θ 2 ) γ ⎩ ⎨ ⎧ B ( X i ) − B ∈ D ∑ λ B B ( X i ) ⎭ ⎬ ⎫ . (35)
Those (A.4), (A.5), (A.6), (A.7), (A.8) and the equality
∑ B ∈ D λ B { B ( X i ) − ∑ B ~ ∈ D λ ~ B ~ B ~ ( X i ) } = 0 \sum_{B\in D}\lambda_B\left\{B(X_i)-\sum_{\widetilde{B}\in D}\widetilde{\lambda}_{\widetilde{B}}\widetilde{B}(X_i)\right\}=0 B ∈ D ∑ λ B ⎩ ⎨ ⎧ B ( X i ) − B ∈ D ∑ λ B B ( X i ) ⎭ ⎬ ⎫ = 0
imply that
L ^ ( μ k + 1 ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) ≤ ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 ε + ∑ B ∈ D λ ~ B 1 n ∑ i = 1 n ( π k + 1 [ u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) + π k + 1 ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) ⋅ { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } ] × { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ ~ , θ 2 ) ) } − π k + 1 Y i { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ ~ , θ 2 ) ) } ) = ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 ε \begin{aligned}
\widehat{L}(\mu_{k+1}(\theta_{1}))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_{2}\right)\right)
&\leq(1-\pi_{k+1})\left\{\widehat{L}(\mu_k(\theta_{1}))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_{2}\right)\right)\right\}+\pi_{k+1}\varepsilon\\
&\quad+\sum_{B\in D}\widetilde{\lambda}_B\frac{1}{n}\sum_{i=1}^{n}\left(\pi_{k+1}\left[u^{-1}\left(T(\theta_{1},\xi_k)(X_i)\right)\right.\right.\\
&\qquad\left.\left.+\pi_{k+1}(U^{*})''(b_k(X_i,\theta_{1},B))\cdot\{T(\theta_{1},B)(X_i)-T(\theta_{1},\xi_k)(X_i)\}\right]\right.\\
&\qquad\left.\times\{T(\theta_{1},B)(X_i)-u\left(\mu\left(X_i,\widetilde{\Lambda},\theta_{2}\right)\right)\}\right.\\
&\qquad\left.-\pi_{k+1}Y_i\{T(\theta_{1},B)(X_i)-u\left(\mu\left(X_i,\widetilde{\Lambda},\theta_{2}\right)\right)\}\right)\\
&=(1-\pi_{k+1})\left\{\widehat{L}(\mu_k(\theta_{1}))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_{2}\right)\right)\right\}+\pi_{k+1}\varepsilon
\end{aligned} L ( μ k + 1 ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) ≤ ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε + B ∈ D ∑ λ B n 1 i = 1 ∑ n ( π k + 1 [ u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) + π k + 1 ( U ∗ ) ′′ ( b k ( X i , θ 1 , B )) ⋅ { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i )} ] × { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ , θ 2 ) ) } − π k + 1 Y i { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ , θ 2 ) ) } ) = ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε
+ ∑ B ∈ D λ ~ B ∼ 1 n ∑ i = 1 n [ u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) − Y i + π k + 1 ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) ⋅ { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } × { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ ~ , θ 2 ) ) } ] = ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 ε + π k + 1 ∑ B ∈ D λ ~ B 1 n ∑ i = 1 n { u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) − Y i } { T ( θ 1 , B ) ( X i ) − u ( μ ( X ~ i , Λ ~ , θ 2 ) ) } + π k + 1 2 ∑ B ∈ D λ ~ B 1 n ∑ i = 1 n ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) ⋅ { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } × { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ ~ , θ 2 ) ) } = ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 ε + π k + 1 1 n ∑ i = 1 n ∑ B ∈ D λ ~ B { u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) − Y i } × [ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) ] + π k + 1 2 ∑ B ∈ D λ ~ B 1 n ∑ i = 1 n ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) m ( X i , θ 1 ) γ × { B ( X i ) − ∑ j = 1 k λ j , k B j ( X i ) } × [ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) + m ( X i , θ 2 ) γ { B ( X i ) − ∑ B ~ ∈ D λ ~ B ~ B ~ ( X i ) } ] = ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 ε + π k + 1 1 n ∑ i = 1 n { u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) − Y i } \begin{aligned}
&+ \sum_{B\in\mathcal{D}} \widetilde{\lambda}_{B}^{\sim} \frac{1}{n}\sum_{i=1}^{n}\left[u^{-1}\left(T(\theta_{1},\xi_{k})(X_{i})\right)-Y_{i}\right.\\
&\qquad+\pi_{k+1}(U^{*})^{\prime\prime}\left(b_{k}(X_{i},\theta_{1},B)\right)\cdot\left\{T(\theta_{1},B)(X_{i})-T(\theta_{1},\xi_{k})(X_{i})\right\}\\
&\qquad\qquad\left.\times\left\{T(\theta_{1},B)(X_{i})-u\left(\mu\left(X_{i},\widetilde{\Lambda},\theta_{2}\right)\right)\right\}\right]\\
&=(1-\pi_{k+1})\left\{\widehat{L}\left(\mu_{k}(\theta_{1})\right)-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_{2}\right)\right)\right\}+\pi_{k+1}\varepsilon\\
&\qquad+\pi_{k+1}\sum_{B\in\mathcal{D}}\widetilde{\lambda}_{B}\frac{1}{n}\sum_{i=1}^{n}\left\{u^{-1}\left(T(\theta_{1},\xi_{k})(X_{i})\right)-Y_{i}\right\}\left\{T(\theta_{1},B)(X_{i})-u\left(\mu\left(\widetilde{X}_{i},\widetilde{\Lambda},\theta_{2}\right)\right)\right\}\\
&\qquad+\pi_{k+1}^{2}\sum_{B\in\mathcal{D}}\widetilde{\lambda}_{B}\frac{1}{n}\sum_{i=1}^{n}(U^{*})^{\prime\prime}\left(b_{k}(X_{i},\theta_{1},B)\right)\cdot\left\{T(\theta_{1},B)(X_{i})-T(\theta_{1},\xi_{k})(X_{i})\right\}\\
&\qquad\qquad\times\left\{T(\theta_{1},B)(X_{i})-u\left(\mu\left(X_{i},\widetilde{\Lambda},\theta_{2}\right)\right)\right\}\\
&=(1-\pi_{k+1})\left\{\widehat{L}\left(\mu_{k}(\theta_{1})\right)-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_{2}\right)\right)\right\}+\pi_{k+1}\varepsilon\\
&\qquad+\pi_{k+1}\frac{1}{n}\sum_{i=1}^{n}\sum_{B\in\mathcal{D}}\widetilde{\lambda}_{B}\left\{u^{-1}\left(T(\theta_{1},\xi_{k})(X_{i})\right)-Y_{i}\right\}\\
&\qquad\qquad\times\left[m(X_{i},\theta_{1})-m(X_{i},\theta_{2})+\left\{m(X_{i},\theta_{1})^{\gamma}-m(X_{i},\theta_{2})^{\gamma}\right\}\cdot B(X_{i})\right]\\
&\qquad+\pi_{k+1}^{2}\sum_{B\in\mathcal{D}}\widetilde{\lambda}_{B}\frac{1}{n}\sum_{i=1}^{n}(U^{*})^{\prime\prime}\left(b_{k}(X_{i},\theta_{1},B)\right)m(X_{i},\theta_{1})^{\gamma}\\
&\qquad\qquad\times\left\{B(X_{i})-\sum_{j=1}^{k}\lambda_{j,k}B_{j}(X_{i})\right\}\\
&\qquad\qquad\times\left[m(X_{i},\theta_{1})-m(X_{i},\theta_{2})+\left\{m(X_{i},\theta_{1})^{\gamma}-m(X_{i},\theta_{2})^{\gamma}\right\}\cdot B(X_{i})\right.\\
&\qquad\qquad\qquad\left.+m(X_{i},\theta_{2})^{\gamma}\left\{B(X_{i})-\sum_{\widetilde{B}\in\mathcal{D}}\widetilde{\lambda}_{\widetilde{B}}\widetilde{B}(X_{i})\right\}\right]\\
&=(1-\pi_{k+1})\left\{\widehat{L}\left(\mu_{k}(\theta_{1})\right)-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_{2}\right)\right)\right\}+\pi_{k+1}\varepsilon\\
&\qquad+\pi_{k+1}\frac{1}{n}\sum_{i=1}^{n}\left\{u^{-1}\left(T(\theta_{1},\xi_{k})(X_{i})\right)-Y_{i}\right\}
\end{aligned} + B ∈ D ∑ λ B ∼ n 1 i = 1 ∑ n [ u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) − Y i + π k + 1 ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) ⋅ { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } × { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ , θ 2 ) ) } ] = ( 1 − π k + 1 ) { L ( μ k ( θ 1 ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε + π k + 1 B ∈ D ∑ λ B n 1 i = 1 ∑ n { u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) − Y i } { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ , θ 2 ) ) } + π k + 1 2 B ∈ D ∑ λ B n 1 i = 1 ∑ n ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) ⋅ { T ( θ 1 , B ) ( X i ) − T ( θ 1 , ξ k ) ( X i ) } × { T ( θ 1 , B ) ( X i ) − u ( μ ( X i , Λ , θ 2 ) ) } = ( 1 − π k + 1 ) { L ( μ k ( θ 1 ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε + π k + 1 n 1 i = 1 ∑ n B ∈ D ∑ λ B { u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) − Y i } × [ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) ] + π k + 1 2 B ∈ D ∑ λ B n 1 i = 1 ∑ n ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) m ( X i , θ 1 ) γ × { B ( X i ) − j = 1 ∑ k λ j , k B j ( X i ) } × [ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) + m ( X i , θ 2 ) γ ⎩ ⎨ ⎧ B ( X i ) − B ∈ D ∑ λ B B ( X i ) ⎭ ⎬ ⎫ = ( 1 − π k + 1 ) { L ( μ k ( θ 1 ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε + π k + 1 n 1 i = 1 ∑ n { u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) − Y i }
× [ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ ∑ B ∈ D λ ~ B B ( X i ) ] + π k + 1 2 ∑ B ∈ D λ ~ B 1 n ∑ i = 1 n ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) m ( X i , θ 1 ) γ { B ( X i ) − ∑ j = 1 k λ j , k B j ( X i ) } × [ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) + m ( X i , θ 2 ) γ { B ( X i ) − ∑ B ~ ∈ D λ ~ B ~ B ~ ( X i ) } ] ≤ ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ^ , θ 2 ) ) } + π k + 1 ε + π k + 1 1 n ∑ i = 1 n ∑ B ∈ D λ ~ B ∣ Y i − u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) ∣ × ∣ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) ∣ + π k + 1 2 ∑ B ∈ D λ ~ B ∑ B ~ ∈ D λ ~ B ~ ∑ j = 1 k λ j , k 1 n ∑ i = 1 n ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) × ∣ m ( X i , θ 1 ) γ ∣ ∣ B ( X i ) − B j ( X i ) ∣ × { ∣ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) ∣ + ∣ m ( X i , θ 2 ) γ { B ( X i ) − B ~ ( X i ) } ∣ } ≤ ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ^ , θ 2 ) ) } + π k + 1 ε + π k + 1 ∥ θ 1 − θ 2 ∥ 2 sup B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] ∥ { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ ∂ θ m ( x , θ ) ∣ θ = θ 2 + β ( θ 1 − θ 2 ) ∥ 2 × 1 n ∑ i = 1 n ∣ Y i − u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) ∣ + π k + 1 2 ∥ θ 1 − θ 2 ∥ 2 sup B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] ∥ { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ ∂ θ m ( x , θ ) ∣ θ = θ 2 + β ( θ 1 − θ 2 ) ∥ 2 \begin{aligned}
&\times\left[m\left(X_i,\theta_1\right)-m\left(X_i,\theta_2\right)+\left\{m\left(X_i,\theta_1\right)^\gamma-m\left(X_i,\theta_2\right)^\gamma\right\}\cdot\sum_{B\in\mathcal{D}}\widetilde{\lambda}_B B\left(X_i\right)\right]\\
&+\pi_{k+1}^2\sum_{B\in\mathcal{D}}\widetilde{\lambda}_B\frac{1}{n}\sum_{i=1}^{n}(U^*)''\left(b_k\left(X_i,\theta_1,B\right)\right)m\left(X_i,\theta_1\right)^\gamma\left\{B\left(X_i\right)-\sum_{j=1}^{k}\lambda_{j,k}B_j\left(X_i\right)\right\}\\
&\times\left[m\left(X_i,\theta_1\right)-m\left(X_i,\theta_2\right)+\left\{m\left(X_i,\theta_1\right)^\gamma-m\left(X_i,\theta_2\right)^\gamma\right\}\cdot B\left(X_i\right)\right.\\
&\qquad\left.+m\left(X_i,\theta_2\right)^\gamma\left\{B\left(X_i\right)-\sum_{\widetilde{B}\in\mathcal{D}}\widetilde{\lambda}_{\widetilde{B}}\widetilde{B}\left(X_i\right)\right\}\right]\\
&\leq\left(1-\pi_{k+1}\right)\left\{\widehat{L}\left(\mu_k\left(\theta_1\right)\right)-\widehat{L}\left(\mu\left(\cdot,\widehat{\Lambda},\theta_2\right)\right)\right\}+\pi_{k+1}\varepsilon\\
&\quad+\pi_{k+1}\frac{1}{n}\sum_{i=1}^{n}\sum_{B\in\mathcal{D}}\widetilde{\lambda}_B\left|Y_i-u^{-1}\left(T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right|\\
&\qquad\times\left|m\left(X_i,\theta_1\right)-m\left(X_i,\theta_2\right)+\left\{m\left(X_i,\theta_1\right)^\gamma-m\left(X_i,\theta_2\right)^\gamma\right\}\cdot B\left(X_i\right)\right|\\
&\quad+\pi_{k+1}^2\sum_{B\in\mathcal{D}}\widetilde{\lambda}_B\sum_{\widetilde{B}\in\mathcal{D}}\widetilde{\lambda}_{\widetilde{B}}\sum_{j=1}^{k}\lambda_{j,k}\frac{1}{n}\sum_{i=1}^{n}(U^*)''\left(b_k\left(X_i,\theta_1,B\right)\right)\\
&\qquad\times\left|m\left(X_i,\theta_1\right)^\gamma\right|\left|B\left(X_i\right)-B_j\left(X_i\right)\right|\\
&\qquad\times\left\{\left|m\left(X_i,\theta_1\right)-m\left(X_i,\theta_2\right)+\left\{m\left(X_i,\theta_1\right)^\gamma-m\left(X_i,\theta_2\right)^\gamma\right\}\cdot B\left(X_i\right)\right|\right.\\
&\qquad\qquad\left.+\left|m\left(X_i,\theta_2\right)^\gamma\left\{B\left(X_i\right)-\widetilde{B}\left(X_i\right)\right\}\right|\right\}\\
&\leq\left(1-\pi_{k+1}\right)\left\{\widehat{L}\left(\mu_k\left(\theta_1\right)\right)-\widehat{L}\left(\mu\left(\cdot,\widehat{\Lambda},\theta_2\right)\right)\right\}+\pi_{k+1}\varepsilon\\
&\quad+\pi_{k+1}\left\|\theta_1-\theta_2\right\|_2\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}\sup_{\beta\in[0,1]}\left\|\left\{1+\gamma m\left(x,\theta\right)^{\gamma-1}B(x)\right\}\left.\frac{\partial}{\partial\theta}m(x,\theta)\right|_{\theta=\theta_2+\beta(\theta_1-\theta_2)}\right\|_2\\
&\qquad\times\frac{1}{n}\sum_{i=1}^{n}\left|Y_i-u^{-1}\left(T\left(\theta_1,\xi_k\right)\left(X_i\right)\right)\right|\\
&\quad+\pi_{k+1}^2\left\|\theta_1-\theta_2\right\|_2\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}\sup_{\beta\in[0,1]}\left\|\left\{1+\gamma m\left(x,\theta\right)^{\gamma-1}B(x)\right\}\left.\frac{\partial}{\partial\theta}m(x,\theta)\right|_{\theta=\theta_2+\beta(\theta_1-\theta_2)}\right\|_2
\end{aligned} × [ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ∈ D ∑ λ B B ( X i ) ] + π k + 1 2 B ∈ D ∑ λ B n 1 i = 1 ∑ n ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) m ( X i , θ 1 ) γ { B ( X i ) − j = 1 ∑ k λ j , k B j ( X i ) } × [ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) + m ( X i , θ 2 ) γ ⎩ ⎨ ⎧ B ( X i ) − B ∈ D ∑ λ B B ( X i ) ⎭ ⎬ ⎫ ≤ ( 1 − π k + 1 ) { L ( μ k ( θ 1 ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε + π k + 1 n 1 i = 1 ∑ n B ∈ D ∑ λ B Y i − u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) × ∣ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) ∣ + π k + 1 2 B ∈ D ∑ λ B B ∈ D ∑ λ B j = 1 ∑ k λ j , k n 1 i = 1 ∑ n ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) × ∣ m ( X i , θ 1 ) γ ∣ ∣ B ( X i ) − B j ( X i ) ∣ × { ∣ m ( X i , θ 1 ) − m ( X i , θ 2 ) + { m ( X i , θ 1 ) γ − m ( X i , θ 2 ) γ } ⋅ B ( X i ) ∣ + m ( X i , θ 2 ) γ { B ( X i ) − B ( X i ) } } ≤ ( 1 − π k + 1 ) { L ( μ k ( θ 1 ) ) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε + π k + 1 ∥ θ 1 − θ 2 ∥ 2 B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] sup { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ θ ∂ m ( x , θ ) θ = θ 2 + β ( θ 1 − θ 2 ) 2 × n 1 i = 1 ∑ n Y i − u − 1 ( T ( θ 1 , ξ k ) ( X i ) ) + π k + 1 2 ∥ θ 1 − θ 2 ∥ 2 B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] sup { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ θ ∂ m ( x , θ ) θ = θ 2 + β ( θ 1 − θ 2 ) 2
× ∑ B ∈ D λ ~ B ∑ j = 1 k λ ~ j , k 1 n ∑ i = 1 n ∣ m ( x , θ 1 ) ∣ γ ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) ∣ B ( X i ) − B j ( X i ) ∣ + π k + 1 2 ∑ B ∈ D λ ~ B ∑ B ~ ∈ D λ ~ B ~ ∑ j = 1 k λ ~ j , k 1 n ∑ i = 1 n ∣ m ( X i , θ 1 ) m ( X i , θ 2 ) ∣ γ × ( U ∗ ) ′ ′ ( b k ( X i , θ 1 , B ) ) ∣ B ( X i ) − B j ( X i ) ∣ ∣ B ( X i ) − B ~ ( X i ) ∣ ≤ ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 ε + π k + 1 ∥ θ 1 − θ 2 ∥ 2 sup B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] ∥ { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ ∂ θ m ( x , θ ) ∣ θ = θ 2 + β ( θ 1 − θ 2 ) ∥ 2 × 1 n ∑ i = 1 n max { ∣ Y i − u − 1 ( T ( θ 1 , sup B ∈ D B ) ( X i ) ) ∣ , ∣ Y i − u − 1 ( T ( θ 1 , inf B ∈ D B ) ( X i ) ) ∣ } + 2 π k + 1 2 sup B ∈ D sup x ∈ D ∣ B ( x ) ∣ ∥ θ 1 − θ 2 ∥ 2 × sup x ∈ D ∣ m ( x , θ 1 ) ∣ γ ∑ B ∈ D λ ~ B sup β ∈ [ 0 , 1 ] ( U ∗ ) ′ ′ ( C ( β ∣ T ( θ 1 , B ) , T ( θ 1 , ξ k ) ( x ) ) ) × sup B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] ∥ { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ ∂ θ m ( x , θ ) ∣ θ = θ 2 + β ( θ 1 − θ 2 ) ∥ 2 + 4 π k + 1 2 sup B ∈ D sup x ∈ D ∣ B ( x ) ∣ 2 sup x ∈ D ∣ m ( x , θ 1 ) m ( x , θ 2 ) ∣ γ sup Λ ∈ W sup x ∈ D ( U ∗ ) ′ ′ ( u ( μ ( x , Λ , θ 1 ) ) ) 2 × ∑ B ∈ D λ ~ B sup β ∈ [ 0 , 1 ] ( U ∗ ) ′ ′ ( C ( β ∣ T ( θ 1 , B ) , T ( θ 1 , ξ k ) ( x ) ) ) ≤ ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 ε + π k + 1 2 [ 4 sup B ∈ D sup x ∈ D ∣ B ( x ) ∣ 2 sup x ∈ D ∣ m ( x , θ 1 ) m ( x , θ 2 ) ∣ γ sup Λ ∈ W sup x ∈ D ( U ∗ ) ′ ′ ( u ( μ ( x , Λ , θ 1 ) ) ) 2 + 2 sup B ∈ D sup x ∈ D ∣ B ( x ) ∣ ∥ θ 1 − θ 2 ∥ 2 sup x ∈ D ∣ m ( x , θ 1 ) ∣ γ sup Λ ∈ W sup x ∈ D ( U ∗ ) ′ ′ ( u ( μ ( x , Λ , θ 1 ) ) ) 2 × sup B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] ∥ { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ ∂ θ m ( x , θ ) ∣ θ = θ 2 + β ( θ 1 − θ 2 ) ∥ 2 ] + π k + 1 ∥ θ 1 − θ 2 ∥ 2 sup B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] ∥ { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ ∂ θ m ( x , θ ) ∣ θ = θ 2 + β ( θ 1 − θ 2 ) ∥ 2 × 1 n ∑ i = 1 n sup B ∈ D ∣ Y i − u − 1 ( T ( θ 1 , B ) ( X i ) ) ∣ = ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 2 { A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) } + π k + 1 ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + π k + 1 ε . \begin{aligned}
&\times\sum_{B\in\mathcal{D}} \tilde{\lambda}_{B}\sum_{j=1}^{k}\tilde{\lambda}_{j,k}\frac{1}{n}\sum_{i=1}^{n}|m(x,\theta_{1})|^{\gamma}(U^{*})^{\prime\prime}\left(b_{k}(X_{i},\theta_{1},B)\right)|B(X_{i})-B_{j}(X_{i})| \\
&\quad+\pi_{k+1}^{2}\sum_{B\in\mathcal{D}}\tilde{\lambda}_{B}\sum_{\widetilde{B}\in\mathcal{D}}\tilde{\lambda}_{\widetilde{B}}\sum_{j=1}^{k}\tilde{\lambda}_{j,k}\frac{1}{n}\sum_{i=1}^{n}|m(X_{i},\theta_{1})m(X_{i},\theta_{2})|^{\gamma} \\
&\qquad\qquad\times(U^{*})^{\prime\prime}\left(b_{k}(X_{i},\theta_{1},B)\right)|B(X_{i})-B_{j}(X_{i})||B(X_{i})-\widetilde{B}(X_{i})| \\
&\leq(1-\pi_{k+1})\left\{\widehat{L}(\mu_{k}(\theta_{1}))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_{2}\right)\right)\right\}+\pi_{k+1}\varepsilon\\
&\quad+\pi_{k+1}\|\theta_{1}-\theta_{2}\|_{2}\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}\sup_{\beta\in[0,1]}\left\|\left\{1+\gamma m(x,\theta)^{\gamma-1}B(x)\right\}\left.\frac{\partial}{\partial\theta}m(x,\theta)\right|_{\theta=\theta_{2}+\beta(\theta_{1}-\theta_{2})}\right\|_{2} \\
&\qquad\times\frac{1}{n}\sum_{i=1}^{n}\max\left\{\left|Y_{i}-u^{-1}\left(T\left(\theta_{1},\sup_{B\in\mathcal{D}}B\right)(X_{i})\right)\right|,\left|Y_{i}-u^{-1}\left(T\left(\theta_{1},\inf_{B\in\mathcal{D}}B\right)(X_{i})\right)\right|\right\} \\
&\quad+2\pi_{k+1}^{2}\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}|B(x)|\|\theta_{1}-\theta_{2}\|_{2} \\
&\qquad\times\sup_{x\in\mathcal{D}}|m(x,\theta_{1})|^{\gamma}\sum_{B\in\mathcal{D}}\tilde{\lambda}_{B}\sup_{\beta\in[0,1]}(U^{*})^{\prime\prime}\left(C\left(\beta|T(\theta_{1},B),T(\theta_{1},\xi_{k})(x)\right)\right) \\
&\qquad\times\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}\sup_{\beta\in[0,1]}\left\|\left\{1+\gamma m(x,\theta)^{\gamma-1}B(x)\right\}\left.\frac{\partial}{\partial\theta}m(x,\theta)\right|_{\theta=\theta_{2}+\beta(\theta_{1}-\theta_{2})}\right\|_{2} \\
&\quad+4\pi_{k+1}^{2}\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}|B(x)|^{2}\sup_{x\in\mathcal{D}}|m(x,\theta_{1})m(x,\theta_{2})|^{\gamma}\sup_{\Lambda\in\mathcal{W}}\sup_{x\in\mathcal{D}}(U^{*})^{\prime\prime}\left(u(\mu(x,\Lambda,\theta_{1}))\right)^{2} \\
&\qquad\times\sum_{B\in\mathcal{D}}\tilde{\lambda}_{B}\sup_{\beta\in[0,1]}(U^{*})^{\prime\prime}\left(C\left(\beta|T(\theta_{1},B),T(\theta_{1},\xi_{k})(x)\right)\right) \\
&\leq(1-\pi_{k+1})\left\{\widehat{L}(\mu_{k}(\theta_{1}))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_{2}\right)\right)\right\}+\pi_{k+1}\varepsilon\\
&\quad+\pi_{k+1}^{2}\left[4\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}|B(x)|^{2}\sup_{x\in\mathcal{D}}|m(x,\theta_{1})m(x,\theta_{2})|^{\gamma}\sup_{\Lambda\in\mathcal{W}}\sup_{x\in\mathcal{D}}(U^{*})^{\prime\prime}\left(u(\mu(x,\Lambda,\theta_{1}))\right)^{2}\right. \\
&\qquad\qquad+2\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}|B(x)|\|\theta_{1}-\theta_{2}\|_{2}\sup_{x\in\mathcal{D}}|m(x,\theta_{1})|^{\gamma}\sup_{\Lambda\in\mathcal{W}}\sup_{x\in\mathcal{D}}(U^{*})^{\prime\prime}\left(u(\mu(x,\Lambda,\theta_{1}))\right)^{2} \\
&\qquad\qquad\left.\times\sup_{B\in\mathcal{D}}\sup_{x\in\mathcal{D}}\sup_{\beta\in[0,1]}\left\|\left\{1+\gamma m(x,\theta)^{\gamma-1}B(x)\right\}\left.\frac{\partial}{\partial\theta}m(x,\theta)\right|_{\theta=\theta_{2}+\beta(\theta_{1}-\theta_{2})}\right\|_{2}\right] \\
&+\pi_{k+1}\lVert\theta_1-\theta_2\rVert_2\sup_{B\in\mathcal{D}}\sup_{x\in\mathbb{D}}\sup_{\beta\in[0,1]}\left\lVert\left\{1+\gamma m(x,\theta)^{\gamma-1}B(x)\right\}\left.\frac{\partial}{\partial\theta}m(x,\theta)\right|_{\theta=\theta_2+\beta(\theta_1-\theta_2)}\right\rVert_2 \\
&\times\frac{1}{n}\sum_{i=1}^{n}\sup_{B\in\mathcal{D}}\left|Y_i-u^{-1}\left(T(\theta_1,B)(X_i)\right)\right| \\
&=(1-\pi_{k+1})\left\{\widehat{L}(\mu_k(\theta_1))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)\right\} \\
&+\pi_{k+1}^{2}\left\{A_1(\theta_1,\theta_2)+\lVert\theta_1-\theta_2\rVert_2 A_3(\theta_1,\theta_2)\right\}+\pi_{k+1}\lVert\theta_1-\theta_2\rVert_2 A_2(\theta_1,\theta_2)+\pi_{k+1}\varepsilon.
\end{aligned} × B ∈ D ∑ λ ~ B j = 1 ∑ k λ ~ j , k n 1 i = 1 ∑ n ∣ m ( x , θ 1 ) ∣ γ ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) ∣ B ( X i ) − B j ( X i ) ∣ + π k + 1 2 B ∈ D ∑ λ ~ B B ∈ D ∑ λ ~ B j = 1 ∑ k λ ~ j , k n 1 i = 1 ∑ n ∣ m ( X i , θ 1 ) m ( X i , θ 2 ) ∣ γ × ( U ∗ ) ′′ ( b k ( X i , θ 1 , B ) ) ∣ B ( X i ) − B j ( X i ) ∣∣ B ( X i ) − B ( X i ) ∣ ≤ ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε + π k + 1 ∥ θ 1 − θ 2 ∥ 2 B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] sup { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ θ ∂ m ( x , θ ) θ = θ 2 + β ( θ 1 − θ 2 ) 2 × n 1 i = 1 ∑ n max { Y i − u − 1 ( T ( θ 1 , B ∈ D sup B ) ( X i ) ) , Y i − u − 1 ( T ( θ 1 , B ∈ D inf B ) ( X i ) ) } + 2 π k + 1 2 B ∈ D sup x ∈ D sup ∣ B ( x ) ∣∥ θ 1 − θ 2 ∥ 2 × x ∈ D sup ∣ m ( x , θ 1 ) ∣ γ B ∈ D ∑ λ ~ B β ∈ [ 0 , 1 ] sup ( U ∗ ) ′′ ( C ( β ∣ T ( θ 1 , B ) , T ( θ 1 , ξ k ) ( x ) ) ) × B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] sup { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ θ ∂ m ( x , θ ) θ = θ 2 + β ( θ 1 − θ 2 ) 2 + 4 π k + 1 2 B ∈ D sup x ∈ D sup ∣ B ( x ) ∣ 2 x ∈ D sup ∣ m ( x , θ 1 ) m ( x , θ 2 ) ∣ γ Λ ∈ W sup x ∈ D sup ( U ∗ ) ′′ ( u ( μ ( x , Λ , θ 1 )) ) 2 × B ∈ D ∑ λ ~ B β ∈ [ 0 , 1 ] sup ( U ∗ ) ′′ ( C ( β ∣ T ( θ 1 , B ) , T ( θ 1 , ξ k ) ( x ) ) ) ≤ ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 ε + π k + 1 2 [ 4 B ∈ D sup x ∈ D sup ∣ B ( x ) ∣ 2 x ∈ D sup ∣ m ( x , θ 1 ) m ( x , θ 2 ) ∣ γ Λ ∈ W sup x ∈ D sup ( U ∗ ) ′′ ( u ( μ ( x , Λ , θ 1 )) ) 2 + 2 B ∈ D sup x ∈ D sup ∣ B ( x ) ∣∥ θ 1 − θ 2 ∥ 2 x ∈ D sup ∣ m ( x , θ 1 ) ∣ γ Λ ∈ W sup x ∈ D sup ( U ∗ ) ′′ ( u ( μ ( x , Λ , θ 1 )) ) 2 × B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] sup { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ θ ∂ m ( x , θ ) θ = θ 2 + β ( θ 1 − θ 2 ) 2 ] + π k + 1 ∥ θ 1 − θ 2 ∥ 2 B ∈ D sup x ∈ D sup β ∈ [ 0 , 1 ] sup { 1 + γ m ( x , θ ) γ − 1 B ( x ) } ∂ θ ∂ m ( x , θ ) θ = θ 2 + β ( θ 1 − θ 2 ) 2 × n 1 i = 1 ∑ n B ∈ D sup Y i − u − 1 ( T ( θ 1 , B ) ( X i ) ) = ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 2 { A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) } + π k + 1 ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + π k + 1 ε .
Proof of Lemma A.2
When k = 1 k=1 k = 1 , the inequalities τ 2 / ( 1 + τ ) = τ − 1 + 1 / ( 1 + τ ) ≥ 1 \tau^2/(1+\tau)=\tau-1+1/(1+\tau)\ge1 τ 2 / ( 1 + τ ) = τ − 1 + 1/ ( 1 + τ ) ≥ 1 and (A.2) with ℓ = 0 \ell=0 ℓ = 0 imply that
L ^ ( μ 1 ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) ≤ a + b ≤ τ 2 1 + τ a + b . (36) \widehat{L}(\mu_1(\theta_1))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)\le a+b\le\frac{\tau^2}{1+\tau}a+b. \tag*{(36)} L ( μ 1 ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) ≤ a + b ≤ 1 + τ τ 2 a + b . (36)
If (A.3) holds with arbitrary k ∈ N k\in\mathbb{N} k ∈ N , then
L ^ ( μ k + 1 ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) ≤ ( 1 − π k + 1 ) { L ^ ( μ k ( θ 1 ) ) − L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) } + π k + 1 2 a + π k + 1 b ≤ ( 1 − π k + 1 ) { τ 2 k + τ a + b } + π k + 1 2 a + π k + 1 b = π k + 1 2 a + ( 1 − π k + 1 ) τ 2 k + τ a + b = { τ 2 ( k + τ ) 2 + k k + τ τ 2 k + τ } a + b = k + 1 ( k + τ ) 2 τ 2 a + b = 1 k + 1 + τ ⋅ ( k + τ ) 2 − ( k + τ ) 2 + ( k + 1 ) ( k + 1 + τ ) ( k + τ ) 2 τ 2 a + b = 1 k + 1 + τ ⋅ { 1 − τ 2 − 3 + ( k − 1 ) ( τ − 2 ) ( k + τ ) 2 } τ 2 a + b ≤ τ 2 k + 1 + τ a + b , \begin{aligned}
\widehat{L}(\mu_{k+1}(\theta_1))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)
&\le(1-\pi_{k+1})\left\{\widehat{L}(\mu_k(\theta_1))-\widehat{L}\left(\mu\left(\cdot,\widetilde{\Lambda},\theta_2\right)\right)\right\}+\pi_{k+1}^{2}a+\pi_{k+1}b\\
&\le(1-\pi_{k+1})\left\{\frac{\tau^2}{k+\tau}a+b\right\}+\pi_{k+1}^{2}a+\pi_{k+1}b\\
&=\pi_{k+1}^{2}a+(1-\pi_{k+1})\frac{\tau^2}{k+\tau}a+b\\
&=\left\{\frac{\tau^2}{(k+\tau)^2}+\frac{k}{k+\tau}\frac{\tau^2}{k+\tau}\right\}a+b\\
&=\frac{k+1}{(k+\tau)^2}\tau^2a+b\\
&=\frac{1}{k+1+\tau}\cdot\frac{(k+\tau)^2-(k+\tau)^2+(k+1)(k+1+\tau)}{(k+\tau)^2}\tau^2a+b\\
&=\frac{1}{k+1+\tau}\cdot\left\{1-\frac{\tau^2-3+(k-1)(\tau-2)}{(k+\tau)^2}\right\}\tau^2a+b\\
&\le\frac{\tau^2}{k+1+\tau}a+b,
\end{aligned} L ( μ k + 1 ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) ≤ ( 1 − π k + 1 ) { L ( μ k ( θ 1 )) − L ( μ ( ⋅ , Λ , θ 2 ) ) } + π k + 1 2 a + π k + 1 b ≤ ( 1 − π k + 1 ) { k + τ τ 2 a + b } + π k + 1 2 a + π k + 1 b = π k + 1 2 a + ( 1 − π k + 1 ) k + τ τ 2 a + b = { ( k + τ ) 2 τ 2 + k + τ k k + τ τ 2 } a + b = ( k + τ ) 2 k + 1 τ 2 a + b = k + 1 + τ 1 ⋅ ( k + τ ) 2 ( k + τ ) 2 − ( k + τ ) 2 + ( k + 1 ) ( k + 1 + τ ) τ 2 a + b = k + 1 + τ 1 ⋅ { 1 − ( k + τ ) 2 τ 2 − 3 + ( k − 1 ) ( τ − 2 ) } τ 2 a + b ≤ k + 1 + τ τ 2 a + b ,
so that induction gives (A.3).
Proof of Lemma 4.1
In the inequality (A.1) in Lemma A.2, we put a = A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) a = A_1(\theta_1,\theta_2) + \lVert\theta_1-\theta_2\rVert_2 A_3(\theta_1,\theta_2) a = A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) and b = ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + ε b = \lVert\theta_1-\theta_2\rVert_2 A_2(\theta_1,\theta_2) + \varepsilon b = ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + ε based on (A.1). Then, for any Λ ~ ∈ W \widetilde{\Lambda} \in\mathcal{W} Λ ∈ W ,
L ^ ( μ ( ⋅ , Λ , θ 1 ) ) ≤ L ^ ( μ ( ⋅ , Λ ~ , θ 2 ) ) + τ 2 I + τ { A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) } \widehat{L}\left(\mu(\cdot,\Lambda,\theta_1)\right) \leq\widehat{L}\left(\mu(\cdot,\widetilde{\Lambda},\theta_2)\right) + \frac{\tau^2}{I+\tau}\left\{A_1(\theta_1,\theta_2) + \lVert\theta_1-\theta_2\rVert_2 A_3(\theta_1,\theta_2)\right\} L ( μ ( ⋅ , Λ , θ 1 ) ) ≤ L ( μ ( ⋅ , Λ , θ 2 ) ) + I + τ τ 2 { A 1 ( θ 1 , θ 2 ) + ∥ θ 1 − θ 2 ∥ 2 A 3 ( θ 1 , θ 2 ) }
+ ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + ε = L ( Λ ~ ) (37) \begin{aligned}
{}+ \lVert\theta_1-\theta_2\rVert_2 A_2(\theta_1,\theta_2) + \varepsilon\\
&= L\left(\widetilde{\Lambda}\right) \tag*{(37)}
\end{aligned} + ∥ θ 1 − θ 2 ∥ 2 A 2 ( θ 1 , θ 2 ) + ε = L ( Λ ) (37)
This inequality (A.10) means that L ^ ( μ ( ⋅ , Λ , θ 1 ) ) \widehat{L}\left(\mu(\cdot,\Lambda,\theta_1)\right) L ( μ ( ⋅ , Λ , θ 1 ) ) is a lower bound of the set { L ( Λ ~ ) ∣ Λ ~ ∈ W } ⊂ R \left\{L\left(\widetilde{\Lambda}\right)\mid\widetilde{\Lambda}\in\mathcal{W}\right\}\subset\mathbb{R} { L ( Λ ) ∣ Λ ∈ W } ⊂ R . Therefore, Lemma 4.1 holds.
Proof of Lemma 4.2
Pick Λ ~ ∈ W \widetilde{\Lambda}\in\mathcal{W} Λ ∈ W . The assumption (4.1) implies that
L ^ ( μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) ≤ L ^ ( μ ( ⋅ , Λ , θ 2 ( X ) ) ) + δ ( X ) . (38) \widehat{L}\left(\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right) \leq\widehat{L}\left(\mu\left(\cdot,\Lambda,\theta_2(\mathbb{X})\right)\right)+\delta(\mathbb{X}). \tag*{(38)} L ( μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) ≤ L ( μ ( ⋅ , Λ , θ 2 ( X ) ) ) + δ ( X ) . (38)
From this inequality (A.11) and the fact
D U ( μ , μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) − L ^ ( μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) = ∫ R × R d { ℓ ( y , x ∣ μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) − ℓ ( y , x ∣ μ ) } f ( y , x ) d y d x \begin{aligned}
D_U\left(\mu,\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)-\widehat{L}\left(\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right) \\
&= \int_{\mathbb{R}\times\mathbb{R}^d}\left\{\ell(y,x\mid\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right))-\ell(y,x\mid\mu)\right\}f(y,x)\,dy\,dx
\end{aligned} D U ( μ , μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) − L ( μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) = ∫ R × R d { ℓ ( y , x ∣ μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) − ℓ ( y , x ∣ μ ) } f ( y , x ) d y d x
− 1 n ∑ i = 1 n ℓ ( Y i , X i ∣ μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) = − E n ( μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) − ∫ R × R d ℓ ( y , x ∣ μ ) f ( y , x ) d y d x , \begin{aligned}
-\frac{1}{n}\sum_{i=1}^{n}\ell\left(Y_i,X_i\mid\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right) \\
&=-\mathcal{E}_n\left(\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)-\int_{\mathbb{R}\times\mathbb{R}^d}\ell(y,x\mid\mu)f(y,x)\,dy\,dx,
\end{aligned} − n 1 i = 1 ∑ n ℓ ( Y i , X i ∣ μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) = − E n ( μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) − ∫ R × R d ℓ ( y , x ∣ μ ) f ( y , x ) d y d x ,
it follows that
D U ( μ , μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) D_U\left(\mu,\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right) D U ( μ , μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) )
= D U ( μ , μ ( ⋅ , Λ ~ , θ 2 ( X ) ) ) − { D U ( μ , μ ( ⋅ , Λ ^ , θ 2 ( X ) ) ) − L ~ ( μ ( ⋅ , Λ ^ ( X ) , θ 1 ( X ) ) ) } + D U ( μ , μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) − L ~ ( μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) ≤ D U ( μ , μ ( ⋅ , Λ ~ , θ 2 ( X ) ) ) − { D U ( μ , μ ( ⋅ , Λ ^ , θ 2 ( X ) ) ) − L ~ ( μ ( ⋅ , Λ ~ ( X ) , θ 2 ( X ) ) ) } + D U ( μ , μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) − L ~ ( μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) + δ ( X ) = D U ( μ , μ ( ⋅ , Λ ~ , θ 2 ( X ) ) ) + E n ( μ ( ⋅ , Λ ~ ( X ) , θ 2 ( X ) ) ) − E n ( μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) + δ ( X ) ≤ D U ( μ , μ ( ⋅ , Λ ~ , θ 2 ( X ) ) ) + sup Λ ∈ W E n ( μ ( ⋅ , Λ , θ 2 ( X ) ) ) + sup Λ ∈ W [ − E n ( μ ( ⋅ , Λ , θ 1 ( X ) ) ) ] + δ ( X ) = D ( Λ ~ ) . (39) \begin{aligned}
={}&D_U\left(\mu,\mu\left(\cdot,\widetilde{\Lambda},\theta_2(\mathbb{X})\right)\right)-\left\{D_U\left(\mu,\mu\left(\cdot,\widehat{\Lambda},\theta_2(\mathbb{X})\right)\right)-\widetilde{L}\left(\mu\left(\cdot,\widehat{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)\right\}\\
&+D_U\left(\mu,\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)-\widetilde{L}\left(\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)\\
\le{}&D_U\left(\mu,\mu\left(\cdot,\widetilde{\Lambda},\theta_2(\mathbb{X})\right)\right)-\left\{D_U\left(\mu,\mu\left(\cdot,\widehat{\Lambda},\theta_2(\mathbb{X})\right)\right)-\widetilde{L}\left(\mu\left(\cdot,\widetilde{\Lambda}(\mathbb{X}),\theta_2(\mathbb{X})\right)\right)\right\}\\
&+D_U\left(\mu,\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)-\widetilde{L}\left(\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)+\delta(\mathbb{X})\\
={}&D_U\left(\mu,\mu\left(\cdot,\widetilde{\Lambda},\theta_2(\mathbb{X})\right)\right)+\mathcal{E}_n\left(\mu\left(\cdot,\widetilde{\Lambda}(\mathbb{X}),\theta_2(\mathbb{X})\right)\right)-\mathcal{E}_n\left(\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)+\delta(\mathbb{X})\\
\le{}&D_U\left(\mu,\mu\left(\cdot,\widetilde{\Lambda},\theta_2(\mathbb{X})\right)\right)+\sup_{\Lambda\in\mathcal{W}}\mathcal{E}_n\left(\mu\left(\cdot,\Lambda,\theta_2(\mathbb{X})\right)\right)+\sup_{\Lambda\in\mathcal{W}}\left[-\mathcal{E}_n\left(\mu\left(\cdot,\Lambda,\theta_1(\mathbb{X})\right)\right)\right]\\
&+\delta(\mathbb{X})\\
={}&D\left(\widetilde{\Lambda}\right). \tag*{(39)}
\end{aligned} = ≤ = ≤ = D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) − { D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) − L ( μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) } + D U ( μ , μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) − L ( μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) − { D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) − L ( μ ( ⋅ , Λ ( X ) , θ 2 ( X ) ) ) } + D U ( μ , μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) − L ( μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) + δ ( X ) D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) + E n ( μ ( ⋅ , Λ ( X ) , θ 2 ( X ) ) ) − E n ( μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) + δ ( X ) D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) + Λ ∈ W sup E n ( μ ( ⋅ , Λ , θ 2 ( X ) ) ) + Λ ∈ W sup [ − E n ( μ ( ⋅ , Λ , θ 1 ( X ) ) ) ] + δ ( X ) D ( Λ ) . (39)
This inequality (A.12) shows that D U ( μ , μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) D_U\left(\mu,\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right) D U ( μ , μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) is a lower bound of the following set { D ( Λ ~ ) ∣ Λ ~ ∈ W } \left\{D\left(\widetilde{\Lambda}\right)\mid\widetilde{\Lambda}\in\mathcal{W}\right\} { D ( Λ ) ∣ Λ ∈ W } . Therefore,
D U ( μ , μ ( ⋅ , Λ ‾ ( X ) , θ 1 ( X ) ) ) ≤ inf Λ ∈ W D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) + sup Λ ∈ W E n ( μ ( ⋅ , Λ , θ 2 ( X ) ) ) + sup Λ ∈ W [ − E n ( μ ( ⋅ , Λ , θ 1 ( X ) ) ) ] + δ ( X ) . D_U\left(\mu,\mu\left(\cdot,\overline{\Lambda}(\mathbb{X}),\theta_1(\mathbb{X})\right)\right)\leq\inf_{\Lambda\in\mathcal{W}}D_U\left(\mu,\mu\left(\cdot,\Lambda,\theta_2(\mathbb{X})\right)\right)+\sup_{\Lambda\in\mathcal{W}}\mathcal{E}_n\left(\mu\left(\cdot,\Lambda,\theta_2(\mathbb{X})\right)\right)
+\sup_{\Lambda\in\mathcal{W}}\left[-\mathcal{E}_n\left(\mu\left(\cdot,\Lambda,\theta_1(\mathbb{X})\right)\right)\right]+\delta(\mathbb{X}). D U ( μ , μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) ) ≤ Λ ∈ W inf D U ( μ , μ ( ⋅ , Λ , θ 2 ( X ) ) ) + Λ ∈ W sup E n ( μ ( ⋅ , Λ , θ 2 ( X ) ) ) + Λ ∈ W sup [ − E n ( μ ( ⋅ , Λ , θ 1 ( X ) ) ) ] + δ ( X ) .
Lemma 4.2 is obtained by taking expectation.
Lemmas for Theorem 4.1 and their proofs
In this subsection, we arrange and prove lemmas used in the next subsection, when investigating the second term of the bound in (5) . Let σ 1 , … , σ n \sigma_1,\ldots,\sigma_n σ 1 , … , σ n be Rademacher random variables, satisfying P ( σ i = 1 ) = P ( σ i = − 1 ) = 1 / 2 \mathbb{P}(\sigma_i=1)=\mathbb{P}(\sigma_i=-1)=1/2 P ( σ i = 1 ) = P ( σ i = − 1 ) = 1/2 , independent of the observations X \mathbb{X} X . We also denote the expectations taken with respect to σ = [ σ 1 ⋯ σ n ] \boldsymbol{\sigma}=[\sigma_1\cdots\sigma_n] σ = [ σ 1 ⋯ σ n ] and the observations X \mathbb{X} X by E σ \mathbb{E}_{\boldsymbol{\sigma}} E σ and E X \mathbb{E}_{\mathbb{X}} E X respectively. The following lemma will be used to bound the term in (5) with what is called a Rademacher complexity.
Lemma A.3. (symmetrization) For a function class F \mathcal{F} F and both α = 0 , 1 \alpha= 0,1 α = 0 , 1 ,
E X [ sup φ ∈ F [ ( − 1 ) α { 1 n ∑ i = 1 n φ ( Y i , X i ) − ∫ R × R d φ ( y , x ) f ( y , x ) d y d x } ] ] \mathbb{E}_{\mathbf{X}}\left[\sup_{\varphi\in\mathcal{F}}\left[(-1)^\alpha\left\{\frac{1}{n}\sum_{i=1}^{n}\varphi(Y_i,X_i)-\int_{\mathbb{R}\times\mathbb{R}^d}\varphi(y,x)f(y,x)\,\mathrm{d}y\mathrm{d}x\right\}\right]\right] E X [ φ ∈ F sup [ ( − 1 ) α { n 1 i = 1 ∑ n φ ( Y i , X i ) − ∫ R × R d φ ( y , x ) f ( y , x ) d y d x } ] ] ≤ 2 E X [ E σ [ sup φ ∈ F [ 1 n ∑ i = 1 n σ i ⋅ φ ( Y i , X i ) ] ] ] . \leq2\mathbb{E}_{\mathbf{X}}\left[\mathbb{E}_{\sigma}\left[\sup_{\varphi\in\mathcal{F}}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot\varphi(Y_i,X_i)\right]\right]\right]. ≤ 2 E X [ E σ [ φ ∈ F sup [ n 1 i = 1 ∑ n σ i ⋅ φ ( Y i , X i ) ] ] ] .
Proof. Let X ′ = { ( Y 1 ′ , X 1 ′ ) , … , ( Y n ′ , X n ′ ) } \mathbf{X}'=\{(Y'_1,X'_1),\ldots,(Y'_n,X'_n)\} X ′ = {( Y 1 ′ , X 1 ′ ) , … , ( Y n ′ , X n ′ )} be independent copies of X \mathbf{X} X . Then
E X [ sup φ ∈ F [ ( − 1 ) α { 1 n ∑ i = 1 n φ ( Y i , X i ) − ∫ R × R d φ ( y , x ) f ( y , x ) d y d x } ] ] = E X [ sup φ ∈ F [ ( − 1 ) α { 1 n ∑ i = 1 n φ ( Y i , X i ) − E X ′ [ 1 n ∑ i = 1 n φ ( Y i ′ , X i ′ ) ] } ] ] ≤ E X , X ′ [ sup φ ∈ F [ ( − 1 ) α 1 n ∑ i = 1 n { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } ] ] . (40) \begin{aligned}
\mathbb{E}_{\mathbf{X}}\left[\sup_{\varphi\in\mathcal{F}}\left[(-1)^\alpha\left\{\frac{1}{n}\sum_{i=1}^{n}\varphi(Y_i,X_i)-\int_{\mathbb{R}\times\mathbb{R}^d}\varphi(y,x)f(y,x)\,\mathrm{d}y\mathrm{d}x\right\}\right]\right] \\
&=\mathbb{E}_{\mathbf{X}}\left[\sup_{\varphi\in\mathcal{F}}\left[(-1)^\alpha\left\{\frac{1}{n}\sum_{i=1}^{n}\varphi(Y_i,X_i)-\mathbb{E}_{\mathbf{X}'}\left[\frac{1}{n}\sum_{i=1}^{n}\varphi(Y'_i,X'_i)\right]\right\}\right]\right] \\
&\leq\mathbb{E}_{\mathbf{X},\mathbf{X}'}\left[\sup_{\varphi\in\mathcal{F}}\left[(-1)^\alpha\frac{1}{n}\sum_{i=1}^{n}\left\{\varphi(Y_i,X_i)-\varphi(Y'_i,X'_i)\right\}\right]\right]. \tag*{(40)}
\end{aligned} E X [ φ ∈ F sup [ ( − 1 ) α { n 1 i = 1 ∑ n φ ( Y i , X i ) − ∫ R × R d φ ( y , x ) f ( y , x ) d y d x } ] ] = E X [ φ ∈ F sup [ ( − 1 ) α { n 1 i = 1 ∑ n φ ( Y i , X i ) − E X ′ [ n 1 i = 1 ∑ n φ ( Y i ′ , X i ′ ) ] } ] ] ≤ E X , X ′ [ φ ∈ F sup [ ( − 1 ) α n 1 i = 1 ∑ n { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } ] ] . (40)
For the term (A.13), we shall show that the following equality
E X , X ′ [ sup φ ∈ F [ ( − 1 ) α 1 n ∑ i = 1 n { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } ] ] = E X , X ′ [ E σ [ sup φ ∈ F 1 n ∑ i = 1 n σ i ⋅ { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } ] ] . (41) \begin{aligned}
\mathbb{E}_{\mathbf{X},\mathbf{X}'}\left[\sup_{\varphi\in\mathcal{F}}\left[(-1)^\alpha\frac{1}{n}\sum_{i=1}^{n}\left\{\varphi(Y_i,X_i)-\varphi(Y'_i,X'_i)\right\}\right]\right] \\
&=\mathbb{E}_{\mathbf{X},\mathbf{X}'}\left[\mathbb{E}_{\sigma}\left[\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot\left\{\varphi(Y_i,X_i)-\varphi(Y'_i,X'_i)\right\}\right]\right]. \tag*{(41)}
\end{aligned} E X , X ′ [ φ ∈ F sup [ ( − 1 ) α n 1 i = 1 ∑ n { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } ] ] = E X , X ′ [ E σ [ φ ∈ F sup n 1 i = 1 ∑ n σ i ⋅ { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } ] ] . (41)
holds: we define
E j [ 1 ] = E X , X ′ [ E [ σ 1 ⋯ σ j − 1 ] [ sup φ ∈ F 1 n [ ∑ i = 1 j − 1 σ i { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } + ( − 1 ) α ∑ i = j n { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } ] ] ] = E X , X ′ [ E [ σ 1 ⋯ σ j − 1 ] [ sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] ] ] , \begin{aligned}
\mathbb{E}^{[1]}_j=\mathbb{E}_{\mathbf{X},\mathbf{X}'}\left[\mathbb{E}_{[\sigma_1\cdots\sigma_{j-1}]}\left[\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\sum_{i=1}^{j-1}\sigma_i\left\{\varphi(Y_i,X_i)-\varphi(Y'_i,X'_i)\right\}+(-1)^\alpha\sum_{i=j}^{n}\left\{\varphi(Y_i,X_i)-\varphi(Y'_i,X'_i)\right\}\right]\right]\right] \\
&=\mathbb{E}_{\mathbf{X},\mathbf{X}'}\left[\mathbb{E}_{[\sigma_1\cdots\sigma_{j-1}]}\left[\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^\alpha\left\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\right\}\right]\right]\right],
\end{aligned} E j [ 1 ] = E X , X ′ [ E [ σ 1 ⋯ σ j − 1 ] [ φ ∈ F sup n 1 [ i = 1 ∑ j − 1 σ i { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } + ( − 1 ) α i = j ∑ n { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } ] ] ] = E X , X ′ [ E [ σ 1 ⋯ σ j − 1 ] [ φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] ] ] ,
Φ ( j ) = ∑ i = 1 j − 1 σ i { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } + ( − 1 ) α ∑ i = j + 1 n { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } \Phi^{(j)}=\sum_{i=1}^{j-1}\sigma_i\left\{\varphi(Y_i,X_i)-\varphi(Y'_i,X'_i)\right\}+(-1)^\alpha\sum_{i=j+1}^{n}\left\{\varphi(Y_i,X_i)-\varphi(Y'_i,X'_i)\right\} Φ ( j ) = i = 1 ∑ j − 1 σ i { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) } + ( − 1 ) α i = j + 1 ∑ n { φ ( Y i , X i ) − φ ( Y i ′ , X i ′ ) }
for j = 1 , 2 , … , n + 1 j=1,2,\ldots,n+1 j = 1 , 2 , … , n + 1 , where we regard E 1 [ 1 ] \mathbb{E}^{[1]}_1 E 1 [ 1 ] as the left-hand side of (A.14), and E n + 1 [ 1 ] \mathbb{E}^{[1]}_{n+1} E n + 1 [ 1 ] coincides with right-hand side of (A.14). Then we can prove the sequence ( E j [ 1 ] ) j = 1 n + 1 \left(\mathbb{E}^{[1]}_j\right)_{j=1}^{n+1} ( E j [ 1 ] ) j = 1 n + 1 to be constant for j = 1 , 2 , … , n + 1 j=1,2,\ldots,n+1 j = 1 , 2 , … , n + 1 , since the equality
E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ sup φ ∈ F [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ sup φ ∈ F [ Φ ( j ) + ( − 1 ) α { φ ( Y j ′ , X j ′ ) − φ ( Y j , X j ) } ] ] . \begin{aligned}
\mathbb{E}_{X,X',[ \sigma_1\cdots\sigma_{j-1}]}\left[\sup_{\varphi\in\mathcal{F}}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right]
&=\mathbb{E}_{X,X',[ \sigma_1\cdots\sigma_{j-1}]}\left[\sup_{\varphi\in\mathcal{F}}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y'_j,X'_j)-\varphi(Y_j,X_j)\}\right]\right].
\end{aligned} E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ φ ∈ F sup [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ φ ∈ F sup [ Φ ( j ) + ( − 1 ) α { φ ( Y j ′ , X j ′ ) − φ ( Y j , X j )} ] ] .
holds for j = 1 , 2 , … , n + 1 j=1,2,\ldots,n+1 j = 1 , 2 , … , n + 1 by the transformation from [ Y j X j Y j ′ X j ′ ] \left[Y_j\ X_j\ Y'_j\ X'_j\right] [ Y j X j Y j ′ X j ′ ] to [ Y j ′ X j ′ Y j X j ] \left[Y'_j\ X'_j\ Y_j\ X_j\right] [ Y j ′ X j ′ Y j X j ] , and hence
E j [ 1 ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ 1 2 sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] + 1 2 sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ 1 2 sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] + E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ 1 2 sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α { φ ( Y j ′ , X j ′ ) − φ ( Y j , X j ) } ] ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ 1 2 sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] + 1 2 sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α + 1 { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ P ( σ j = ( − 1 ) α ) sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] + P ( σ j = ( − 1 ) α + 1 ) sup φ ∈ F 1 n [ Φ ( j ) + ( − 1 ) α + 1 { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ ) } ] ] . \begin{aligned}
E_j^{[1]}
&=\mathbb{E}_{X,X',[ \sigma_1\cdots\sigma_{j-1}]}\left[\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right]\\
&=\mathbb{E}_{X,X',[ \sigma_1\cdots\sigma_{j-1}]}\left[\frac{1}{2}\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right.\\
&\qquad\left.+\frac{1}{2}\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right]\\
&=\mathbb{E}_{X,X',[ \sigma_1\cdots\sigma_{j-1}]}\left[\frac{1}{2}\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right.\\
&\qquad\left.+\mathbb{E}_{X,X',[ \sigma_1\cdots\sigma_{j-1}]}\left[\frac{1}{2}\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y'_j,X'_j)-\varphi(Y_j,X_j)\}\right]\right]\right]\\
&=\mathbb{E}_{X,X',[ \sigma_1\cdots\sigma_{j-1}]}\left[\frac{1}{2}\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right.\\
&\qquad\left.+\frac{1}{2}\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^{\alpha+1}\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right]\\
&=\mathbb{E}_{X,X',[ \sigma_1\cdots\sigma_{j-1}]}\left[\mathbb{P}\left(\sigma_j=(-1)^\alpha\right)\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^\alpha\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right.\\
&\qquad\left.+\mathbb{P}\left(\sigma_j=(-1)^{\alpha+1}\right)\sup_{\varphi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+(-1)^{\alpha+1}\{\varphi(Y_j,X_j)-\varphi(Y'_j,X'_j)\}\right]\right].
\end{aligned} E j [ 1 ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ 2 1 φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] + 2 1 φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ 2 1 φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] + E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ 2 1 φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α { φ ( Y j ′ , X j ′ ) − φ ( Y j , X j )} ] ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ 2 1 φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] + 2 1 φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α + 1 { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] ] = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ P ( σ j = ( − 1 ) α ) φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] + P ( σ j = ( − 1 ) α + 1 ) φ ∈ F sup n 1 [ Φ ( j ) + ( − 1 ) α + 1 { φ ( Y j , X j ) − φ ( Y j ′ , X j ′ )} ] ] .
= E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ E σ j [ sup ϕ ∈ F 1 n [ Φ ( j ) + σ j { ϕ ( Y j , X j ) − ϕ ( Y j ′ , X j ′ ) } ] ] ] = E X , X ′ [ E [ σ 1 ⋯ σ j ] [ sup ϕ ∈ F 1 n [ ∑ i = 1 j σ i { ϕ ( Y i , X i ) − ϕ ( Y i ′ , X i ′ ) } + ( − 1 ) α ∑ i = j + 1 n { ϕ ( Y i , X i ) − ϕ ( Y i ′ , X i ′ ) } ] ] ] = E X , X ′ [ E [ σ 1 ⋯ σ j ] [ sup ϕ ∈ F 1 n [ Φ ( j + 1 ) + ( − 1 ) α { ϕ ( Y j + 1 , X j + 1 ) − ϕ ( Y j + 1 ′ , X j + 1 ′ ) } ] ] ] = E j + 1 [ 1 ] , \begin{aligned}
&= \mathbb{E}_{X,X',[\sigma_1\ \cdots\ \sigma_{j-1}]}\left[\mathbb{E}_{\sigma_j}\left[\sup_{\phi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j)}+\sigma_j\left\{\phi(Y_j,X_j)-\phi(Y'_j,X'_j)\right\}\right]\right]\right]\\
&= \mathbb{E}_{X,X'}\left[\mathbb{E}_{[\sigma_1\ \cdots\ \sigma_j]}\left[\sup_{\phi\in\mathcal{F}}\frac{1}{n}\left[\sum_{i=1}^{j}\sigma_i\left\{\phi(Y_i,X_i)-\phi(Y'_i,X'_i)\right\}\right.\right.\right.\\
&\qquad\qquad\left.\left.\left.+(-1)^\alpha\sum_{i=j+1}^{n}\left\{\phi(Y_i,X_i)-\phi(Y'_i,X'_i)\right\}\right]\right]\right]\\
&= \mathbb{E}_{X,X'}\left[\mathbb{E}_{[\sigma_1\ \cdots\ \sigma_j]}\left[\sup_{\phi\in\mathcal{F}}\frac{1}{n}\left[\Phi^{(j+1)}+(-1)^\alpha\left\{\phi(Y_{j+1},X_{j+1})-\phi(Y'_{j+1},X'_{j+1})\right\}\right]\right]\right]\\
&= \mathbb{E}_{j+1}^{[1]},
\end{aligned} = E X , X ′ , [ σ 1 ⋯ σ j − 1 ] [ E σ j [ ϕ ∈ F sup n 1 [ Φ ( j ) + σ j { ϕ ( Y j , X j ) − ϕ ( Y j ′ , X j ′ ) } ] ] ] = E X , X ′ [ E [ σ 1 ⋯ σ j ] [ ϕ ∈ F sup n 1 [ i = 1 ∑ j σ i { ϕ ( Y i , X i ) − ϕ ( Y i ′ , X i ′ ) } + ( − 1 ) α i = j + 1 ∑ n { ϕ ( Y i , X i ) − ϕ ( Y i ′ , X i ′ ) } ] ] ] = E X , X ′ [ E [ σ 1 ⋯ σ j ] [ ϕ ∈ F sup n 1 [ Φ ( j + 1 ) + ( − 1 ) α { ϕ ( Y j + 1 , X j + 1 ) − ϕ ( Y j + 1 ′ , X j + 1 ′ ) } ] ] ] = E j + 1 [ 1 ] ,
so that the equality (A.14) is shown. From (i) the inequality (A.13), (ii) the equality (A.14) and (iii) the equality E σ [ g ( σ ) ] = E σ [ g ( − σ ) ] \mathbb{E}_{\sigma}[g(\sigma)]=\mathbb{E}_{\sigma}[g(-\sigma)] E σ [ g ( σ )] = E σ [ g ( − σ )] for a function g g g via the transformation from σ \sigma σ to − σ -\sigma − σ , it follows that
E X [ sup ϕ ∈ F [ ( − 1 ) α { 1 n ∑ i = 1 n ϕ ( Y i , X i ) − ∫ R × R d ϕ ( y , x ) f ( y , x ) d y d x } ] ] ≤ E X , X ′ [ sup ϕ ∈ F [ ( − 1 ) α { 1 n ∑ i = 1 n ( ϕ ( Y i , X i ) − ϕ ( Y i ′ , X i ′ ) ) } ] ] = E X , X ′ [ E σ [ sup ϕ ∈ F [ 1 n ∑ i = 1 n σ i { ϕ ( Y i , X i ) − ϕ ( Y i ′ , X i ′ ) } ] ] ] ≤ E X , X ′ [ E σ [ sup ϕ ∈ F [ 1 n ∑ i = 1 n σ i ⋅ ϕ ( Y i , X i ) ] ] ] + E X , X ′ [ E σ [ sup ϕ ∈ F [ 1 n ∑ i = 1 n ( − σ i ) ⋅ ϕ ( Y i ′ , X i ′ ) ] ] ] = 2 E X [ E σ [ sup ϕ ∈ F [ 1 n ∑ i = 1 n σ i ⋅ ϕ ( Y i , X i ) ] ] ] . \begin{aligned}
\mathbb{E}_{X}\left[\sup_{\phi\in\mathcal{F}}\left[(-1)^\alpha\left\{\frac{1}{n}\sum_{i=1}^{n}\phi(Y_i,X_i)-\int_{\mathbb{R}\times\mathbb{R}^{d}}\phi(y,x)f(y,x)\,dy\,dx\right\}\right]\right]\\
\leq\mathbb{E}_{X,X'}\left[\sup_{\phi\in\mathcal{F}}\left[(-1)^\alpha\left\{\frac{1}{n}\sum_{i=1}^{n}\left(\phi(Y_i,X_i)-\phi(Y'_i,X'_i)\right)\right\}\right]\right]\\
= \mathbb{E}_{X,X'}\left[\mathbb{E}_{\sigma}\left[\sup_{\phi\in\mathcal{F}}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\left\{\phi(Y_i,X_i)-\phi(Y'_i,X'_i)\right\}\right]\right]\right]\\
\leq\mathbb{E}_{X,X'}\left[\mathbb{E}_{\sigma}\left[\sup_{\phi\in\mathcal{F}}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot\phi(Y_i,X_i)\right]\right]\right]\\
\qquad+\mathbb{E}_{X,X'}\left[\mathbb{E}_{\sigma}\left[\sup_{\phi\in\mathcal{F}}\left[\frac{1}{n}\sum_{i=1}^{n}(-\sigma_i)\cdot\phi(Y'_i,X'_i)\right]\right]\right]\\
=2\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{\phi\in\mathcal{F}}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot\phi(Y_i,X_i)\right]\right]\right].
\end{aligned} E X [ ϕ ∈ F sup [ ( − 1 ) α { n 1 i = 1 ∑ n ϕ ( Y i , X i ) − ∫ R × R d ϕ ( y , x ) f ( y , x ) d y d x } ] ] ≤ E X , X ′ [ ϕ ∈ F sup [ ( − 1 ) α { n 1 i = 1 ∑ n ( ϕ ( Y i , X i ) − ϕ ( Y i ′ , X i ′ ) ) } ] ] = E X , X ′ [ E σ [ ϕ ∈ F sup [ n 1 i = 1 ∑ n σ i { ϕ ( Y i , X i ) − ϕ ( Y i ′ , X i ′ ) } ] ] ] ≤ E X , X ′ [ E σ [ ϕ ∈ F sup [ n 1 i = 1 ∑ n σ i ⋅ ϕ ( Y i , X i ) ] ] ] + E X , X ′ [ E σ [ ϕ ∈ F sup [ n 1 i = 1 ∑ n ( − σ i ) ⋅ ϕ ( Y i ′ , X i ′ ) ] ] ] = 2 E X [ E σ [ ϕ ∈ F sup [ n 1 i = 1 ∑ n σ i ⋅ ϕ ( Y i , X i ) ] ] ] .
When addressing function classes in the supremum for Rademacher complexity, we will utilize the next three lemmas A.4–A.6:
Lemma A.4. For i = 1 , … , n i=1,\ldots,n i = 1 , … , n , let ℓ X i : R → R \ell_{X_i}:\mathbb{R}\to\mathbb{R} ℓ X i : R → R be functions depending on X X X and satisfying the following condition: there exists L X i > 0 L_{X_i}>0 L X i > 0 such that ∣ ℓ X i ( z 1 ) − ℓ X i ( z 2 ) ∣ ≤ L X i ∣ z 1 − z 2 ∣ \lvert\ell_{X_i}(z_1)-\ell_{X_i}(z_2)\rvert\leq L_{X_i}\lvert z_1-z_2\rvert ∣ ℓ X i ( z 1 ) − ℓ X i ( z 2 )∣ ≤ L X i ∣ z 1 − z 2 ∣ for any z 1 , z 2 ∈ R z_1,z_2\in\mathbb{R} z 1 , z 2 ∈ R . For such ℓ X i \ell_{X_i} ℓ X i and a function class G \mathcal{G} G , the following inequality of random variables holds:
E σ [ sup g ∈ G 1 n ∑ i = 1 n σ i ⋅ ℓ X i ( g ( X i ) ) ] ≤ E σ [ sup g ∈ G 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] . (42) \mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot\ell_{X_i}(g(X_i))\right]\leq\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\right]. \tag*{(42)} E σ [ g ∈ G sup n 1 i = 1 ∑ n σ i ⋅ ℓ X i ( g ( X i )) ] ≤ E σ [ g ∈ G sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] . (42)
Proof. Using the notation
E j [ 2 ] = E σ [ sup g ∈ G 1 n [ ∑ i = 1 j − 1 σ i ⋅ L X i ⋅ g ( X i ) + ∑ i = j n σ i ⋅ ℓ X i ( g ( X i ) ) ] ] = E σ [ sup g ∈ G 1 n [ S X ( j ) ( σ , g ) + σ j ⋅ ℓ X j ( g ( X j ) ) ] ] , \begin{aligned}
E_j^{[2]}=\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\left[\sum_{i=1}^{j-1}\sigma_i\cdot L_{X_i}\cdot g(X_i)+\sum_{i=j}^{n}\sigma_i\cdot\ell_{X_i}(g(X_i))\right]\right] \\
&=\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\left[\mathcal{S}_{X}^{(j)}(\sigma,g)+\sigma_j\cdot\ell_{X_j}(g(X_j))\right]\right],
\end{aligned} E j [ 2 ] = E σ [ g ∈ G sup n 1 [ i = 1 ∑ j − 1 σ i ⋅ L X i ⋅ g ( X i ) + i = j ∑ n σ i ⋅ ℓ X i ( g ( X i )) ] ] = E σ [ g ∈ G sup n 1 [ S X ( j ) ( σ , g ) + σ j ⋅ ℓ X j ( g ( X j )) ] ] ,
S X ( j ) ( σ , g ) = ∑ i = 1 j − 1 σ i ⋅ L X i ⋅ g ( X i ) + ∑ i = j + 1 n σ i ⋅ ℓ X i ( g ( X i ) ) , \mathcal{S}_{X}^{(j)}(\sigma,g)=\sum_{i=1}^{j-1}\sigma_i\cdot L_{X_i}\cdot g(X_i)+\sum_{i=j+1}^{n}\sigma_i\cdot\ell_{X_i}(g(X_i)), S X ( j ) ( σ , g ) = i = 1 ∑ j − 1 σ i ⋅ L X i ⋅ g ( X i ) + i = j + 1 ∑ n σ i ⋅ ℓ X i ( g ( X i )) ,
for j = 1 , 2 , … , n + 1 j=1,2,\ldots,n+1 j = 1 , 2 , … , n + 1 , (A.15) is equivalent to E 1 [ 2 ] ≤ E n + 1 [ 2 ] E_1^{[2]}\leq E_{n+1}^{[2]} E 1 [ 2 ] ≤ E n + 1 [ 2 ] . We shall show E j [ 2 ] ≤ E j + 1 [ 2 ] E_j^{[2]}\leq E_{j+1}^{[2]} E j [ 2 ] ≤ E j + 1 [ 2 ] for j = 1 , 2 , … , n + 1 j=1,2,\ldots,n+1 j = 1 , 2 , … , n + 1 : let E σ ( j ) \mathbb{E}_{\sigma(j)} E σ ( j ) be the expectation taken with respect to [ σ 1 ⋯ σ j − 1 σ j + 1 ⋯ σ n ] [\sigma_1\cdots\sigma_{j-1}\ \sigma_{j+1}\cdots\sigma_n] [ σ 1 ⋯ σ j − 1 σ j + 1 ⋯ σ n ] . Then, for any j = 1 , 2 , … , n + 1 j=1,2,\ldots,n+1 j = 1 , 2 , … , n + 1 , from the following equality of sets
{ S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ~ ) + L X j ⋅ ∣ g ( X j ) − g ~ ( X i ) ∣ | g ∈ G , g ~ ∈ G } = { S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ~ ) + L X j ⋅ { g ( X j ) − g ~ ( X i ) } | g ∈ G , g ~ ∈ G } , \begin{aligned}
\left\{\mathcal{S}_{X}^{(j)}(\sigma,g)+\mathcal{S}_{X}^{(j)}(\sigma,\widetilde{g})+L_{X_j}\cdot\left|g(X_j)-\widetilde{g}(X_i)\right|\middle|g\in\mathcal{G},\widetilde{g}\in\mathcal{G}\right\} \\
&=\left\{\mathcal{S}_{X}^{(j)}(\sigma,g)+\mathcal{S}_{X}^{(j)}(\sigma,\widetilde{g})+L_{X_j}\cdot\left\{g(X_j)-\widetilde{g}(X_i)\right\}\middle|g\in\mathcal{G},\widetilde{g}\in\mathcal{G}\right\},
\end{aligned} { S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ) + L X j ⋅ ∣ g ( X j ) − g ( X i ) ∣ g ∈ G , g ∈ G } = { S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ) + L X j ⋅ { g ( X j ) − g ( X i ) } g ∈ G , g ∈ G } ,
it follows that
E j [ 2 ] = E σ [ sup g ∈ G 1 n ( S X ( j ) ( σ , g ) + σ j ⋅ ℓ X j ( g ( X j ) ) ) ] = E σ ( j ) [ 1 2 sup g ∈ G 1 n ( S X ( j ) ( σ , g ) + ℓ X j ( g ( X j ) ) ) \begin{aligned}
E_j^{[2]}=\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\left(\mathcal{S}_{X}^{(j)}(\sigma,g)+\sigma_j\cdot\ell_{X_j}(g(X_j))\right)\right] \\
&=\mathbb{E}_{\sigma(j)}\left[\frac{1}{2}\sup_{g\in\mathcal{G}}\frac{1}{n}\left(\mathcal{S}_{X}^{(j)}(\sigma,g)+\ell_{X_j}(g(X_j))\right)\right.
\end{aligned} E j [ 2 ] = E σ [ g ∈ G sup n 1 ( S X ( j ) ( σ , g ) + σ j ⋅ ℓ X j ( g ( X j )) ) ] = E σ ( j ) [ 2 1 g ∈ G sup n 1 ( S X ( j ) ( σ , g ) + ℓ X j ( g ( X j )) )
+ 1 2 sup g ~ ∈ G 1 n ( S X ( j ) ( σ , g ~ ) + ( − 1 ) ⋅ ℓ X j ( g ~ ( X i ) ) ) ] = E σ ( j ) [ 1 2 sup g ∈ G , g ~ ∈ G 1 n [ S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ~ ) + { ℓ X j ( g ( X j ) ) − ℓ X j ( g ~ ( X i ) ) } ] ] ≤ E σ ( j ) [ 1 2 sup g ∈ G , g ~ ∈ G 1 n [ S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ~ ) + L X j ⋅ ∣ g ( X j ) − g ~ ( X i ) ∣ ] ] \begin{aligned}
\left.+\frac{1}{2}\sup_{\widetilde{g}\in\mathcal{G}}\frac{1}{n}\left(\mathcal{S}_{X}^{(j)}(\sigma,\widetilde{g})+(-1)\cdot\ell_{X_j}(\widetilde{g}(X_i))\right)\right] \\
&=\mathbb{E}_{\sigma(j)}\left[\frac{1}{2}\sup_{\substack{g\in\mathcal{G},\\\widetilde{g}\in\mathcal{G}}}\frac{1}{n}\left[\mathcal{S}_{X}^{(j)}(\sigma,g)+\mathcal{S}_{X}^{(j)}(\sigma,\widetilde{g})+\left\{\ell_{X_j}(g(X_j))-\ell_{X_j}(\widetilde{g}(X_i))\right\}\right]\right] \\
&\leq\mathbb{E}_{\sigma(j)}\left[\frac{1}{2}\sup_{\substack{g\in\mathcal{G},\\\widetilde{g}\in\mathcal{G}}}\frac{1}{n}\left[\mathcal{S}_{X}^{(j)}(\sigma,g)+\mathcal{S}_{X}^{(j)}(\sigma,\widetilde{g})+L_{X_j}\cdot\left|g(X_j)-\widetilde{g}(X_i)\right|\right]\right]
\end{aligned} + 2 1 g ∈ G sup n 1 ( S X ( j ) ( σ , g ) + ( − 1 ) ⋅ ℓ X j ( g ( X i )) ) ] = E σ ( j ) 2 1 g ∈ G , g ∈ G sup n 1 [ S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ) + { ℓ X j ( g ( X j )) − ℓ X j ( g ( X i )) } ] ≤ E σ ( j ) 2 1 g ∈ G , g ∈ G sup n 1 [ S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ) + L X j ⋅ ∣ g ( X j ) − g ( X i ) ∣ ]
= E σ − j [ 1 2 sup g ∈ G , g ~ ∈ G 1 n [ S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ~ ) + L X j ⋅ { g ( X j ) − g ~ ( X i ) } ] ] = E σ ( j ) [ 1 2 sup g ∈ G 1 n [ S X ( j ) ( σ , g ) + L X j ⋅ g ( X j ) ] + 1 2 sup g ~ ∈ G 1 n [ S X ( j ) ( σ , g ~ ) + ( − 1 ) ⋅ L X j ⋅ g ~ ( X i ) ] ] = E σ [ sup g ∈ G 1 n [ S X ( j ) ( σ , g ) + σ j ⋅ L X j ⋅ g ( X j ) ] ] = E σ [ sup g ∈ G 1 n [ ∑ i = 1 j σ i ⋅ L X i ⋅ g ( X i ) + ∑ i = j + 1 n σ i ⋅ ℓ X i ( g ( X i ) ) ] ] = E σ [ sup g ∈ G 1 n [ S X ( j + 1 ) ( σ , g ) + σ j + 1 ⋅ ℓ X j + 1 ( g ( X j + 1 ) ) ] ] = E j + 1 [ 2 ] . \begin{aligned}
&= \mathbb{E}_{\sigma_{-j}}\left[\frac{1}{2}\sup_{g\in\mathcal{G},\,\widetilde{g}\in\mathcal{G}}\frac{1}{n}\left[\mathcal{S}_{\mathsf{X}}^{(j)}(\sigma,g)+\mathcal{S}_{\mathsf{X}}^{(j)}(\sigma,\widetilde{g})+L_{X_j}\cdot\{g(X_j)-\widetilde{g}(X_i)\}\right]\right]\\
&= \mathbb{E}_{\sigma_{(j)}}\left[\frac{1}{2}\sup_{g\in\mathcal{G}}\frac{1}{n}\left[\mathcal{S}_{\mathsf{X}}^{(j)}(\sigma,g)+L_{X_j}\cdot g(X_j)\right]\right.\\
&\qquad\left.+\frac{1}{2}\sup_{\widetilde{g}\in\mathcal{G}}\frac{1}{n}\left[\mathcal{S}_{\mathsf{X}}^{(j)}(\sigma,\widetilde{g})+(-1)\cdot L_{X_j}\cdot\widetilde{g}(X_i)\right]\right]\\
&= \mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\left[\mathcal{S}_{\mathsf{X}}^{(j)}(\sigma,g)+\sigma_j\cdot L_{X_j}\cdot g(X_j)\right]\right]\\
&= \mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\left[\sum_{i=1}^{j}\sigma_i\cdot L_{X_i}\cdot g(X_i)+\sum_{i=j+1}^{n}\sigma_i\cdot\ell_{X_i}(g(X_i))\right]\right]\\
&= \mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\left[\mathcal{S}_{\mathsf{X}}^{(j+1)}(\sigma,g)+\sigma_{j+1}\cdot\ell_{X_{j+1}}(g(X_{j+1}))\right]\right]\\
&= E_{j+1}^{[2]}.
\end{aligned} = E σ − j [ 2 1 g ∈ G , g ∈ G sup n 1 [ S X ( j ) ( σ , g ) + S X ( j ) ( σ , g ) + L X j ⋅ { g ( X j ) − g ( X i )} ] ] = E σ ( j ) [ 2 1 g ∈ G sup n 1 [ S X ( j ) ( σ , g ) + L X j ⋅ g ( X j ) ] + 2 1 g ∈ G sup n 1 [ S X ( j ) ( σ , g ) + ( − 1 ) ⋅ L X j ⋅ g ( X i ) ] ] = E σ [ g ∈ G sup n 1 [ S X ( j ) ( σ , g ) + σ j ⋅ L X j ⋅ g ( X j ) ] ] = E σ [ g ∈ G sup n 1 [ i = 1 ∑ j σ i ⋅ L X i ⋅ g ( X i ) + i = j + 1 ∑ n σ i ⋅ ℓ X i ( g ( X i )) ] ] = E σ [ g ∈ G sup n 1 [ S X ( j + 1 ) ( σ , g ) + σ j + 1 ⋅ ℓ X j + 1 ( g ( X j + 1 )) ] ] = E j + 1 [ 2 ] .
Lemma A.5. For real numbers L X 1 , … , L X n L_{X_1},\ldots,L_{X_n} L X 1 , … , L X n depending on X \mathsf{X} X and for a function class G \mathcal{G} G ,
E σ [ sup g ∈ co ( G ) 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] = E σ [ sup g ∈ G 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] . \mathbb{E}_{\sigma}\left[\sup_{g\in\operatorname{co}(\mathcal{G})}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\right]
=
\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\right]. E σ [ g ∈ co ( G ) sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] = E σ [ g ∈ G sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] .
Proof. Since co ( G ) ⊃ G \operatorname{co}(\mathcal{G})\supset\mathcal{G} co ( G ) ⊃ G , the following inequality holds:
sup g ∈ co ( G ) 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ≥ sup g ∈ G 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) . \sup_{g\in\operatorname{co}(\mathcal{G})}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)
\ge
\sup_{g\in\mathcal{G}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i). g ∈ co ( G ) sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ≥ g ∈ G sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) .
Moreover, for any g ˉ = ∑ g ~ ∈ G λ g ~ ⋅ g ~ ∈ co ( G ) \bar{g}=\sum_{\widetilde{g}\in\mathcal{G}}\lambda_{\widetilde{g}}\cdot\widetilde{g}\in\operatorname{co}(\mathcal{G}) g ˉ = ∑ g ∈ G λ g ⋅ g ∈ co ( G ) ,
∑ i = 1 n σ i ⋅ L X i ⋅ g ˉ ( X i ) = ∑ i = 1 n σ i ⋅ L X i ⋅ { ∑ g ~ ∈ G λ g ~ ⋅ g ~ ( X i ) } = ∑ g ~ ∈ G λ g ~ ∑ i = 1 n σ i ⋅ L X i ⋅ g ~ ( X i ) ≤ sup g ∈ G ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) , \begin{aligned}
\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\bar{g}(X_i)
&=\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\left\{\sum_{\widetilde{g}\in\mathcal{G}}\lambda_{\widetilde{g}}\cdot\widetilde{g}(X_i)\right\}\\
&=\sum_{\widetilde{g}\in\mathcal{G}}\lambda_{\widetilde{g}}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\widetilde{g}(X_i)\\
&\le\sup_{g\in\mathcal{G}}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i),
\end{aligned} i = 1 ∑ n σ i ⋅ L X i ⋅ g ˉ ( X i ) = i = 1 ∑ n σ i ⋅ L X i ⋅ ⎩ ⎨ ⎧ g ∈ G ∑ λ g ⋅ g ( X i ) ⎭ ⎬ ⎫ = g ∈ G ∑ λ g i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ≤ g ∈ G sup i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ,
so that the other inequality also holds:
sup g ∈ co ( G ) 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ≤ sup g ∈ G 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) \sup_{g\in\operatorname{co}(\mathcal{G})} \frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\leq\sup_{g\in\mathcal{G}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i) g ∈ co ( G ) sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ≤ g ∈ G sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i )
Lemma A.6. For some function classes A \mathcal{A} A and B \mathcal{B} B , let A + B = { g 1 + g 2 ∣ g 1 ∈ A , g 2 ∈ B } \mathcal{A}+\mathcal{B}=\{g_1+g_2|g_1\in\mathcal{A},g_2\in\mathcal{B}\} A + B = { g 1 + g 2 ∣ g 1 ∈ A , g 2 ∈ B } and A ⋅ B = { g 1 ⋅ g 2 ∣ g 1 ∈ A , g 2 ∈ B } \mathcal{A}\cdot\mathcal{B}=\{g_1\cdot g_2|g_1\in\mathcal{A},g_2\in\mathcal{B}\} A ⋅ B = { g 1 ⋅ g 2 ∣ g 1 ∈ A , g 2 ∈ B } . Then
sup g ∈ A + B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) = sup g ∈ A 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) + sup g ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) . \sup_{g\in\mathcal{A}+\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)=\sup_{g\in\mathcal{A}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)+\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i). g ∈ A + B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) = g ∈ A sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) + g ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) . Moreover, for sup g ∈ A sup x ∈ R d ∣ g ( x ) ∣ = C A \sup_{g\in\mathcal{A}}\sup_{x\in\mathbb{R}^{d}}|g(x)|=C_{\mathcal{A}} sup g ∈ A sup x ∈ R d ∣ g ( x ) ∣ = C A and sup g ∈ B sup x ∈ R d ∣ g ( x ) ∣ = C B \sup_{g\in\mathcal{B}}\sup_{x\in\mathbb{R}^{d}}|g(x)|=C_{\mathcal{B}} sup g ∈ B sup x ∈ R d ∣ g ( x ) ∣ = C B ,
E σ [ sup g ∈ A ⋅ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] \mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{A}\cdot\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\right] E σ [ g ∈ A ⋅ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] ≤ ( C A + C B ) { E σ [ sup g ∈ A 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] + E σ [ sup g ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] } . \leq(C_{\mathcal{A}}+C_{\mathcal{B}})\left\{\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{A}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\right]+\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\right]\right\}. ≤ ( C A + C B ) { E σ [ g ∈ A sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] + E σ [ g ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] } .
Proof. The definition of A + B \mathcal{A}+\mathcal{B} A + B implies that
sup g ∈ A + B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) = sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ { g 1 ( X i ) + g 2 ( X i ) } = sup g 1 ∈ A 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g 1 ( X i ) + sup g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g 2 ( X i ) = sup g ∈ A 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) + sup g ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) . \begin{aligned}
\sup_{g\in\mathcal{A}+\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)=\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\{g_1(X_i)+g_2(X_i)\} \\
&=\sup_{g_1\in\mathcal{A}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g_1(X_i)+\sup_{g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g_2(X_i) \\
&=\sup_{g\in\mathcal{A}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)+\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i).
\end{aligned} g ∈ A + B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) = g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ { g 1 ( X i ) + g 2 ( X i )} = g 1 ∈ A sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g 1 ( X i ) + g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g 2 ( X i ) = g ∈ A sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) + g ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) .
By applying Lemma A.4 with ℓ X i ( z ) = s ( z ) = z 2 \ell_{X_i}(z)=s(z)=z^2 ℓ X i ( z ) = s ( z ) = z 2 for z ∈ [ − ( C A + C B ) , C A + C B ] z\in[-(C_{\mathcal{A}}+C_{\mathcal{B}}),C_{\mathcal{A}}+C_{\mathcal{B}}] z ∈ [ − ( C A + C B ) , C A + C B ] , which satisfies ∣ s ( z 1 ) − s ( z 2 ) ∣ = ∣ z 1 2 − z 2 2 ∣ = ∣ z 1 + z 2 ∣ ∣ z 1 − z 2 ∣ ≤ 2 ( C A + C B ) ∣ z 1 − z 2 ∣ |s(z_1)-s(z_2)|=|z_1^2-z_2^2|=|z_1+z_2||z_1-z_2|\leq2(C_{\mathcal{A}}+C_{\mathcal{B}})|z_1-z_2| ∣ s ( z 1 ) − s ( z 2 ) ∣ = ∣ z 1 2 − z 2 2 ∣ = ∣ z 1 + z 2 ∣∣ z 1 − z 2 ∣ ≤ 2 ( C A + C B ) ∣ z 1 − z 2 ∣ for any z 1 , z 2 ∈ [ − ( C A + C B ) , C A + C B ] z_1,z_2\in[-(C_{\mathcal{A}}+C_{\mathcal{B}}),C_{\mathcal{A}}+C_{\mathcal{B}}] z 1 , z 2 ∈ [ − ( C A + C B ) , C A + C B ] , we see that
E σ [ sup g ∈ A ⋅ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] = E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g 1 ( X i ) g 2 ( X i ) ] = E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ 1 4 [ { g 1 ( X i ) + g 2 ( X i ) } 2 − { g 1 ( X i ) − g 2 ( X i ) } 2 ] ] \begin{aligned}
\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{A}\cdot\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\right] \\
&=\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g_1(X_i)g_2(X_i)\right] \\
&=\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\frac{1}{4}\left[\{g_1(X_i)+g_2(X_i)\}^{2}-\{g_1(X_i)-g_2(X_i)\}^{2}\right]\right]
\end{aligned} E σ [ g ∈ A ⋅ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] = E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g 1 ( X i ) g 2 ( X i ) ] = E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ 4 1 [ { g 1 ( X i ) + g 2 ( X i ) } 2 − { g 1 ( X i ) − g 2 ( X i ) } 2 ] ]
≤ 1 4 E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ { g 1 ( X i ) + g 2 ( X i ) } 2 ] + 1 4 E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n ( − σ i ) ⋅ L X i ⋅ { g 1 ( X i ) − g 2 ( X i ) } 2 ] = 1 4 E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ { g 1 ( X i ) + g 2 ( X i ) } 2 ] + 1 4 E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ { g 1 ( X i ) − g 2 ( X i ) } 2 ] = 1 4 E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ s ( g 1 ( X i ) + g 2 ( X i ) ) ] + 1 4 E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ s ( g 1 ( X i ) − g 2 ( X i ) ) ] ≤ C A + C B 2 E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ { g 1 ( X i ) + g 2 ( X i ) } ] + C A + C B 2 E σ [ sup g 1 ∈ A , g 2 ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ { g 1 ( X i ) − g 2 ( X i ) } ] = ( C A + C B ) { E σ [ sup g ∈ A 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] + E σ [ sup g ∈ B 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] } . \begin{aligned}
\le{}& \frac{1}{4}\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\{g_1(\mathbf{X}_i)+g_2(\mathbf{X}_i)\}^{2}\right]\\
&+\frac{1}{4}\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}(-\sigma_i)\cdot L_{X_i}\cdot\{g_1(\mathbf{X}_i)-g_2(\mathbf{X}_i)\}^{2}\right]\\
={}&\frac{1}{4}\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\{g_1(\mathbf{X}_i)+g_2(\mathbf{X}_i)\}^{2}\right]\\
&+\frac{1}{4}\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\{g_1(\mathbf{X}_i)-g_2(\mathbf{X}_i)\}^{2}\right]\\
={}&\frac{1}{4}\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot s\left(g_1(\mathbf{X}_i)+g_2(\mathbf{X}_i)\right)\right]\\
&+\frac{1}{4}\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot s\left(g_1(\mathbf{X}_i)-g_2(\mathbf{X}_i)\right)\right]\\
\le{}&\frac{C_{\mathcal{A}}+C_{\mathcal{B}}}{2}\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\{g_1(\mathbf{X}_i)+g_2(\mathbf{X}_i)\}\right]\\
&+\frac{C_{\mathcal{A}}+C_{\mathcal{B}}}{2}\mathbb{E}_{\sigma}\left[\sup_{g_1\in\mathcal{A},g_2\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\{g_1(\mathbf{X}_i)-g_2(\mathbf{X}_i)\}\right]\\
={}&(C_{\mathcal{A}}+C_{\mathcal{B}})\left\{\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{A}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(\mathbf{X}_i)\right]+\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(\mathbf{X}_i)\right]\right\}.
\end{aligned} ≤ = = ≤ = 4 1 E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ { g 1 ( X i ) + g 2 ( X i ) } 2 ] + 4 1 E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n ( − σ i ) ⋅ L X i ⋅ { g 1 ( X i ) − g 2 ( X i ) } 2 ] 4 1 E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ { g 1 ( X i ) + g 2 ( X i ) } 2 ] + 4 1 E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ { g 1 ( X i ) − g 2 ( X i ) } 2 ] 4 1 E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ s ( g 1 ( X i ) + g 2 ( X i ) ) ] + 4 1 E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ s ( g 1 ( X i ) − g 2 ( X i ) ) ] 2 C A + C B E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ { g 1 ( X i ) + g 2 ( X i )} ] + 2 C A + C B E σ [ g 1 ∈ A , g 2 ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ { g 1 ( X i ) − g 2 ( X i )} ] ( C A + C B ) { E σ [ g ∈ A sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] + E σ [ g ∈ B sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] } .
Let N ( ϵ , G , ρ ) N(\epsilon,\mathcal{G},\rho) N ( ϵ , G , ρ ) be the ϵ \epsilon ϵ -covering number of a class G \mathcal{G} G with respect to a pseudometric ρ \rho ρ , see Definition 5.1 and Section 5.3.3 of Wainwright [30 ] , and Definition 2.1.5 of van der Vaart and Wellner [29 ] . We will finally bound the second term on the right-hand side of (5) from above with the covering number, by utilizing the next lemma:
Lemma A.7. For sup g ∈ G sup x ∈ R d g ( x ) ≤ Δ \sup_{g\in\mathcal{G}}\sup_{\mathbf{x}\in\mathbb{R}^{d}}g(\mathbf{x})\le\Delta sup g ∈ G sup x ∈ R d g ( x ) ≤ Δ and any n ∈ N n\in\mathbb{N} n ∈ N ,
E σ [ sup g ∈ G 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] ≤ 32 ∑ i = 1 n L X i n ∫ 0 2 Δ ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ . \mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(\mathbf{X}_i)\right]\le32\frac{\sqrt{\sum_{i=1}^{n}L_{X_i}}}{n}\int_{0}^{2\Delta}\sqrt{\ln N(\epsilon,\mathcal{G},\lVert\cdot\rVert_{\infty})}\,\mathrm{d}\epsilon. E σ [ g ∈ G sup n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] ≤ 32 n ∑ i = 1 n L X i ∫ 0 2Δ ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ .
Proof. We shall first prove that in terms of σ \sigma σ , the following
{ 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) | g ∈ G } = { Z g ∣ g ∈ G } \left\{\frac{1}{n}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(\mathbf{X}_i)\middle|g\in\mathcal{G}\right\}=\{Z_g\mid g\in\mathcal{G}\} { n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) g ∈ G } = { Z g ∣ g ∈ G }
is a 0-mean sub-Gaussian process with respect to a pseudometric
ρ ( g , g ~ ) = ( 1 / n ) ∑ i = 1 n L X i 2 { g ( X i ) − g ~ ( X i ) } 2 . \rho(g,\widetilde{g}) = \sqrt{(1/n)\sum_{i=1}^{n} L_{X_i}^{2}\{g(X_i)-\widetilde{g}(X_i)\}^{2}}. ρ ( g , g ) = ( 1/ n ) i = 1 ∑ n L X i 2 { g ( X i ) − g ( X i ) } 2 .
The 0-mean property is satisfied since
E σ [ Z g ] = E σ [ 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ g ( X i ) ] = 1 n ∑ i = 1 n E σ [ σ i ] L X i ⋅ g ( X i ) = 1 n ∑ i = 1 n 0 ⋅ L X i ⋅ g ( X i ) = 0. \begin{aligned}
\mathbb{E}_{\sigma}[Z_g] &= \mathbb{E}_{\sigma}\left[\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot g(X_i)\right] \\
&= \frac{1}{\sqrt{n}}\sum_{i=1}^{n}\mathbb{E}_{\sigma}[\sigma_i]L_{X_i}\cdot g(X_i) \\
&= \frac{1}{\sqrt{n}}\sum_{i=1}^{n}0\cdot L_{X_i}\cdot g(X_i) \\
&= 0.
\end{aligned} E σ [ Z g ] = E σ [ n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ g ( X i ) ] = n 1 i = 1 ∑ n E σ [ σ i ] L X i ⋅ g ( X i ) = n 1 i = 1 ∑ n 0 ⋅ L X i ⋅ g ( X i ) = 0.
To prove the sub-Gaussianity, we now show that { Z g ∣ g ∈ G } \{Z_g|g\in\mathcal{G}\} { Z g ∣ g ∈ G } satisfies Definition 5.16 of Wainwright [30 ] : that is, for any g , g ~ ∈ G g,\widetilde{g}\in\mathcal{G} g , g ∈ G and any λ ∈ R \lambda\in\mathbb{R} λ ∈ R ,
E [ exp { λ ( Z g − Z g ~ ) } ] ≤ E [ exp { λ 2 ρ ( g , g ~ ) 2 / 2 } ] . \mathbb{E}\left[\exp\left\{\lambda(Z_g-Z_{\widetilde{g}})\right\}\right]\leq\mathbb{E}\left[\exp\left\{\lambda^2\rho(g,\widetilde{g})^2/2\right\}\right]. E [ exp { λ ( Z g − Z g ) } ] ≤ E [ exp { λ 2 ρ ( g , g ) 2 /2 } ] .
This is shown by using the inequality E σ i [ e σ i x ] ≤ e x 2 / 2 \mathbb{E}_{\sigma_i}[e^{\sigma_i x}]\leq e^{x^2/2} E σ i [ e σ i x ] ≤ e x 2 /2 in Example 2.3 of Wainwright [30 ] as follows:
E σ [ e λ ⋅ ( Z g − Z g ~ ) ] = E σ [ exp [ λ ⋅ 1 n ∑ i = 1 n σ i ⋅ L X i ⋅ { g ( X i ) − g ~ ( X i ) } ] ] = E σ [ ∏ i = 1 n exp [ λ ⋅ 1 n σ i ⋅ L X i ⋅ { g ( X i ) − g ~ ( X i ) } ] ] = ∏ i = 1 n E σ i [ exp [ σ i ⋅ λ ⋅ 1 n L X i ⋅ { g ( X i ) − g ~ ( X i ) } ] ] ≤ ∏ i = 1 n exp [ λ 2 2 [ 1 n L X i ⋅ { g ( X i ) − g ~ ( X i ) } ] 2 ] = exp [ λ 2 2 ⋅ 1 n ∑ i = 1 n L X i 2 { g ( X i ) − g ~ ( X i ) } 2 ] = exp [ λ 2 2 ρ ( g , g ~ ) 2 ] . \begin{aligned}
\mathbb{E}_{\sigma}\left[e^{\lambda\cdot(Z_g-Z_{\widetilde{g}})}\right]
&= \mathbb{E}_{\sigma}\left[\exp\left[\lambda\cdot\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\sigma_i\cdot L_{X_i}\cdot\{g(X_i)-\widetilde{g}(X_i)\}\right]\right] \\
&= \mathbb{E}_{\sigma}\left[\prod_{i=1}^{n}\exp\left[\lambda\cdot\frac{1}{\sqrt{n}}\sigma_i\cdot L_{X_i}\cdot\{g(X_i)-\widetilde{g}(X_i)\}\right]\right] \\
&= \prod_{i=1}^{n}\mathbb{E}_{\sigma_i}\left[\exp\left[\sigma_i\cdot\lambda\cdot\frac{1}{\sqrt{n}}L_{X_i}\cdot\{g(X_i)-\widetilde{g}(X_i)\}\right]\right] \\
&\leq\prod_{i=1}^{n}\exp\left[\frac{\lambda^2}{2}\left[\frac{1}{\sqrt{n}}L_{X_i}\cdot\{g(X_i)-\widetilde{g}(X_i)\}\right]^2\right] \\
&= \exp\left[\frac{\lambda^2}{2}\cdot\frac{1}{n}\sum_{i=1}^{n}L_{X_i}^{2}\{g(X_i)-\widetilde{g}(X_i)\}^{2}\right] \\
&= \exp\left[\frac{\lambda^2}{2}\rho(g,\widetilde{g})^2\right].
\end{aligned} E σ [ e λ ⋅ ( Z g − Z g ) ] = E σ [ exp [ λ ⋅ n 1 i = 1 ∑ n σ i ⋅ L X i ⋅ { g ( X i ) − g ( X i )} ] ] = E σ [ i = 1 ∏ n exp [ λ ⋅ n 1 σ i ⋅ L X i ⋅ { g ( X i ) − g ( X i )} ] ] = i = 1 ∏ n E σ i [ exp [ σ i ⋅ λ ⋅ n 1 L X i ⋅ { g ( X i ) − g ( X i )} ] ] ≤ i = 1 ∏ n exp [ 2 λ 2 [ n 1 L X i ⋅ { g ( X i ) − g ( X i )} ] 2 ] = exp [ 2 λ 2 ⋅ n 1 i = 1 ∑ n L X i 2 { g ( X i ) − g ( X i ) } 2 ] = exp [ 2 λ 2 ρ ( g , g ) 2 ] .
Hence, we have finished proving that { Z g ∣ g ∈ G } \{Z_g|g \in\mathcal{G}\} { Z g ∣ g ∈ G } is a 0-mean sub-Gaussian process with respect to ρ \rho ρ , and therefore we can apply Theorem 5.22 of Wainwright [30 ] to this process. By noting that
ρ ( g , g ~ ) = 1 n ∑ i = 1 n L X i 2 ⋅ { g ( X i ) − g ~ ( X i ) } 2 ≤ 1 n ∑ i = 1 n L X i 2 ⋅ sup x ∈ D ∣ g ( x ) − g ~ ( x ) ∣ 2 ≤ 1 n ∑ i = 1 n L X i 2 ⋅ sup x ∈ D ∣ g ( x ) − g ~ ( x ) ∣ = C ⋅ ∥ g − g ~ ∥ ∞ . \begin{aligned}
\rho(g,\widetilde{g}) &=
\sqrt{\frac{1}{n}\sum_{i=1}^{n}L_{X_i}^{2}\cdot\{g(X_i)-\widetilde{g}(X_i)\}^{2}} \\
&\leq\sqrt{\frac{1}{n}\sum_{i=1}^{n}L_{X_i}^{2}\cdot\sup_{x\in\mathcal{D}}|g(x)-\widetilde{g}(x)|^{2}} \\
&\leq\sqrt{\frac{1}{n}\sum_{i=1}^{n}L_{X_i}^{2}}\cdot\sup_{x\in\mathcal{D}}|g(x)-\widetilde{g}(x)| \\
&= C\cdot\lVert g-\widetilde{g}\rVert_{\infty}.
\end{aligned} ρ ( g , g ) = n 1 i = 1 ∑ n L X i 2 ⋅ { g ( X i ) − g ( X i ) } 2 ≤ n 1 i = 1 ∑ n L X i 2 ⋅ x ∈ D sup ∣ g ( x ) − g ( x ) ∣ 2 ≤ n 1 i = 1 ∑ n L X i 2 ⋅ x ∈ D sup ∣ g ( x ) − g ( x ) ∣ = C ⋅ ∥ g − g ∥ ∞ .
and that
N ( ϵ , G , ρ ) ≤ N ( ϵ , G , C ⋅ ∥ ⋅ ∥ ∞ ) = N ( ϵ / C , G , ∥ ⋅ ∥ ∞ ) N(\epsilon,\mathcal{G},\rho)\leq N(\epsilon,\mathcal{G},C\cdot\lVert\cdot\rVert_{\infty})=N(\epsilon/C,\mathcal{G},\lVert\cdot\rVert_{\infty}) N ( ϵ , G , ρ ) ≤ N ( ϵ , G , C ⋅ ∥ ⋅ ∥ ∞ ) = N ( ϵ / C , G , ∥ ⋅ ∥ ∞ )
(since for any g ~ ∈ G \widetilde{g}\in\mathcal{G} g ∈ G , { g ∈ G ∣ ρ ( g , g ~ ) < ϵ } ⊃ { g ∈ G ∣ C ⋅ ∥ g − g ~ ∥ ∞ < ϵ } \{g\in\mathcal{G}|\rho(g,\widetilde{g})<\epsilon\}\supset\{g\in\mathcal{G}|C\cdot\lVert g-\widetilde{g}\rVert_{\infty}<\epsilon\} { g ∈ G ∣ ρ ( g , g ) < ϵ } ⊃ { g ∈ G ∣ C ⋅ ∥ g − g ∥ ∞ < ϵ } ), Theorem 5.22 of Wainwright [30 ] implies the following evaluation for any g ~ ∈ G \widetilde{g}\in\mathcal{G} g ∈ G and any δ > 0 \delta>0 δ > 0 :
E σ [ sup g ∈ G Z g ] = E σ [ sup g ∈ G [ Z g − Z g ~ + Z g ~ ] ] = E σ [ sup g ∈ G [ Z g − Z g ~ ] ] + E σ [ Z g ~ ] ≤ E σ [ sup g , g ~ ∈ G [ Z g − Z g ~ ] ] + 0 ≤ 2 E σ [ sup g , g ~ ∈ G ρ ( g , g ~ ) ≤ δ [ Z g − Z g ~ ] ] + 32 ∫ δ / 4 sup g , g ~ ∈ G ρ ( g , g ~ ) ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ ≤ 2 E σ [ sup g , g ~ ∈ G ρ ( g , g ~ ) ≤ δ 1 n ∑ i = 1 n ∣ σ i ∣ ∣ L X i ∣ ∣ g ( X i ) − g ~ ( X i ) ∣ ] . + 32 ∫ δ / 4 sup g , g ~ ∈ G ρ ( g , g ~ ) ln N ( ϵ C , G , ∥ ⋅ ∥ ∞ ) d ϵ \begin{aligned}
\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}Z_g\right]
&=\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}[Z_g-Z_{\widetilde{g}}+Z_{\widetilde{g}}]\right]\\
&=\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}[Z_g-Z_{\widetilde{g}}]\right]+\mathbb{E}_{\sigma}[Z_{\widetilde{g}}]\\
&\leq\mathbb{E}_{\sigma}\left[\sup_{g,\widetilde{g}\in\mathcal{G}}[Z_g-Z_{\widetilde{g}}]\right]+0\\
&\leq2\mathbb{E}_{\sigma}\left[\sup_{\substack{g,\widetilde{g}\in\mathcal{G}\\\rho(g,\widetilde{g})\leq\delta}}[Z_g-Z_{\widetilde{g}}]\right]+32\int_{\delta/4}^{\sup_{g,\widetilde{g}\in\mathcal{G}}\rho(g,\widetilde{g})}\sqrt{\ln N(\epsilon,\mathcal{G},\lVert\cdot\rVert_{\infty})}\,d\epsilon\\
&\leq2\mathbb{E}_{\sigma}\left[\sup_{\substack{g,\widetilde{g}\in\mathcal{G}\\\rho(g,\widetilde{g})\leq\delta}}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}|\sigma_i||L_{X_i}||g(X_i)-\widetilde{g}(X_i)|\right]. \\
&+32\int_{\delta/4}^{\sup_{g,\widetilde{g}\in\mathcal{G}}\rho(g,\widetilde{g})}\sqrt{\ln N\left(\frac{\epsilon}{C},\mathcal{G},\|\cdot\|_{\infty}\right)}\,d\epsilon
\end{aligned} E σ [ g ∈ G sup Z g ] = E σ [ g ∈ G sup [ Z g − Z g + Z g ] ] = E σ [ g ∈ G sup [ Z g − Z g ] ] + E σ [ Z g ] ≤ E σ [ g , g ∈ G sup [ Z g − Z g ] ] + 0 ≤ 2 E σ g , g ∈ G ρ ( g , g ) ≤ δ sup [ Z g − Z g ] + 32 ∫ δ /4 s u p g , g ∈ G ρ ( g , g ) ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ ≤ 2 E σ g , g ∈ G ρ ( g , g ) ≤ δ sup n 1 i = 1 ∑ n ∣ σ i ∣∣ L X i ∣∣ g ( X i ) − g ( X i ) ∣ . + 32 ∫ δ /4 s u p g , g ∈ G ρ ( g , g ) ln N ( C ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ
≤ 2 E σ [ sup g , g ~ ∈ G ρ ( g , g ~ ) ≤ δ 1 n ∑ i = 1 n 1 2 ⋅ ∑ i = 1 n L X i 2 ⋅ { g ( X i ) − g ~ ( X i ) } 2 ] + 32 C ∫ δ / ( 4 C ) sup g , g ~ ∈ G ρ ( g , g ~ ) / C ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ \leq2\mathbb{E}\sigma\left[\sup_{\substack{g,\widetilde{g}\in\mathcal{G}\\\rho(g,\widetilde{g})\leq\delta}}\frac{1}{\sqrt{n}}\sqrt{\sum_{i=1}^{n}1^{2}}\cdot\sqrt{\sum_{i=1}^{n}L_{X_i}^{2}\cdot\{g(X_i)-\widetilde{g}(X_i)\}^{2}}\right]
+32C\int_{\delta/(4C)}^{\sup_{g,\widetilde{g}\in\mathcal{G}}\rho(g,\widetilde{g})/C}\sqrt{\ln N(\epsilon,\mathcal{G},\|\cdot\|_{\infty})}\,d\epsilon ≤ 2 E σ g , g ∈ G ρ ( g , g ) ≤ δ sup n 1 i = 1 ∑ n 1 2 ⋅ i = 1 ∑ n L X i 2 ⋅ { g ( X i ) − g ( X i ) } 2 + 32 C ∫ δ / ( 4 C ) s u p g , g ∈ G ρ ( g , g ) / C ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ
≤ 2 E σ [ sup g , g ~ ∈ G ρ ( g , g ~ ) ≤ δ n ⋅ ρ ( g , g ~ ) ] + 32 C ∫ δ / ( 4 C ) sup g , g ~ ∈ G ∥ g − g ~ ∥ ∞ C / C ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ \leq2\mathbb{E}\sigma\left[\sup_{\substack{g,\widetilde{g}\in\mathcal{G}\\\rho(g,\widetilde{g})\leq\delta}}\sqrt{n}\cdot\rho(g,\widetilde{g})\right]+32C\int_{\delta/(4C)}^{\sup_{g,\widetilde{g}\in\mathcal{G}}\|g-\widetilde{g}\|_{\infty}C/C}\sqrt{\ln N(\epsilon,\mathcal{G},\|\cdot\|_{\infty})}\,d\epsilon ≤ 2 E σ g , g ∈ G ρ ( g , g ) ≤ δ sup n ⋅ ρ ( g , g ) + 32 C ∫ δ / ( 4 C ) s u p g , g ∈ G ∥ g − g ∥ ∞ C / C ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ
≤ 2 δ n + 32 C ∫ δ / ( 4 C ) 2 sup g ∈ G ∥ g ∥ ∞ ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ . (43) \leq2\delta\sqrt{n}+32C\int_{\delta/(4C)}^{2\sup_{g\in\mathcal{G}}\|g\|_{\infty}}\sqrt{\ln N(\epsilon,\mathcal{G},\|\cdot\|_{\infty})}\,d\epsilon. \tag*{(43)} ≤ 2 δ n + 32 C ∫ δ / ( 4 C ) 2 s u p g ∈ G ∥ g ∥ ∞ ln N ( ϵ , G , ∥ ⋅ ∥ ∞ ) d ϵ . (43)
By multiplying 1 / n 1/\sqrt{n} 1/ n and δ ↘ 0 \delta\searrow0 δ ↘ 0 in (A.16), we obtain the inequality in Lemma A.7.
Let M = { m ( ⋅ , θ ) : D → R ∣ θ ∈ Θ c } \mathcal{M}=\{m(\cdot,\theta):\mathcal{D}\to\mathbb{R}\mid\theta\in\Theta_c\} M = { m ( ⋅ , θ ) : D → R ∣ θ ∈ Θ c } , where the set Θ c ⊂ R p \Theta_c\subset\mathbb{R}^{p} Θ c ⊂ R p for c > 0 c>0 c > 0 is already introduced in § 4 . We evaluate the convering number of M \mathcal{M} M here:
Lemma A.8. For C 3 = 2 sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 C_3=2\sup_{x\in\mathbb{R}^{d}}\sup_{\theta\in\Theta_c}\|(\partial/\partial\theta)m(x,\theta)\|_2 C 3 = 2 sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 and C 4 = ( 3 c ⋅ C 3 ) p C_4=(3c\cdot C_3)^p C 4 = ( 3 c ⋅ C 3 ) p ,
N ( ϵ , M , ∥ ⋅ ∥ ∞ ) ≤ I ( ϵ < c ⋅ C 3 ) ⋅ C 4 ⋅ ( 1 ϵ ) p + I ( ϵ ≥ c ⋅ C 3 ) . N(\epsilon,\mathcal{M},\|\cdot\|_{\infty})\leq\mathbb{I}(\epsilon<c\cdot C_3)\cdot C_4\cdot\left(\frac{1}{\epsilon}\right)^p+\mathbb{I}(\epsilon\geq c\cdot C_3). N ( ϵ , M , ∥ ⋅ ∥ ∞ ) ≤ I ( ϵ < c ⋅ C 3 ) ⋅ C 4 ⋅ ( ϵ 1 ) p + I ( ϵ ≥ c ⋅ C 3 ) .
Proof. From (i) Lemma 9.18 of Kosorok [20 ] , (ii) Theorem 2.7.11 of van der Vaart and Wellner (1996) (or Theorem 9.23 of Kosorok [20 ] ) with F = M \mathcal{F}=\mathcal{M} F = M , T = Θ c T=\Theta_c T = Θ c , d ( s , t ) = ∥ s − t ∥ 2 d(s,t)=\|s-t\|_2 d ( s , t ) = ∥ s − t ∥ 2 and F ( x ) = sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 F(x)=\sup_{x\in\mathbb{R}^{d}}\sup_{\theta\in\Theta_c}\|(\partial/\partial\theta)m(x,\theta)\|_2 F ( x ) = sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 , (iii) Lemma 6.27 (and the statement in Lemma 6.26 for the notation) of Mohri et al. (2018) with X = Θ c \mathcal{X}=\Theta_c X = Θ c and R = c R=c R = c , and (iv) a trivial fact N ( ϵ , Θ c , ∥ ⋅ ∥ ∞ ) = 1 N(\epsilon,\Theta_c,\|\cdot\|_{\infty})=1 N ( ϵ , Θ c , ∥ ⋅ ∥ ∞ ) = 1 for ϵ ≥ c \epsilon\geq c ϵ ≥ c , it follows that
N ( ϵ , M , ∥ ⋅ ∥ ∞ ) ≤ N ( ϵ 2 sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 , Θ c , ∥ ⋅ ∥ 2 ) ≤ I ( ϵ < c ⋅ 2 sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 ) ( 3 c 2 sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 ϵ ) p + I ( ε ≥ c ⋅ 2 sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 ) . \begin{aligned}
N(\epsilon,\mathcal{M},\|\cdot\|_{\infty}) \\
&\leq N\left(\frac{\epsilon}{2\sup_{x\in\mathbb{R}^{d}}\sup_{\theta\in\Theta_c}\|(\partial/\partial\theta)m(x,\theta)\|_2},\Theta_c,\|\cdot\|_2\right) \\
&\leq\mathbb{I}\left(\epsilon<c\cdot2\sup_{x\in\mathbb{R}^{d}}\sup_{\theta\in\Theta_c}\|(\partial/\partial\theta)m(x,\theta)\|_2\right)\left(3c\frac{2\sup_{x\in\mathbb{R}^{d}}\sup_{\theta\in\Theta_c}\|(\partial/\partial\theta)m(x,\theta)\|_2}{\epsilon}\right)^p \\
&+\mathbb{I}\left(\varepsilon\ge c\cdot2\sup_{x\in\mathbb{R}^{d}}\sup_{\theta\in\Theta_{c}}\left\|\left(\partial/\partial\theta\right)m(x,\theta)\right\|_{2}\right).
\end{aligned} N ( ϵ , M , ∥ ⋅ ∥ ∞ ) ≤ N ( 2 sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 ϵ , Θ c , ∥ ⋅ ∥ 2 ) ≤ I ( ϵ < c ⋅ 2 x ∈ R d sup θ ∈ Θ c sup ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 ) ( 3 c ϵ 2 sup x ∈ R d sup θ ∈ Θ c ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 ) p + I ( ε ≥ c ⋅ 2 x ∈ R d sup θ ∈ Θ c sup ∥ ( ∂ / ∂ θ ) m ( x , θ ) ∥ 2 ) .
Proof of Theorem 4.1
Putting θ 1 ( X ) = θ 2 ( X ) = θ ^ \theta_{1}(\mathbb{X})=\theta_{2}(\mathbb{X})=\hat{\theta} θ 1 ( X ) = θ 2 ( X ) = θ ^ in Lemma 4.1, it is shown that
L ^ ( μ ^ ) ≤ inf Λ ∈ W L ^ ( μ ( ⋅ , Λ , θ ^ ) ) + τ 2 I + τ A 1 ( θ ^ , θ ^ ) + ε . \hat{L}(\hat{\mu})\leq\inf_{\Lambda\in\mathcal{W}}\hat{L}\left(\mu\left(\cdot,\Lambda,\hat{\theta}\right)\right)+\frac{\tau^{2}}{I+\tau}A_{1}\left(\hat{\theta},\hat{\theta}\right)+\varepsilon. L ^ ( μ ^ ) ≤ Λ ∈ W inf L ^ ( μ ( ⋅ , Λ , θ ^ ) ) + I + τ τ 2 A 1 ( θ ^ , θ ^ ) + ε .
This inequality allows us to apply Lemma 4.2 with θ 1 ( X ) = θ 2 ( X ) = θ ^ \theta_{1}(\mathbb{X})=\theta_{2}(\mathbb{X})=\hat{\theta} θ 1 ( X ) = θ 2 ( X ) = θ ^ , μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) = μ ^ \mu\left(\cdot,\Lambda\left(\mathbb{X}\right),\theta_{1}(\mathbb{X})\right)=\hat{\mu} μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) = μ ^ and δ ( X ) = { τ 2 / ( I + τ ) } A 1 ( θ ^ , θ ^ ) + ε \delta\left(\mathbb{X}\right)=\left\{\tau^{2}/\left(I+\tau\right)\right\}A_{1}\left(\hat{\theta},\hat{\theta}\right)+\varepsilon δ ( X ) = { τ 2 / ( I + τ ) } A 1 ( θ ^ , θ ^ ) + ε . Then
E [ D U ( μ , μ ^ ) ] ≤ E [ inf g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) D U ( μ , g ) ] + E [ sup Λ ∈ W E n ( μ ( ⋅ , Λ , θ ^ ) ) ] + E [ sup Λ ∈ W [ − E n ( μ ( ⋅ , Λ , θ ^ ) ) ] ] + E [ τ 2 I + τ A 1 ( θ ^ , θ ^ ) + ε ] = E [ inf g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) D U ( μ , g ) ] + ∑ α = 0 1 E [ sup g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) [ ( − 1 ) α E n ( g ) ] ] + τ 2 I + τ C 1 ( n ) + ε . \begin{aligned}
\mathbb{E}\left[D_{\mathcal{U}}\left(\mu,\hat{\mu}\right)\right]
&\leq\mathbb{E}\left[\inf_{g\in u^{-1}\left(\operatorname{co}\left(T(\hat{\theta},\mathcal{D})\right)\right)}D_{\mathcal{U}}(\mu,g)\right]
+\mathbb{E}\left[\sup_{\Lambda\in\mathcal{W}}\mathcal{E}_{n}\left(\mu\left(\cdot,\Lambda,\hat{\theta}\right)\right)\right]\\
&\quad+\mathbb{E}\left[\sup_{\Lambda\in\mathcal{W}}\left[-\mathcal{E}_{n}\left(\mu\left(\cdot,\Lambda,\hat{\theta}\right)\right)\right]\right]
+\mathbb{E}\left[\frac{\tau^{2}}{I+\tau}A_{1}\left(\hat{\theta},\hat{\theta}\right)+\varepsilon\right]\\
&=\mathbb{E}\left[\inf_{g\in u^{-1}\left(\operatorname{co}\left(T(\hat{\theta},\mathcal{D})\right)\right)}D_{\mathcal{U}}(\mu,g)\right]\\
&\quad+\sum_{\alpha=0}^{1}\mathbb{E}\left[\sup_{g\in u^{-1}\left(\operatorname{co}\left(T(\hat{\theta},\mathcal{D})\right)\right)}\left[(-1)^{\alpha}\mathcal{E}_{n}(g)\right]\right]
+\frac{\tau^{2}}{I+\tau}C_{1}(n)\\
&\quad+\varepsilon.
\end{aligned} E [ D U ( μ , μ ^ ) ] ≤ E [ g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) inf D U ( μ , g ) ] + E [ Λ ∈ W sup E n ( μ ( ⋅ , Λ , θ ^ ) ) ] + E [ Λ ∈ W sup [ − E n ( μ ( ⋅ , Λ , θ ^ ) ) ] ] + E [ I + τ τ 2 A 1 ( θ ^ , θ ^ ) + ε ] = E [ g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) inf D U ( μ , g ) ] + α = 0 ∑ 1 E g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) sup [ ( − 1 ) α E n ( g ) ] + I + τ τ 2 C 1 ( n ) + ε .
which shows the inequality (5) .
From now on, we investigate the rate of convergence for the term
E [ sup g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) [ ( − 1 ) α E n ( g ) ] ] \mathbb{E}\left[\sup_{g\in u^{-1}\left(\operatorname{co}\left(T(\hat{\theta},\mathcal{D})\right)\right)}\left[(-1)^{\alpha}\mathcal{E}_{n}(g)\right]\right] E g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) sup [ ( − 1 ) α E n ( g ) ]
with α = 0 \alpha=0 α = 0 and 1 1 1 . By denoting
u ˉ − 1 ( co ( T ( Θ c , B ) ) ) = ⋃ θ ∈ Θ c u − 1 ( co ( T ( θ , B ) ) ) , \bar{u}^{-1}\left(\operatorname{co}\left(T\left(\Theta_{c},\mathcal{B}\right)\right)\right)=\bigcup_{\theta\in\Theta_{c}}u^{-1}\left(\operatorname{co}\left(T\left(\theta,\mathcal{B}\right)\right)\right), u ˉ − 1 ( co ( T ( Θ c , B ) ) ) = θ ∈ Θ c ⋃ u − 1 ( co ( T ( θ , B ) ) ) ,
ℓ ( ⋅ | u − 1 ( co ( T ( Θ c , B ) ) ) ) = { ℓ ( ⋅ ∣ g ) : R d × R → R | g ∈ u − 1 ( co ( T ( Θ c , B ) ) ) } , \ell\left(\cdot\,\middle|\,u^{-1}\left(\operatorname{co}\left(T\left(\Theta_{c},\mathcal{B}\right)\right)\right)\right)=\left\{\ell(\cdot\,|g):\mathbb{R}^{d}\times\mathbb{R}\to\mathbb{R}\,\middle|\,g\in u^{-1}\left(\operatorname{co}\left(T\left(\Theta_{c},\mathcal{B}\right)\right)\right)\right\}, ℓ ( ⋅ u − 1 ( co ( T ( Θ c , B ) ) ) ) = { ℓ ( ⋅ ∣ g ) : R d × R → R g ∈ u − 1 ( co ( T ( Θ c , B ) ) ) } ,
F ( y ) = sup z ∈ [ − Δ 1 , Δ 1 ] ∣ U ∗ ( z ) ∣ + ∣ y ∣ Δ 1 , F(y)=\sup_{z\in[-\Delta_{1},\Delta_{1}]}\left|U^{*}(z)\right|+|y|\Delta_{1}, F ( y ) = z ∈ [ − Δ 1 , Δ 1 ] sup ∣ U ∗ ( z ) ∣ + ∣ y ∣ Δ 1 ,
and using Lemma A.3 , it follows for both α = 0 , 1 \alpha= 0,1 α = 0 , 1 that
E [ sup g ∈ U − 1 ( co ( T ( θ ^ , D ) ) ) [ ( − 1 ) α E n ( g ) ] ] ≤ E [ sup g ∈ U − 1 ( co ( T ( θ , B ) ) ) [ ( − 1 ) α E n ( g ) ] ] = E [ [ I { ∥ θ ^ − θ ∗ ∥ 2 ≤ c } + I { ∥ θ ^ − θ ∗ ∥ 2 > c } ] sup g ∈ U − 1 ( co ( T ( θ ^ , B ) ) ) [ ( − 1 ) α E n ( g ) ] ] ≤ E [ I { ∥ θ ^ − θ ∗ ∥ 2 ≤ c } sup g ∈ U − 1 ( co ( T ( θ ^ , B ) ) ) [ ( − 1 ) α E n ( g ) ] ] + E [ I { ∥ θ ^ − θ ∗ ∥ 2 > c } sup g ∈ U − 1 ( co ( T ( θ ^ , B ) ) ) [ E n ( g ) ] ] ≤ E [ sup φ ∈ ℓ ( ⋅ , U − 1 ( co ( T ( θ c , B ) ) ) { 1 n ∑ i = 1 n φ ( Y i , X i ) − ∫ R × R d φ ( y , x ) f ( y , x ) d y d x } ] + P ( ∥ θ ^ − θ ∗ ∥ 2 > c ) 1 / 2 E [ sup g ∈ U − 1 ( co ( T ( θ ^ , B ) ) ) [ E n ( g ) 2 ] ] 1 / 2 ≤ 2 E [ E σ [ sup φ ∈ ℓ ( ⋅ , U − 1 ( co ( T ( θ c , B ) ) ) 1 n ∑ i = 1 n σ i ⋅ φ ( Y i , X i ) ] ] + 1 c E [ ∥ θ ^ − θ ∗ ∥ 2 2 ] 1 / 2 E [ { 1 n ∑ i = 1 n F ( Y i ) + ∫ R × R d F ( y ) f ( y , x ) d y d x } 2 ] 1 / 2 = 2 E [ E σ [ sup g ∈ co ( T ( θ c , B ) ) 1 n ∑ i = 1 n σ i ⋅ { U ∗ ( g ( X i ) ) − Y i ⋅ g ( X i ) } ] ] + 1 c ⋅ O ( 1 n ) ⋅ E [ [ 2 sup z ∈ [ − Δ 1 , Δ 1 ] ∣ U ∗ ( z ) ∣ + Δ 1 ( 1 n ∑ i = 1 n ∣ Y i ∣ + ∫ R × R d ∣ y ∣ f ( y , x ) d y d x ) ] 2 ] 1 / 2 = E + O ( 1 n ) . (44) \begin{aligned}
&\mathbb{E}\left[\sup_{g\in\mathcal{U}^{-1}(\operatorname{co}(T(\widehat{\theta},\mathcal{D})))}\left[(-1)^{\alpha}\mathcal{E}_{n}(g)\right]\right]\\
&\leq\mathbb{E}\left[\sup_{g\in\mathcal{U}^{-1}(\operatorname{co}(T(\theta,\mathcal{B})))}\left[(-1)^{\alpha}\mathcal{E}_{n}(g)\right]\right]\\
&=\mathbb{E}\left[\left[\mathbb{I}\left\{\|\widehat{\theta}-\theta^{*}\|_{2}\leq c\right\}+\mathbb{I}\left\{\|\widehat{\theta}-\theta^{*}\|_{2}>c\right\}\right]\sup_{g\in\mathcal{U}^{-1}(\operatorname{co}(T(\widehat{\theta},\mathcal{B})))}\left[(-1)^{\alpha}\mathcal{E}_{n}(g)\right]\right]\\
&\leq\mathbb{E}\left[\mathbb{I}\left\{\|\widehat{\theta}-\theta^{*}\|_{2}\leq c\right\}\sup_{g\in\mathcal{U}^{-1}(\operatorname{co}(T(\widehat{\theta},\mathcal{B})))}\left[(-1)^{\alpha}\mathcal{E}_{n}(g)\right]\right]\\
&\quad+\mathbb{E}\left[\mathbb{I}\left\{\|\widehat{\theta}-\theta^{*}\|_{2}>c\right\}\sup_{g\in\mathcal{U}^{-1}(\operatorname{co}(T(\widehat{\theta},\mathcal{B})))}\left[\mathcal{E}_{n}(g)\right]\right]\\
&\leq\mathbb{E}\left[\sup_{\varphi\in\ell(\cdot,\mathcal{U}^{-1}(\operatorname{co}(T(\theta_{c},\mathcal{B})))}\left\{\frac{1}{n}\sum_{i=1}^{n}\varphi(Y_{i},X_{i})-\int_{\mathbb{R}\times\mathbb{R}^{d}}\varphi(y,x)f(y,x)\,dy\,dx\right\}\right]\\
&\quad+\mathbb{P}\left(\|\widehat{\theta}-\theta^{*}\|_{2}>c\right)^{1/2}\mathbb{E}\left[\sup_{g\in\mathcal{U}^{-1}(\operatorname{co}(T(\widehat{\theta},\mathcal{B})))}\left[\mathcal{E}_{n}(g)^{2}\right]\right]^{1/2}\\
&\leq2\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{\varphi\in\ell(\cdot,\mathcal{U}^{-1}(\operatorname{co}(T(\theta_{c},\mathcal{B})))}\frac{1}{n}\sum_{i=1}^{n}\sigma_{i}\cdot\varphi(Y_{i},X_{i})\right]\right]\\
&\quad+\frac{1}{c}\mathbb{E}\left[\|\widehat{\theta}-\theta^{*}\|_{2}^{2}\right]^{1/2}\mathbb{E}\left[\left\{\frac{1}{n}\sum_{i=1}^{n}F(Y_{i})+\int_{\mathbb{R}\times\mathbb{R}^{d}}F(y)f(y,x)\,dy\,dx\right\}^{2}\right]^{1/2}\\
&=2\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\operatorname{co}(T(\theta_{c},\mathcal{B}))}\frac{1}{n}\sum_{i=1}^{n}\sigma_{i}\cdot\{U^{*}(g(X_{i}))-Y_{i}\cdot g(X_{i})\}\right]\right]\\
&\quad+\frac{1}{c}\cdot O\left(\frac{1}{\sqrt{n}}\right)\cdot\mathbb{E}\left[\left[2\sup_{z\in[-\Delta_{1},\Delta_{1}]}|U^{*}(z)|+\Delta_{1}\left(\frac{1}{n}\sum_{i=1}^{n}|Y_{i}|+\int_{\mathbb{R}\times\mathbb{R}^{d}}|y|f(y,x)\,dy\,dx\right)\right]^{2}\right]^{1/2}\\
&=\mathbb{E}+O\left(\frac{1}{\sqrt{n}}\right). \tag*{(44)}
\end{aligned} E [ g ∈ U − 1 ( co ( T ( θ , D ))) sup [ ( − 1 ) α E n ( g ) ] ] ≤ E [ g ∈ U − 1 ( co ( T ( θ , B ))) sup [ ( − 1 ) α E n ( g ) ] ] = E [ [ I { ∥ θ − θ ∗ ∥ 2 ≤ c } + I { ∥ θ − θ ∗ ∥ 2 > c } ] g ∈ U − 1 ( co ( T ( θ , B ))) sup [ ( − 1 ) α E n ( g ) ] ] ≤ E [ I { ∥ θ − θ ∗ ∥ 2 ≤ c } g ∈ U − 1 ( co ( T ( θ , B ))) sup [ ( − 1 ) α E n ( g ) ] ] + E [ I { ∥ θ − θ ∗ ∥ 2 > c } g ∈ U − 1 ( co ( T ( θ , B ))) sup [ E n ( g ) ] ] ≤ E [ φ ∈ ℓ ( ⋅ , U − 1 ( co ( T ( θ c , B ))) sup { n 1 i = 1 ∑ n φ ( Y i , X i ) − ∫ R × R d φ ( y , x ) f ( y , x ) d y d x } ] + P ( ∥ θ − θ ∗ ∥ 2 > c ) 1/2 E [ g ∈ U − 1 ( co ( T ( θ , B ))) sup [ E n ( g ) 2 ] ] 1/2 ≤ 2 E [ E σ [ φ ∈ ℓ ( ⋅ , U − 1 ( co ( T ( θ c , B ))) sup n 1 i = 1 ∑ n σ i ⋅ φ ( Y i , X i ) ] ] + c 1 E [ ∥ θ − θ ∗ ∥ 2 2 ] 1/2 E { n 1 i = 1 ∑ n F ( Y i ) + ∫ R × R d F ( y ) f ( y , x ) d y d x } 2 1/2 = 2 E [ E σ [ g ∈ co ( T ( θ c , B )) sup n 1 i = 1 ∑ n σ i ⋅ { U ∗ ( g ( X i )) − Y i ⋅ g ( X i )} ] ] + c 1 ⋅ O ( n 1 ) ⋅ E [ 2 z ∈ [ − Δ 1 , Δ 1 ] sup ∣ U ∗ ( z ) ∣ + Δ 1 ( n 1 i = 1 ∑ n ∣ Y i ∣ + ∫ R × R d ∣ y ∣ f ( y , x ) d y d x ) ] 2 1/2 = E + O ( n 1 ) . (44)
To evaluate the above E \mathbb{E} E , we will apply Lemma A.4 with ℓ X i ( z ) = U ∗ ( z ) − Y i ⋅ z \ell_{X_i}(z)=U^{*}(z)-Y_i\cdot z ℓ X i ( z ) = U ∗ ( z ) − Y i ⋅ z , i = 1 , … , n i=1,\ldots,n i = 1 , … , n . Denoting Δ 2 ′ = sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ + Δ 2 sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ γ \Delta_{2}^{\prime}=\sup_{\theta\in\Theta_{c}}\sup_{x\in\mathbb{R}^{d}}|m(x,\theta)|+\Delta_{2}\sup_{\theta\in\Theta_{c}}\sup_{x\in\mathbb{R}^{d}}|m(x,\theta)|^{\gamma} Δ 2 ′ = sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ + Δ 2 sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ γ , the functions ℓ X i ( z ) \ell_{X_i}(z) ℓ X i ( z ) , i = 1 , … , n i=1,\ldots,n i = 1 , … , n with z ∈ [ − Δ 2 ′ , Δ 2 ′ ] z \in[-\Delta'_2,\Delta'_2] z ∈ [ − Δ 2 ′ , Δ 2 ′ ] can be evaluated as follows: from the mean value theorem, there exists c z 1 , z 2 ∈ ( 0 , 1 ) c_{z_1,z_2} \in(0,1) c z 1 , z 2 ∈ ( 0 , 1 ) such that
∣ ℓ X i ( z 1 ) − ℓ X i ( z 2 ) ∣ = ∣ u − 1 ( z 1 + c z 1 , z 2 ( z 1 − z 2 ) ) − Y i ∣ ⋅ ∣ z 1 − z 2 ∣ ≤ sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ ⋅ ∣ z 1 − z 2 ∣ . \begin{aligned}
\left|\ell_{X_i}(z_1)-\ell_{X_i}(z_2)\right| &= \left|u^{-1}\left(z_1+c_{z_1,z_2}(z_1-z_2)\right)-Y_i\right|\cdot|z_1-z_2| \\
&\le\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|\cdot|z_1-z_2|.
\end{aligned} ∣ ℓ X i ( z 1 ) − ℓ X i ( z 2 ) ∣ = u − 1 ( z 1 + c z 1 , z 2 ( z 1 − z 2 ) ) − Y i ⋅ ∣ z 1 − z 2 ∣ ≤ z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i ⋅ ∣ z 1 − z 2 ∣.
Thus, putting
C 1 = 1 + I ( γ = 1 ) { sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ γ + Δ 2 } , C_1=1+\mathbb{I}(\gamma=1)\left\{\sup_{\theta\in\Theta_c}\sup_{x\in\mathbb{R}^d}|m(x,\theta)|^\gamma+\Delta_2\right\}, C 1 = 1 + I ( γ = 1 ) { θ ∈ Θ c sup x ∈ R d sup ∣ m ( x , θ ) ∣ γ + Δ 2 } ,
C 2 = I ( γ = 0 ) + I ( γ = 1 ) { sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ γ + Δ 2 } C_2=\mathbb{I}(\gamma=0)+\mathbb{I}(\gamma=1)\left\{\sup_{\theta\in\Theta_c}\sup_{x\in\mathbb{R}^d}|m(x,\theta)|^\gamma+\Delta_2\right\} C 2 = I ( γ = 0 ) + I ( γ = 1 ) { θ ∈ Θ c sup x ∈ R d sup ∣ m ( x , θ ) ∣ γ + Δ 2 }
and M γ = { m ( ⋅ , θ ) γ : D → R ∣ θ ∈ Θ c } \mathcal{M}^{\gamma}=\{m(\cdot,\theta)^\gamma:\mathcal{D}\to\mathbb{R}\mid\theta\in\Theta_c\} M γ = { m ( ⋅ , θ ) γ : D → R ∣ θ ∈ Θ c } , where γ = 1 \gamma=1 γ = 1 leads to M γ = M \mathcal{M}^{\gamma}=\mathcal{M} M γ = M , already defined in previous subsection. Lemma A.4 and Lemma A.8 imply that
E ≤ E [ E σ [ sup g ∈ co ( T ( Θ c , B ) ) 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] = E [ E σ [ sup g ∈ T ( Θ c , B ) 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] ≤ E [ E σ [ sup g ∈ M + M γ ⋅ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] = E [ E σ [ sup g ∈ M 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] + E [ E σ [ sup g ∈ M γ ⋅ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] = E [ E σ [ sup g ∈ M 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] + I ( γ = 0 ) E [ E σ [ sup g ∈ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] \begin{aligned}
\mathbb{E}\leq\mathbb{E}\Bigg[\mathbb{E}_{\sigma}\Bigg[&\sup_{g\in\operatorname{co}(\mathcal{T}(\Theta_c,\mathcal{B}))}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\Bigg]\Bigg]\\
=\mathbb{E}\Bigg[\mathbb{E}_{\sigma}\Bigg[&\sup_{g\in\mathcal{T}(\Theta_c,\mathcal{B})}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\Bigg]\Bigg]\\
\leq\mathbb{E}\Bigg[\mathbb{E}_{\sigma}\Bigg[&\sup_{g\in\mathcal{M}+\mathcal{M}^{\gamma}\cdot\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\Bigg]\Bigg]\\
=\mathbb{E}\Bigg[\mathbb{E}_{\sigma}\Bigg[&\sup_{g\in\mathcal{M}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\Bigg]\Bigg]\\
+\mathbb{E}\Bigg[\mathbb{E}_{\sigma}\Bigg[&\sup_{g\in\mathcal{M}^{\gamma}\cdot\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\Bigg]\Bigg]\\
=\mathbb{E}\Bigg[\mathbb{E}_{\sigma}\Bigg[&\sup_{g\in\mathcal{M}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\Bigg]\Bigg]\\
+\mathbb{I}(\gamma=0)\mathbb{E}\Bigg[\mathbb{E}_{\sigma}\Bigg[&\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\Bigg]\Bigg]
\end{aligned} E ≤ E [ E σ [ = E [ E σ [ ≤ E [ E σ [ = E [ E σ [ + E [ E σ [ = E [ E σ [ + I ( γ = 0 ) E [ E σ [ g ∈ co ( T ( Θ c , B )) sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] g ∈ T ( Θ c , B ) sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] g ∈ M + M γ ⋅ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] g ∈ M sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] g ∈ M γ ⋅ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] g ∈ M sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] g ∈ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ]
+ I ( γ = 1 ) E [ E σ [ sup g ∈ M ⋅ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] ≤ E [ E σ [ sup g ∈ M 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] + I ( γ = 0 ) E [ E σ [ sup g ∈ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] + I ( γ = 1 ) { sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ γ + Δ 2 } × { E [ E σ [ sup g ∈ M 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] + E [ E σ [ sup g ∈ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] } = [ 1 + I ( γ = 1 ) { sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ γ + Δ 2 } ] × E [ E σ [ sup g ∈ M 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] + [ I ( γ = 0 ) + I ( γ = 1 ) { sup θ ∈ Θ c sup x ∈ R d ∣ m ( x , θ ) ∣ γ + Δ 2 } ] × E [ E σ [ sup g ∈ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] = C 1 E [ E σ [ sup g ∈ M 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] + C 2 E [ E σ [ sup g ∈ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] . (45) \begin{aligned}
&+\mathbb{I}(\gamma=1)\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{M}\cdot\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\\
&\leq\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{M}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\\
&\quad+\mathbb{I}(\gamma=0)\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\\
&\quad+\mathbb{I}(\gamma=1)\left\{\sup_{\theta\in\Theta_c}\sup_{x\in\mathbb{R}^d}|m(x,\theta)|^\gamma+\Delta_2\right\}\\
&\qquad\times\left\{\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{M}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\right.\\
&\qquad\qquad\left.+\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\right\}\\
&=\left[1+\mathbb{I}(\gamma=1)\left\{\sup_{\theta\in\Theta_c}\sup_{x\in\mathbb{R}^d}|m(x,\theta)|^\gamma+\Delta_2\right\}\right]\\
&\qquad\times\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{M}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\\
&\quad+\left[\mathbb{I}(\gamma=0)+\mathbb{I}(\gamma=1)\left\{\sup_{\theta\in\Theta_c}\sup_{x\in\mathbb{R}^d}|m(x,\theta)|^\gamma+\Delta_2\right\}\right]\\
&\qquad\times\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\\
&=C_1\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{M}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\\
&\quad+C_2\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]. \tag*{(45)}
\end{aligned} + I ( γ = 1 ) E [ E σ [ g ∈ M ⋅ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] ≤ E [ E σ [ g ∈ M sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] + I ( γ = 0 ) E [ E σ [ g ∈ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] + I ( γ = 1 ) { θ ∈ Θ c sup x ∈ R d sup ∣ m ( x , θ ) ∣ γ + Δ 2 } × { E [ E σ [ g ∈ M sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] + E [ E σ [ g ∈ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] } = [ 1 + I ( γ = 1 ) { θ ∈ Θ c sup x ∈ R d sup ∣ m ( x , θ ) ∣ γ + Δ 2 } ] × E [ E σ [ g ∈ M sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] + [ I ( γ = 0 ) + I ( γ = 1 ) { θ ∈ Θ c sup x ∈ R d sup ∣ m ( x , θ ) ∣ γ + Δ 2 } ] × E [ E σ [ g ∈ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] = C 1 E [ E σ [ g ∈ M sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] + C 2 E [ E σ [ g ∈ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] . (45)
We will apply Lemma A.7 to (A.18). From triangle inequality, a + b ≤ a + b \sqrt{a+b}\leq\sqrt{a}+\sqrt{b} a + b ≤ a + b ( a , b ≥ 0 ) (a,b\geq0) ( a , b ≥ 0 ) and Jensen’s inequality, it holds that
E [ ∑ i = 1 n L X i ] = E [ ∑ i = 1 n sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ ] ≤ n sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) ∣ 1 / 2 + E [ ∑ i = 1 n ∣ Y i ∣ ] ≤ n sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) ∣ 1 / 2 + E [ ∑ i = 1 n ∣ Y i ∣ ] 1 / 2 = n { sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) ∣ 1 / 2 + E [ ∣ Y i ∣ ] 1 / 2 } = ⟨ Constant for n ⟩ ⋅ n . \begin{aligned}
\mathbb{E}\left[\sqrt{\sum_{i=1}^{n}L_{X_i}}\right]
&=\mathbb{E}\left[\sqrt{\sum_{i=1}^{n}\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|}\right]\\
&\le\sqrt{n}\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)\right|^{1/2}+\mathbb{E}\left[\sqrt{\sum_{i=1}^{n}|Y_i|}\right]\\
&\le\sqrt{n}\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)\right|^{1/2}+\mathbb{E}\left[\sum_{i=1}^{n}|Y_i|\right]^{1/2}\\
&=\sqrt{n}\left\{\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)\right|^{1/2}+\mathbb{E}[|Y_i|]^{1/2}\right\}\\
&=\langle\text{Constant for }n\rangle\cdot\sqrt{n}.
\end{aligned} E [ i = 1 ∑ n L X i ] = E [ i = 1 ∑ n z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup ∣ u − 1 ( z ) − Y i ∣ ] ≤ n z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) 1/2 + E [ i = 1 ∑ n ∣ Y i ∣ ] ≤ n z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) 1/2 + E [ i = 1 ∑ n ∣ Y i ∣ ] 1/2 = n { z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) 1/2 + E [ ∣ Y i ∣ ] 1/2 } = ⟨ Constant for n ⟩ ⋅ n .
Noting that, by the change of variable t = ln ( 1 / ϵ ) + C 4 t=\ln(1/\epsilon)+C_4 t = ln ( 1/ ϵ ) + C 4 , we have
∫ 0 min { c ⋅ C 3 , 2 Δ 2 } ln 1 ϵ + C 4 d ϵ = ∫ ∞ ln [ 1 / min { c ⋅ C 3 , 2 Δ 2 } ] + C 4 t ⋅ ( − e C 4 ⋅ e − t ) d t = e C 4 ∫ ln [ 1 / min { c ⋅ C 3 , 2 Δ 2 } ] + C 4 ∞ t 1 / 2 ⋅ e − t d t ≤ e C 4 ⋅ Γ ( 3 2 ) < ∞ , \begin{aligned}
\int_{0}^{\min\{c\cdot C_3,2\Delta_2\}}\sqrt{\ln\frac{1}{\epsilon}+C_4}\,d\epsilon
&=\int_{\infty}^{\ln[1/\min\{c\cdot C_3,2\Delta_2\}]+C_4}\sqrt{t}\cdot(-e^{C_4}\cdot e^{-t})dt\\
&=e^{C_4}\int_{\ln[1/\min\{c\cdot C_3,2\Delta_2\}]+C_4}^{\infty}t^{1/2}\cdot e^{-t}dt\\
&\le e^{C_4}\cdot\Gamma\left(\frac{3}{2}\right)<\infty,
\end{aligned} ∫ 0 m i n { c ⋅ C 3 , 2 Δ 2 } ln ϵ 1 + C 4 d ϵ = ∫ ∞ l n [ 1/ m i n { c ⋅ C 3 , 2 Δ 2 }] + C 4 t ⋅ ( − e C 4 ⋅ e − t ) d t = e C 4 ∫ l n [ 1/ m i n { c ⋅ C 3 , 2 Δ 2 }] + C 4 ∞ t 1/2 ⋅ e − t d t ≤ e C 4 ⋅ Γ ( 2 3 ) < ∞ ,
Lemma A.7 and Lemma A.8 imply that
E ≤ C 1 E [ E σ [ sup g ∈ M 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] + C 2 E [ E σ [ sup g ∈ B 1 n ∑ i = 1 n σ i sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ g ( X i ) ] ] ≤ C 1 E [ 32 ∑ i = 1 n sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ n ∫ 0 2 Δ 2 ln N ( ϵ , M , ρ ) d ϵ ] + C 2 E [ 32 ∑ i = 1 n sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ n ∫ 0 2 Δ 2 ln N ( ϵ , B , ρ ) d ϵ ] \begin{aligned}
\mathbb{E}
&\le C_1\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{M}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\\
&\quad+C_2\mathbb{E}\left[\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{B}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sup_{z\in[-\Delta'_2,\Delta'_2]}\left|u^{-1}(z)-Y_i\right|g(X_i)\right]\right]\\
&\le C_1\mathbb{E}\left[32\frac{\sqrt{\sum_{i=1}^{n}\sup_{z\in[-\Delta'_2,\Delta'_2]}|u^{-1}(z)-Y_i|}}{n}\int_{0}^{2\Delta_2}\sqrt{\ln N(\epsilon,\mathcal{M},\rho)}\,d\epsilon\right]\\
&\quad+C_2\mathbb{E}\left[32\frac{\sqrt{\sum_{i=1}^{n}\sup_{z\in[-\Delta'_2,\Delta'_2]}|u^{-1}(z)-Y_i|}}{n}\int_{0}^{2\Delta_2}\sqrt{\ln N(\epsilon,\mathcal{B},\rho)}\,d\epsilon\right]
\end{aligned} E ≤ C 1 E [ E σ [ g ∈ M sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] + C 2 E [ E σ [ g ∈ B sup n 1 i = 1 ∑ n σ i z ∈ [ − Δ 2 ′ , Δ 2 ′ ] sup u − 1 ( z ) − Y i g ( X i ) ] ] ≤ C 1 E 32 n ∑ i = 1 n sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ ∫ 0 2 Δ 2 ln N ( ϵ , M , ρ ) d ϵ + C 2 E 32 n ∑ i = 1 n sup z ∈ [ − Δ 2 ′ , Δ 2 ′ ] ∣ u − 1 ( z ) − Y i ∣ ∫ 0 2 Δ 2 ln N ( ϵ , B , ρ ) d ϵ
≤ ⟨ Constant for n ⟩ n max { C 1 , C 2 } ⋅ 32 × { E [ ∫ 0 c 2 Δ 2 ln N ( ϵ , M , ∥ ⋅ ∥ ∞ ) d ϵ ] + E [ ∫ 0 c 2 Δ 2 ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ ] } ≤ ⟨ Constant for n ⟩ n { ∫ 0 2 Δ 2 ln [ I ( ϵ < c ⋅ C 3 ) ⋅ C 4 ⋅ ( 1 ϵ ) p + I ( ϵ ≥ c ⋅ C 3 ) ] d ϵ + ∫ 0 2 Δ 2 ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ } = ⟨ Constant for n ⟩ n [ { ∫ 0 min { c ⋅ C 3 , 2 Δ 2 } p ln 1 ϵ + ln C 4 d ϵ + ∫ c ⋅ C 3 max { c ⋅ C 3 , 2 Δ 2 } 1 d ϵ } + { ∫ 0 min { 1 , 2 Δ 2 } ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ + ∫ 1 max { 1 , 2 Δ 2 } q ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ } ] = ⟨ Constant for n ⟩ n [ { e C 4 ⋅ Γ ( 3 2 ) + max { c ⋅ C 3 , 2 Δ 2 } − 1 } + { ∫ 0 min { 1 , 2 Δ 2 } ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ + ln N ( 1 , B , ∥ ⋅ ∥ ∞ ) { max { 1 , 2 Δ 2 } − 1 } } ] ≤ 1 n ⋅ ⟨ Constant for n ⟩ . (46) \begin{aligned}
&\le\frac{\langle\text{Constant for }n\rangle}{\sqrt{n}}\max\{C_{1},C_{2}\}\cdot32\\
&\quad\times\left\{\mathbb{E}\left[\int_{0}^{c^{2}\Delta_{2}}\sqrt{\ln N(\epsilon,\mathcal{M},\|\cdot\|_{\infty})}\,d\epsilon\right]+\mathbb{E}\left[\int_{0}^{c^{2}\Delta_{2}}\sqrt{\ln N(\epsilon,\mathcal{B},\|\cdot\|_{\infty})}\,d\epsilon\right]\right\}\\
&\le\frac{\langle\text{Constant for }n\rangle}{\sqrt{n}}\left\{\int_{0}^{2\Delta_{2}}\sqrt{\ln\left[\mathbb{I}(\epsilon<c\cdot C_{3})\cdot C_{4}\cdot\left(\frac{1}{\epsilon}\right)^{p}+\mathbb{I}(\epsilon\ge c\cdot C_{3})\right]}\,d\epsilon\right.\\
&\qquad\left.+\int_{0}^{2\Delta_{2}}\sqrt{\ln N(\epsilon,\mathcal{B},\|\cdot\|_{\infty})}\,d\epsilon\right\}\\
&= \frac{\langle\text{Constant for }n\rangle}{\sqrt{n}}\left[\left\{\int_{0}^{\min\{c\cdot C_{3},2\Delta_{2}\}}\sqrt{p\ln\frac{1}{\epsilon}+\ln C_{4}}\,d\epsilon+\int_{c\cdot C_{3}}^{\max\{c\cdot C_{3},2\Delta_{2}\}}1\,d\epsilon\right\}\right.\\
&\qquad\left.+\left\{\int_{0}^{\min\{1,2\Delta_{2}\}}\sqrt{\ln N(\epsilon,\mathcal{B},\|\cdot\|_{\infty})}\,d\epsilon+\int_{1}^{\max\{1,2\Delta_{2}\}}\sqrt{q\ln N(\epsilon,\mathcal{B},\|\cdot\|_{\infty})}\,d\epsilon\right\}\right]\\
&= \frac{\langle\text{Constant for }n\rangle}{\sqrt{n}}\left[\left\{e^{C_{4}}\cdot\Gamma\left(\frac{3}{2}\right)+\max\{c\cdot C_{3},2\Delta_{2}\}-1\right\}\right.\\
&\qquad\left.+\left\{\int_{0}^{\min\{1,2\Delta_{2}\}}\sqrt{\ln N(\epsilon,\mathcal{B},\|\cdot\|_{\infty})}\,d\epsilon+\sqrt{\ln N(1,\mathcal{B},\|\cdot\|_{\infty})}\{\max\{1,2\Delta_{2}\}-1\}\right\}\right]\\
&\le\frac{1}{\sqrt{n}}\cdot\langle\text{Constant for }n\rangle. \tag*{(46)}
\end{aligned} ≤ n ⟨ Constant for n ⟩ max { C 1 , C 2 } ⋅ 32 × { E [ ∫ 0 c 2 Δ 2 ln N ( ϵ , M , ∥ ⋅ ∥ ∞ ) d ϵ ] + E [ ∫ 0 c 2 Δ 2 ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ ] } ≤ n ⟨ Constant for n ⟩ { ∫ 0 2 Δ 2 ln [ I ( ϵ < c ⋅ C 3 ) ⋅ C 4 ⋅ ( ϵ 1 ) p + I ( ϵ ≥ c ⋅ C 3 ) ] d ϵ + ∫ 0 2 Δ 2 ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ } = n ⟨ Constant for n ⟩ [ { ∫ 0 m i n { c ⋅ C 3 , 2 Δ 2 } p ln ϵ 1 + ln C 4 d ϵ + ∫ c ⋅ C 3 m a x { c ⋅ C 3 , 2 Δ 2 } 1 d ϵ } + { ∫ 0 m i n { 1 , 2 Δ 2 } ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ + ∫ 1 m a x { 1 , 2 Δ 2 } q ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ } ] = n ⟨ Constant for n ⟩ [ { e C 4 ⋅ Γ ( 2 3 ) + max { c ⋅ C 3 , 2 Δ 2 } − 1 } + { ∫ 0 m i n { 1 , 2 Δ 2 } ln N ( ϵ , B , ∥ ⋅ ∥ ∞ ) d ϵ + ln N ( 1 , B , ∥ ⋅ ∥ ∞ ) { max { 1 , 2 Δ 2 } − 1 } } ] ≤ n 1 ⋅ ⟨ Constant for n ⟩ . (46)
The above (A.17), (A.18) and (A.19) finally yield the conclusion.
Proof of Theorem 4.2
Putting θ 1 ( X ) = θ 2 ( X ) = θ ^ \theta_{1}(\mathbb{X})=\theta_{2}(\mathbb{X})=\hat{\theta} θ 1 ( X ) = θ 2 ( X ) = θ ^ , δ ( X ) = ϵ \delta(\mathbb{X})=\epsilon δ ( X ) = ϵ and μ ( ⋅ , Λ ( X ) , θ 1 ( X ) ) = μ 0 \mu(\cdot,\Lambda(\mathbb{X}),\theta_{1}(\mathbb{X}))=\mu^{0} μ ( ⋅ , Λ ( X ) , θ 1 ( X )) = μ 0 in Lemma 4.2,
E [ D U ( μ , μ ^ ) ] ≤ E [ inf g ∈ u − 1 ( co ( T ( θ , D ) ) ) D U ( μ , g ) ] + E [ sup Λ ∈ W E n ( μ ( ⋅ , Λ , θ ^ ( X ) ) ) ] + E [ sup Λ ∈ W [ − E n ( μ ( ⋅ , Λ , θ ^ ( X ) ) ) ] ] + E ∣ ϵ ∣ = E [ inf g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) D U ( μ , g ) ] + ∑ α = 0 1 E [ sup g ∈ u − 1 ( co ( T ( θ ^ , D ) ) ) [ ( − 1 ) α E n ( g ) ] ] + ϵ , \begin{aligned}
\mathbb{E}\left[D_{U}(\mu,\hat{\mu})\right]
&\le\mathbb{E}\left[\inf_{g\in u^{-1}(\operatorname{co}(T(\theta,\mathcal{D})))}D_{U}(\mu,g)\right]\\
&\quad+\mathbb{E}\left[\sup_{\Lambda\in\mathcal{W}}\mathcal{E}_{n}\left(\mu(\cdot,\Lambda,\hat{\theta}(\mathbb{X}))\right)\right]+\mathbb{E}\left[\sup_{\Lambda\in\mathcal{W}}\left[-\mathcal{E}_{n}\left(\mu(\cdot,\Lambda,\hat{\theta}(\mathbb{X}))\right)\right]\right]+\mathbb{E}|\epsilon|\\
&= \mathbb{E}\left[\inf_{g\in u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D})))}D_{U}(\mu,g)\right]+\sum_{\alpha=0}^{1}\mathbb{E}\left[\sup_{g\in u^{-1}(\operatorname{co}(T(\hat{\theta},\mathcal{D})))}\left[(-1)^{\alpha}\mathcal{E}_{n}(g)\right]\right]+\epsilon,
\end{aligned} E [ D U ( μ , μ ^ ) ] ≤ E [ g ∈ u − 1 ( co ( T ( θ , D ))) inf D U ( μ , g ) ] + E [ Λ ∈ W sup E n ( μ ( ⋅ , Λ , θ ^ ( X )) ) ] + E [ Λ ∈ W sup [ − E n ( μ ( ⋅ , Λ , θ ^ ( X )) ) ] ] + E ∣ ϵ ∣ = E [ g ∈ u − 1 ( co ( T ( θ ^ , D ))) inf D U ( μ , g ) ] + α = 0 ∑ 1 E [ g ∈ u − 1 ( co ( T ( θ ^ , D ))) sup [ ( − 1 ) α E n ( g ) ] ] + ϵ ,
which shows the inequality ( 4.2 ) (4.2) ( 4.2 ) .
Proof of Proposition 5.1
Defining P 0 = R × R d × H 0 \mathcal{P}_0 = \mathbb{R} \times\mathbb{R}^d \times\mathbb{H}_0 P 0 = R × R d × H 0 , we put
B 0 ( ⋅ ∣ θ , Z , P 0 ) = B 0 ( ⋅ ∣ θ , Z , ( v , w ) , H ) = K H ( w − ⋅ ) m ( w , θ ) γ { v − u − 1 ( m ( w , θ ) ) } ( 1 / n ) ∑ j = 1 n K H ( w j − ⋅ ) m ( w j , θ ) 2 γ ( U ∗ ) ′ ′ ( m ( w j , θ ) ) , \begin{aligned}
B_0(\cdot\mid\theta,\mathcal{Z},\mathcal{P}_0)=B_0(\cdot\mid\theta,\mathcal{Z},(v,w),H) \\
&=\frac{K_H(w-\cdot)m(w,\theta)^\gamma\{v-u^{-1}(m(w,\theta))\}}{(1/n)\sum_{j=1}^{n}K_H(w_j-\cdot)m(w_j,\theta)^{2\gamma}(U^*)^{\prime\prime}(m(w_j,\theta))},
\end{aligned} B 0 ( ⋅ ∣ θ , Z , P 0 ) = B 0 ( ⋅ ∣ θ , Z , ( v , w ) , H ) = ( 1/ n ) ∑ j = 1 n K H ( w j − ⋅ ) m ( w j , θ ) 2 γ ( U ∗ ) ′′ ( m ( w j , θ )) K H ( w − ⋅ ) m ( w , θ ) γ { v − u − 1 ( m ( w , θ ))} ,
θ ∈ Θ , ( ( v , w ) , H ) ∈ P 0 , \theta\in\Theta,\ ((v,w),H)\in\mathcal{P}_0, θ ∈ Θ , (( v , w ) , H ) ∈ P 0 ,
which has the same form as the word introduced below Lemma 4.2 in Section 4. Then we set
P 0 ∗ = { ( ( y i , x i ) , H s ) ∈ R × R d × H 0 ∣ i = 1 , … , n , s = 1 , … , S 0 } \mathcal{P}_0^*=\{((y_i,x_i),H_s)\in\mathbb{R}\times\mathbb{R}^d\times\mathbb{H}_0\mid i=1,\ldots,n,\ s=1,\ldots,S_0\} P 0 ∗ = {(( y i , x i ) , H s ) ∈ R × R d × H 0 ∣ i = 1 , … , n , s = 1 , … , S 0 } for some S 0 ∈ N S_0\in\mathbb{N} S 0 ∈ N and
B 0 ( ⋅ ∣ θ ^ ( X ) , X , ( y i , x i ) , H s ) = K H s ( x i − ⋅ ) m ( x i , θ ^ ( X ) ) γ { y i − u − 1 ( m ( x i , θ ^ ( X ) ) ) } ( 1 / n ) ∑ j = 1 n K H s ( x j − ⋅ ) m ( x j , θ ^ ( X ) ) 2 γ ( U ∗ ) ′ ′ ( m ( x j , θ ^ ( X ) ) ) , (47) \begin{aligned}
B_0\left(\cdot\mid\widehat{\theta}(\mathcal{X}),\mathcal{X},(y_i,x_i),H_s\right) \\
&=\frac{K_{H_s}(x_i-\cdot)m\left(x_i,\widehat{\theta}(\mathcal{X})\right)^\gamma\left\{y_i-u^{-1}\left(m\left(x_i,\widehat{\theta}(\mathcal{X})\right)\right)\right\}}{(1/n)\sum_{j=1}^{n}K_{H_s}(x_j-\cdot)m\left(x_j,\widehat{\theta}(\mathcal{X})\right)^{2\gamma}(U^*)^{\prime\prime}\left(m\left(x_j,\widehat{\theta}(\mathcal{X})\right)\right)}, \tag*{(47)}
\end{aligned} B 0 ( ⋅ ∣ θ ( X ) , X , ( y i , x i ) , H s ) = ( 1/ n ) ∑ j = 1 n K H s ( x j − ⋅ ) m ( x j , θ ( X ) ) 2 γ ( U ∗ ) ′′ ( m ( x j , θ ( X ) ) ) K H s ( x i − ⋅ ) m ( x i , θ ( X ) ) γ { y i − u − 1 ( m ( x i , θ ( X ) ) ) } , (47)
( ( y i , x i ) , H s ) ∈ P 0 ∗ , ((y_i,x_i),H_s)\in\mathcal{P}_0^*, (( y i , x i ) , H s ) ∈ P 0 ∗ ,
which coincides with a word in D 0 \mathcal{D}_0 D 0 .
Words for D 1 \mathcal{D}_1 D 1 and D 2 \mathcal{D}_2 D 2 can be written in a similar way: setting P 1 = R × R × H 1 \mathcal{P}_1=\mathbb{R}\times\mathbb{R}\times\mathbb{H}_1 P 1 = R × R × H 1 ,
B 1 ( ⋅ ∣ θ , Z , ( v , w ) , H ) = K H ( w − ⋅ ) m k ( w , θ ) γ { v − u − 1 ( m k ( w , θ ) ) } ( 1 / n ) ∑ j = 1 n K H ( w j k − ⋅ ) m k ( w j k , θ ) 2 γ ( U ∗ ) ′ ′ ( m k ( w j k , θ ) ) , B_1(\cdot\mid\theta,\mathcal{Z},(v,w),H)=\frac{K_H(w-\cdot)m_k(w,\theta)^\gamma\{v-u^{-1}(m_k(w,\theta))\}}{(1/n)\sum_{j=1}^{n}K_H(w_{jk}-\cdot)m_k(w_{jk},\theta)^{2\gamma}(U^*)^{\prime\prime}(m_k(w_{jk},\theta))}, B 1 ( ⋅ ∣ θ , Z , ( v , w ) , H ) = ( 1/ n ) ∑ j = 1 n K H ( w j k − ⋅ ) m k ( w j k , θ ) 2 γ ( U ∗ ) ′′ ( m k ( w j k , θ )) K H ( w − ⋅ ) m k ( w , θ ) γ { v − u − 1 ( m k ( w , θ ))} ,
θ ∈ Θ , ( ( v , w ) , H ) ∈ P 2 , 1 ≤ k ≤ d , \theta\in\Theta,\ ((v,w),H)\in\mathcal{P}_2,\ 1\leq k\leq d, θ ∈ Θ , (( v , w ) , H ) ∈ P 2 , 1 ≤ k ≤ d ,
and P 1 ∗ = { ( ( y i , x i k ) , H s ) ∈ R × R × H 1 ∣ i = 1 , … , n , 1 ≤ k ≤ d , s = 1 , … , S 1 } \mathcal{P}_1^*=\{((y_i,x_{ik}),H_s)\in\mathbb{R}\times\mathbb{R}\times\mathbb{H}_1\mid i=1,\ldots,n,\ 1\leq k\leq d,\ s=1,\ldots,S_1\} P 1 ∗ = {(( y i , x ik ) , H s ) ∈ R × R × H 1 ∣ i = 1 , … , n , 1 ≤ k ≤ d , s = 1 , … , S 1 } for some S 1 ∈ N S_1\in\mathbb{N} S 1 ∈ N ,
B 1 ( ⋅ ∣ θ ^ ( X ) , X , ( y i , x i k ) , H s ) = K H s ( x i k − ⋅ ) m k ( x i k , θ ^ ( X ) ) γ { y i − u − 1 ( m k ( x i k , θ ^ ( X ) ) ) } ( 1 / n ) ∑ j = 1 n K H s ( x j k − ⋅ ) m k ( x j k , θ ^ ( X ) ) 2 γ ( U ∗ ) ′ ′ ( m k ( x j k , θ ^ ( X ) ) ) (48) \begin{aligned}
B_1\left(\cdot\mid\widehat{\theta}(\mathcal{X}),\mathcal{X},(y_i,x_{ik}),H_s\right) \\
&=\frac{K_{H_s}(x_{ik}-\cdot)m_k\left(x_{ik},\widehat{\theta}(\mathcal{X})\right)^\gamma\left\{y_i-u^{-1}\left(m_k\left(x_{ik},\widehat{\theta}(\mathcal{X})\right)\right)\right\}}{(1/n)\sum_{j=1}^{n}K_{H_s}(x_{jk}-\cdot)m_k\left(x_{jk},\widehat{\theta}(\mathcal{X})\right)^{2\gamma}(U^*)^{\prime\prime}\left(m_k\left(x_{jk},\widehat{\theta}(\mathcal{X})\right)\right)} \tag*{(48)}
\end{aligned} B 1 ( ⋅ ∣ θ ( X ) , X , ( y i , x ik ) , H s ) = ( 1/ n ) ∑ j = 1 n K H s ( x j k − ⋅ ) m k ( x j k , θ ( X ) ) 2 γ ( U ∗ ) ′′ ( m k ( x j k , θ ( X ) ) ) K H s ( x ik − ⋅ ) m k ( x ik , θ ( X ) ) γ { y i − u − 1 ( m k ( x ik , θ ( X ) ) ) } (48)
( ( y i , x i k ) , H s ) ∈ P 1 ∗ ((y_i,x_{ik}),H_s)\in\mathcal{P}_1^* (( y i , x ik ) , H s ) ∈ P 1 ∗ ;
setting P 2 = R × R 2 × H 2 \mathcal{P}_2=\mathbb{R}\times\mathbb{R}^2\times\mathbb{H}_2 P 2 = R × R 2 × H 2 ,
B 2 ( θ , Z , ( v , w ) , H ) = K H ( w − ⋅ ) m k ℓ ( w , θ ) γ { v − u − 1 ( m k ℓ ( w , θ ) ) } ( 1 / n ) ∑ j = 1 n K H ( w j k ℓ − ⋅ ) m k ℓ ( w j k ℓ , θ ) 2 γ ( U ∗ ) ′ ′ ( m k ℓ ( w j k ℓ , θ ) ) , \begin{aligned}
B_2\left(\left.\theta,\mathcal{Z},(v,w),H\right)\right. \\
&=\frac{K_H(w-\cdot)m_{k\ell}(w,\theta)^\gamma\left\{v-u^{-1}\left(m_{k\ell}(w,\theta)\right)\right\}}{(1/n)\sum_{j=1}^{n}K_H(w_{jk\ell}-\cdot)m_{k\ell}(w_{jk\ell},\theta)^{2\gamma}\left(U^*\right)^{\prime\prime}\left(m_{k\ell}(w_{jk\ell},\theta)\right)},
\end{aligned} B 2 ( θ , Z , ( v , w ) , H ) = ( 1/ n ) ∑ j = 1 n K H ( w j k ℓ − ⋅ ) m k ℓ ( w j k ℓ , θ ) 2 γ ( U ∗ ) ′′ ( m k ℓ ( w j k ℓ , θ ) ) K H ( w − ⋅ ) m k ℓ ( w , θ ) γ { v − u − 1 ( m k ℓ ( w , θ ) ) } ,
θ ∈ Θ \theta\in\Theta θ ∈ Θ , ( ( v , w ) , H ) ∈ P 2 ((v,w),H)\in\mathcal{P}_2 (( v , w ) , H ) ∈ P 2 , 1 ≤ k < ℓ ≤ d 1\leq k<\ell\leq d 1 ≤ k < ℓ ≤ d ,
and P 2 ∗ = { ( ( y i , x i k ℓ ) , H s ) ∈ R × R 2 × H 2 ∣ i = 1 , … , n , 1 ≤ k < ℓ ≤ d , s = 1 , … , S 2 } \mathcal{P}_2^*=\{((y_i,x_{i k\ell}),H_s)\in\mathbb{R}\times\mathbb{R}^2\times\mathbb{H}_2\mid i=1,\ldots,n,\ 1\leq k<\ell\leq d,\ s=1,\ldots,S_2\} P 2 ∗ = {(( y i , x ik ℓ ) , H s ) ∈ R × R 2 × H 2 ∣ i = 1 , … , n , 1 ≤ k < ℓ ≤ d , s = 1 , … , S 2 } for S 2 ∈ N S_2\in\mathbb{N} S 2 ∈ N ,
B 2 ( θ ^ ( X ) , X , ( y i , x i k ℓ ) , H s ) = K H s ( x i k ℓ − ⋅ ) m k ℓ ( x i k ℓ , θ ^ ( X ) ) γ { y − u − 1 ( m k ℓ ( x i k ℓ , θ ^ ( X ) ) ) } ( 1 / n ) ∑ j = 1 n K H s ( x j k ℓ − ⋅ ) m k ℓ ( x j k ℓ , θ ^ ( X ) ) 2 γ ( U ∗ ) ′ ′ ( m k ℓ ( x j k ℓ , θ ^ ( X ) ) ) , (49) \begin{aligned}
B_2\left(\left.\hat{\theta}(\mathcal{X}),\mathcal{X},(y_i,x_{ik\ell}),H_s\right)\right. \\
&=\frac{K_{H_s}(x_{ik\ell}-\cdot)m_{k\ell}\left(x_{ik\ell},\hat{\theta}(\mathcal{X})\right)^\gamma\left\{y-u^{-1}\left(m_{k\ell}\left(x_{ik\ell},\hat{\theta}(\mathcal{X})\right)\right)\right\}}{(1/n)\sum_{j=1}^{n}K_{H_s}(x_{jk\ell}-\cdot)m_{k\ell}\left(x_{jk\ell},\hat{\theta}(\mathcal{X})\right)^{2\gamma}\left(U^*\right)^{\prime\prime}\left(m_{k\ell}\left(x_{jk\ell},\hat{\theta}(\mathcal{X})\right)\right)}, \tag*{(49)}
\end{aligned} B 2 ( θ ^ ( X ) , X , ( y i , x ik ℓ ) , H s ) = ( 1/ n ) ∑ j = 1 n K H s ( x j k ℓ − ⋅ ) m k ℓ ( x j k ℓ , θ ^ ( X ) ) 2 γ ( U ∗ ) ′′ ( m k ℓ ( x j k ℓ , θ ^ ( X ) ) ) K H s ( x ik ℓ − ⋅ ) m k ℓ ( x ik ℓ , θ ^ ( X ) ) γ { y − u − 1 ( m k ℓ ( x ik ℓ , θ ^ ( X ) ) ) } , (49)
( ( y i , x i k ℓ ) , H s ) ∈ P 2 ∗ ((y_i,x_{ik\ell}),H_s)\in\mathcal{P}_2^* (( y i , x ik ℓ ) , H s ) ∈ P 2 ∗ .
Now we fix an element in P 0 ∗ , P 1 ∗ , P 2 ∗ \mathcal{P}_0^*,\mathcal{P}_1^*,\mathcal{P}_2^* P 0 ∗ , P 1 ∗ , P 2 ∗ , and briefly denote the above
B 0 ( θ ^ ( X ) , X , ( y i , x i ) , H s ) , B 1 ( θ ^ ( X ) , X , ( y i , x i k ) , H s ) , B 2 ( θ ^ ( X ) , X , ( y i , x i k ℓ ) , H s ) B_0\left(\left.\hat{\theta}(\mathcal{X}),\mathcal{X},(y_i,x_i),H_s\right)\right.,\ B_1\left(\left.\hat{\theta}(\mathcal{X}),\mathcal{X},(y_i,x_{ik}),H_s\right)\right.,\ B_2\left(\left.\hat{\theta}(\mathcal{X}),\mathcal{X},(y_i,x_{ik\ell}),H_s\right)\right. B 0 ( θ ^ ( X ) , X , ( y i , x i ) , H s ) , B 1 ( θ ^ ( X ) , X , ( y i , x ik ) , H s ) , B 2 ( θ ^ ( X ) , X , ( y i , x ik ℓ ) , H s )
on D \mathbb{D} D by B 0 ( x ) , B 1 ( x ) , B 2 ( x ) B_0(x),B_1(x),B_2(x) B 0 ( x ) , B 1 ( x ) , B 2 ( x ) for x ∈ D x\in\mathbb{D} x ∈ D . From (A.20), (A.21) and (A.22), the functions B 0 B_0 B 0 , B 1 B_1 B 1 , B 2 B_2 B 2 belong to the class C ∞ ( D ) C^\infty(\mathbb{D}) C ∞ ( D ) of smooth functions on D \mathbb{D} D , since they depend on the input in D \mathbb{D} D only through the smooth function K K K .
We shall show the words B 0 B_0 B 0 , B 1 B_1 B 1 , B 2 B_2 B 2 belong to a function class C M α ( D ) C_M^\alpha(\mathbb{D}) C M α ( D ) , which appears in Section 2.7.1 of van der Vaart and Wellner [29 ] . Since B 0 , B 1 , B 2 ∈ C ∞ ( D ) B_0,B_1,B_2\in C^\infty(\mathbb{D}) B 0 , B 1 , B 2 ∈ C ∞ ( D ) , and continuous functions on compact sets are bounded, we can see
sup x ∈ D D k B i ( x ) < ∞ , i = 0 , 1 , 2 \sup_{x\in\mathbb{D}}D^kB_i(x)<\infty,\quad i=0,1,2 x ∈ D sup D k B i ( x ) < ∞ , i = 0 , 1 , 2
for any multi-index k = [ k 1 ⋯ k d ] \boldsymbol{k}=[k_1\cdots k_d] k = [ k 1 ⋯ k d ] , where D k = ∂ ∣ k ∣ / ( ∂ x k 1 ⋯ ∂ x k d ) D^k=\partial^{|k|}/(\partial x^{k_1}\cdots\partial x^{k_d}) D k = ∂ ∣ k ∣ / ( ∂ x k 1 ⋯ ∂ x k d ) . Moreover, the mean value theorem implies that there exists t ∈ ( 0 , 1 ) t\in(0,1) t ∈ ( 0 , 1 ) such that
sup x , y ∈ D ∣ D k B i ( x ) − D k B i ( y ) ∣ ∥ x − y ∥ 2 = sup x , y ∈ D ∣ ( ∂ / ∂ z ⊤ ) D k B i ( z ) ∣ z = t x + ( 1 − t ) y ( x − y ) ∣ ∥ x − y ∥ 2 ≤ sup x , y ∈ D sup t ∈ ( 0 , 1 ) ∥ ∂ ∂ z D k B i ( z ) ∣ z = t x + ( 1 − t ) y ∥ ∥ x − y ∥ 2 ∥ x − y ∥ 2 ≤ sup x ∈ co ( D ) ∥ ∂ ∂ x D k B i ( x ) ∥ 2 < ∞ . \begin{aligned}
\sup_{x,y\in\mathbb{D}}\frac{\left|D^kB_i(x)-D^kB_i(y)\right|}{\|x-y\|_2}
=
\sup_{x,y\in\mathbb{D}}\frac{\left|(\partial/\partial z^\top)D^kB_i(z)\big|_{z=tx+(1-t)y}(x-y)\right|}{\|x-y\|_2} \\
&\le\sup_{x,y\in\mathbb{D}}\sup_{t\in(0,1)}
\frac{\left\lVert\left.\frac{\partial}{\partial z}D^kB_i(z)\right|_{z=tx+(1-t)y}\right\rVert\lVert x-y\rVert_2}{\lVert x-y\rVert_2} \\
&\le\sup_{x\in\operatorname{co}(\mathbb{D})}\left\lVert\frac{\partial}{\partial x}D^kB_i(x)\right\rVert_2 \\
&<\infty.
\end{aligned} x , y ∈ D sup ∥ x − y ∥ 2 D k B i ( x ) − D k B i ( y ) = x , y ∈ D sup ∥ x − y ∥ 2 ( ∂ / ∂ z ⊤ ) D k B i ( z ) z = t x + ( 1 − t ) y ( x − y ) ≤ x , y ∈ D sup t ∈ ( 0 , 1 ) sup ∥ x − y ∥ 2 ∂ z ∂ D k B i ( z ) z = t x + ( 1 − t ) y ∥ x − y ∥ 2 ≤ x ∈ co ( D ) sup ∂ x ∂ D k B i ( x ) 2 < ∞.
Thus by putting
M = max ∣ k ∣ ≤ α − 1 sup x ∈ D D k B i ( x ) + max ∣ k ∣ = α − 1 sup x ∈ D sup x ∈ co ( D ) ∥ ∂ ∂ x D k B i ( x ) ∥ 2 < ∞ , \begin{aligned}
M=\max_{|k|\le\alpha-1}\sup_{x\in\mathbb{D}}D^kB_i(x)+\max_{|k|=\alpha-1}\sup_{x\in\mathbb{D}}\sup_{x\in\operatorname{co}(\mathbb{D})}\left\lVert\frac{\partial}{\partial x}D^kB_i(x)\right\rVert_2 \\
&<\infty,
\end{aligned} M = ∣ k ∣ ≤ α − 1 max x ∈ D sup D k B i ( x ) + ∣ k ∣ = α − 1 max x ∈ D sup x ∈ co ( D ) sup ∂ x ∂ D k B i ( x ) 2 < ∞ ,
it holds that D 0 ( Δ 1 , Δ 2 , c ) , D 1 ( Δ 1 , Δ 2 , c ) , D 2 ( Δ 1 , Δ 2 , c ) ⊂ C M α ( D ) \mathcal{D}_0(\Delta_1,\Delta_2,c),\mathcal{D}_1(\Delta_1,\Delta_2,c),\mathcal{D}_2(\Delta_1,\Delta_2,c)\subset C^\alpha_M(\mathbb{D}) D 0 ( Δ 1 , Δ 2 , c ) , D 1 ( Δ 1 , Δ 2 , c ) , D 2 ( Δ 1 , Δ 2 , c ) ⊂ C M α ( D ) for α > d / 2 \alpha>d/2 α > d /2 . We now evaluate the integral of the covering number of C M α ( D ) C^\alpha_M(\mathbb{D}) C M α ( D ) . Theorem 2.7.1 of van der Vaart and Wellner [29 ] implies that there exists a constant C C C such that
∫ 0 1 ln N ( ε , C M α ( D ) , ∥ ⋅ ∥ ∞ ) d ε = ∫ 0 1 ln N ( ε , M − 1 ⋅ C 1 α ( D ) , ∥ ⋅ ∥ ∞ ) d ε = ∫ 0 1 ln N ( ε M , C 1 α ( D ) , ∥ ⋅ ∥ ∞ ) d ε ≤ ∫ 0 1 C ( ε M ) − d / α d ε = C M − d / ( 2 α ) [ ε − d / ( 2 α ) 1 − d / ( 2 α ) ] 0 1 < ∞ . \begin{aligned}
\int_0^1\sqrt{\ln\mathcal{N}\left(\varepsilon,C^\alpha_M(\mathbb{D}),\lVert\cdot\rVert_\infty\right)}\,d\varepsilon
=
\int_0^1\sqrt{\ln\mathcal{N}\left(\varepsilon,M^{-1}\cdot C^\alpha_1(\mathbb{D}),\lVert\cdot\rVert_\infty\right)}\,d\varepsilon\\
&=
\int_0^1\sqrt{\ln\mathcal{N}\left(\varepsilon M,C^\alpha_1(\mathbb{D}),\lVert\cdot\rVert_\infty\right)}\,d\varepsilon\\
&\le\int_0^1\sqrt{C(\varepsilon M)^{-d/\alpha}}\,d\varepsilon\\
&=\sqrt{C}M^{-d/(2\alpha)}
\left[\frac{\varepsilon^{-d/(2\alpha)}}{1-d/(2\alpha)}\right]_0^1 \\
&<\infty.
\end{aligned} ∫ 0 1 ln N ( ε , C M α ( D ) , ∥ ⋅ ∥ ∞ ) d ε = ∫ 0 1 ln N ( ε , M − 1 ⋅ C 1 α ( D ) , ∥ ⋅ ∥ ∞ ) d ε = ∫ 0 1 ln N ( εM , C 1 α ( D ) , ∥ ⋅ ∥ ∞ ) d ε ≤ ∫ 0 1 C ( εM ) − d / α d ε = C M − d / ( 2 α ) [ 1 − d / ( 2 α ) ε − d / ( 2 α ) ] 0 1 < ∞.
Proof of Lemma 6.1
Note that
argmin θ ∈ R d + 1 D U ( μ , u − 1 ( [ 1 x ⊤ ] θ ) ) = argmin θ ∫ R d d U ( μ ( x ) , u − 1 ( m ( x , θ ) ) ) q ( x ) d x = argmin θ ∫ [ − 1 , 1 ] d { U ∗ ( [ 1 x ⊤ ] θ ) − μ ( x ) ⋅ [ 1 x ⊤ ] θ } d x = argmin θ L ( θ ) \begin{aligned}
\underset{\theta\in\mathbb{R}^{d+1}}{\operatorname{argmin}}\;D_U\left(\mu,u^{-1}\left(\left[1\ \ x^\top\right]\theta\right)\right)
=
\underset{\theta}{\operatorname{argmin}}\int_{\mathbb{R}^d}d_U\left(\mu(x),u^{-1}(m(x,\theta))\right)q(x)\,dx \\
&=
\underset{\theta}{\operatorname{argmin}}\int_{[-1,1]^d}\left\{U^*\left(\left[1\ \ x^\top\right]\theta\right)-\mu(x)\cdot\left[1\ \ x^\top\right]\theta\right\}\,dx \\
&= \underset{\theta}{\operatorname{argmin}}\, L(\theta)
\end{aligned} θ ∈ R d + 1 argmin D U ( μ , u − 1 ( [ 1 x ⊤ ] θ ) ) = θ argmin ∫ R d d U ( μ ( x ) , u − 1 ( m ( x , θ )) ) q ( x ) d x = θ argmin ∫ [ − 1 , 1 ] d { U ∗ ( [ 1 x ⊤ ] θ ) − μ ( x ) ⋅ [ 1 x ⊤ ] θ } d x = θ argmin L ( θ )
is unique, since the function θ ↦ L ( θ ) \theta\mapsto L(\theta) θ ↦ L ( θ ) is strictly convex. For θ ∗ \theta^* θ ∗ , we set
θ ′ = [ θ 0 ∗ − θ 1 ∗ θ 2 ∗ θ 3 ∗ θ 4 ∗ − θ 5 ∗ − θ 6 ∗ ⋯ − θ d ∗ ] T . \theta' = [\theta_0^*-\theta_1^*\ \theta_2^*\ \theta_3^*\ \theta_4^*\ -\theta_5^*\ -\theta_6^*\ \cdots\ -\theta_d^*]^{\mathsf{T}}. θ ′ = [ θ 0 ∗ − θ 1 ∗ θ 2 ∗ θ 3 ∗ θ 4 ∗ − θ 5 ∗ − θ 6 ∗ ⋯ − θ d ∗ ] T .
Then from the fact
μ ( [ − x 1 x 2 x 3 x 4 − x 5 − x 6 ⋯ − x d ] T ) = μ ( x ) \mu\left([-x_1\ x_2\ x_3\ x_4\ -x_5\ -x_6\ \cdots\ -x_d]^{\mathsf{T}}\right)=\mu(x) μ ( [ − x 1 x 2 x 3 x 4 − x 5 − x 6 ⋯ − x d ] T ) = μ ( x )
and by the definition of η ( x ) \eta(x) η ( x ) , it follows that
L ( θ ′ ) = ∫ [ − 1 , 1 ] d { U ∗ ( [ 1 x T ] θ ′ ) − μ ( x ) ⋅ [ 1 x T ] θ ′ } d x = ∫ [ − 1 , 1 ] d { U ∗ ( [ − x 1 x 2 x 3 x 4 − x 5 − x 6 ⋯ − x d ] θ ′ ) − μ ( [ − x 1 x 2 x 3 x 4 − x 5 − x 6 ⋯ − x d ] ) ⋅ [ 1 − x 1 x 2 x 3 x 4 − x 5 − x 6 ⋯ − x d ] θ ′ } d x = ∫ [ − 1 , 1 ] d { U ∗ ( [ 1 x T ] θ ∗ ) − μ ( x ) ⋅ [ 1 x T ] θ ∗ } d x = L ( θ ∗ ) . \begin{aligned}
L(\theta') &=
\int_{[-1,1]^d}\left\{U^*\left([1\ x^{\mathsf{T}}]\theta'\right)-\mu(x)\cdot[1\ x^{\mathsf{T}}]\theta'\right\}\,dx\\
&=\int_{[-1,1]^d}\left\{U^*\left([-x_1\ x_2\ x_3\ x_4\ -x_5\ -x_6\ \cdots\ -x_d]\theta'\right)\right.\\
&\qquad\left.-\mu\left([-x_1\ x_2\ x_3\ x_4\ -x_5\ -x_6\ \cdots\ -x_d]\right)\cdot[1\ -x_1\ x_2\ x_3\ x_4\ -x_5\ -x_6\ \cdots\ -x_d]\theta'\right\}\,dx\\
&=\int_{[-1,1]^d}\left\{U^*\left([1\ x^{\mathsf{T}}]\theta^*\right)-\mu(x)\cdot[1\ x^{\mathsf{T}}]\theta^*\right\}\,dx\\
&=L(\theta^*).
\end{aligned} L ( θ ′ ) = ∫ [ − 1 , 1 ] d { U ∗ ( [ 1 x T ] θ ′ ) − μ ( x ) ⋅ [ 1 x T ] θ ′ } d x = ∫ [ − 1 , 1 ] d { U ∗ ( [ − x 1 x 2 x 3 x 4 − x 5 − x 6 ⋯ − x d ] θ ′ ) − μ ( [ − x 1 x 2 x 3 x 4 − x 5 − x 6 ⋯ − x d ] ) ⋅ [ 1 − x 1 x 2 x 3 x 4 − x 5 − x 6 ⋯ − x d ] θ ′ } d x = ∫ [ − 1 , 1 ] d { U ∗ ( [ 1 x T ] θ ∗ ) − μ ( x ) ⋅ [ 1 x T ] θ ∗ } d x = L ( θ ∗ ) .
The uniqueness of θ ∗ \theta^* θ ∗ implies θ ′ = θ ∗ \theta'=\theta^* θ ′ = θ ∗ , and then θ 1 ∗ = θ 5 ∗ = ⋯ = θ d ∗ = 0 \theta_1^*=\theta_5^*=\cdots=\theta_d^*=0 θ 1 ∗ = θ 5 ∗ = ⋯ = θ d ∗ = 0 .
If u = id u=\operatorname{id} u = id , we can directly calculate θ ∗ \theta^* θ ∗ as follows:
θ ∗ = argmin θ ∫ R d { μ ( x ) − θ T [ 1 x ] } 2 q ( x ) d x = argmin θ ∫ [ − 1 , 1 ] d { μ ( x ) − θ T [ 1 x ] } 2 d x = argmin θ [ 2 θ T ∫ [ − 1 , 1 ] d μ ( x ) [ 1 x ] d x + θ T ∫ [ − 1 , 1 ] d [ 1 x ] [ 1 x T ] d x θ ] = solution θ [ 2 ∫ [ − 1 , 1 ] d μ ( x ) [ 1 x ] d x + 2 ∫ [ − 1 , 1 ] d [ 1 x ] [ 1 x T ] d x θ = 0 ] . = solution θ [ [ 2 d − 1 ⋅ { 2 / 3 + 0 + ( e − e − 1 ) + 0 } 2 d − 1 [ 0 + 0 + 0 + 0 0 + 2 / 3 + 0 + 0 0 + 0 + 2 / e + 0 0 + 0 + 0 + 2 / π 0 d − 4 ] + [ 2 d 0 d T 0 d ( 2 d / 3 ) I d ] ] ] θ = 0 = [ 1 / 3 + sinh ( 1 ) 0 1 3 / e 3 / π 0 d − 4 ] . \begin{aligned}
\theta^* &= \underset{\theta}{\operatorname{argmin}}\int_{\mathbb{R}^d}\left\{\mu(x)-\theta^{\mathsf{T}}\begin{bmatrix}1\\x\end{bmatrix}\right\}^2q(x)\,dx\\
&=\underset{\theta}{\operatorname{argmin}}\int_{[-1,1]^d}\left\{\mu(x)-\theta^{\mathsf{T}}\begin{bmatrix}1\\x\end{bmatrix}\right\}^2\,dx\\
&=\underset{\theta}{\operatorname{argmin}}\left[2\theta^{\mathsf{T}}\int_{[-1,1]^d}\mu(x)\begin{bmatrix}1\\x\end{bmatrix}\,dx+\theta^{\mathsf{T}}\int_{[-1,1]^d}\begin{bmatrix}1\\x\end{bmatrix}[1\ x^{\mathsf{T}}]\,dx\theta\right]\\
&=\underset{\theta}{\operatorname{solution}}\left[2\int_{[-1,1]^d}\mu(x)\begin{bmatrix}1\\x\end{bmatrix}\,dx+2\int_{[-1,1]^d}\begin{bmatrix}1\\x\end{bmatrix}[1\ x^{\mathsf{T}}]\,dx\theta=0\right]. \\
&=\operatorname{solution}_{\theta}\left[
\begin{bmatrix}
2^{d-1}\cdot\{2/3+0+(e-e^{-1})+0\}\\
2^{d-1}
\begin{bmatrix}
0+0+0+0\\
0+2/3+0+0\\
0+0+2/e+0\\
0+0+0+2/\pi\\
\mathbf{0}_{d-4}
\end{bmatrix}
+
\begin{bmatrix}
2^{d} & \mathbf{0}_{d}^{\mathsf{T}}\\
\mathbf{0}_{d} & (2^{d}/3)I_{d}
\end{bmatrix}
\end{bmatrix}
\right]\theta=0 \\
&=
\begin{bmatrix}
1/3+\sinh(1)\\
0\\
1\\
3/e\\
3/\pi\\
\mathbf{0}_{d-4}
\end{bmatrix}.
\end{aligned} θ ∗ = θ argmin ∫ R d { μ ( x ) − θ T [ 1 x ] } 2 q ( x ) d x = θ argmin ∫ [ − 1 , 1 ] d { μ ( x ) − θ T [ 1 x ] } 2 d x = θ argmin [ 2 θ T ∫ [ − 1 , 1 ] d μ ( x ) [ 1 x ] d x + θ T ∫ [ − 1 , 1 ] d [ 1 x ] [ 1 x T ] d x θ ] = θ solution [ 2 ∫ [ − 1 , 1 ] d μ ( x ) [ 1 x ] d x + 2 ∫ [ − 1 , 1 ] d [ 1 x ] [ 1 x T ] d x θ = 0 ] . = solution θ 2 d − 1 ⋅ { 2/3 + 0 + ( e − e − 1 ) + 0 } 2 d − 1 0 + 0 + 0 + 0 0 + 2/3 + 0 + 0 0 + 0 + 2/ e + 0 0 + 0 + 0 + 2/ π 0 d − 4 + [ 2 d 0 d 0 d T ( 2 d /3 ) I d ] θ = 0 = 1/3 + sinh ( 1 ) 0 1 3/ e 3/ π 0 d − 4 .
Acknowledgments
The research of the second author is supported in part by MEXT, Japan KAKENHI Grant Number 23K28043.
References