Introduction
In-Context learning (ICL) refers to an emerging prediction paradigm in which an estimator, trained once across a collection of tasks, automatically adapts its predictions to an unseen task solely through the task’s in-context samples, without any refitting of model parameters ([6 , 12 ] ). This contrasts sharply with conventional statistical learning, where a separate model must be retrained for each new task. The adaptivity of ICL calls for a theoretical explanation, for which the Bayesian perspective provides a compelling account ([38 ] ); the optimal predictor is the posterior predictive mean which is a fixed functional of the in-context
In-Context Learning in Measure-Indexed RKHS Framework
samples and the query that automatically adapts to any new task through the context alone. This perspective has motivated a growing body of work connecting ICL to classical statistical estimators. For linear function classes, a single layer of linear attention can implement ridge regression on the in-context samples [3 , 35 , 4 ] . For nonparametric regression over smooth function classes, Ching et al. [10 ] showed that transformers trained via meta-learning can attain minimax-optimal prediction rates, and Wakayama and Suzuki [37 ] established a rigorous connection between ICL and Bayesian inference for characterizing the optimal predictor and its generalization error.
Despite this rapid progress, two gaps remain. The first is concerned with uncertainty quantification. Existing convergence results establish that the ICL predictor converges to the truth, but say nothing about its distributional behavior, though the adaptivity of ICL is most valuable precisely when one needs to quantify uncertainty about a prediction made on a previously unseen task. Constructing confidence or prediction intervals in this setting requires asymptotic normality, which no existing result provides. The second gap is concerned with the geometry of the input domain. Directly using the raw context sample as input can lead to the “curse of dimensionality,” motivating a dimension-reduction strategy that transforms the input space into a more structured representation. The natural mathematical object representing an in-context sample is its empirical distribution [20 ] . The ICL predictor is therefore a function whose argument includes a probability measure as part of its input. Classical nonparametric methods, including the reproducing kernel Hilbert space (RKHS) based methods such as kernel ridge regression or related ones, are mostly built for Euclidean domains. They do not directly accommodate this measure-valued structure. Adapting them requires a nontrivial generalization of the standard RKHS framework.
In this paper, we propose a new RKHS framework, called measure-indexed RKHS (MI-RKHS), with measure-valued inputs for ICL that addresses both gaps simultaneously. The key observation is that ICL defines a fixed mean functional from a (context, query) pair to predictions; the adaptivity of the estimator across tasks is naturally captured by the dependence of this functional on the empirical distribution of the context. We model this function as an element in an MI-RKHS defined over the product of a space of probability measures and a feature space, which allows classical RKHS analysis to be extended to the measure-valued setting. For instance, we propose a measure-indexed kernel ridge regression (MI-KRR) estimator computed from multiple independent tasks and establish its convergence rate. We prove that the MI-KRR estimator is pointwise asymptotically normal, enabling the construction of confidence intervals for the mean and prediction intervals for the response on an unseen task. Notably, the estimator adapts to a new task without any refitting, yet still achieves asymptotic normality. The proposed MI-KRR method extends classical kernel ridge regression to the measure-valued setting, with both convergence rate and asymptotic normality established within this new framework. As a special case, when all tasks share the same fixed underlying function, our framework degenerates to the well-known distributed nonparametric regression, recovering known results in that literature such as optimal convergence rate [43 , 29 , 30 , 21 , 32 ] . More generally, our framework is conceptually related to meta-learning or transfer learning since both paradigms require the underlying tasks are different but still possess similarity.
The rest of this paper is organized as follows. In § 3 , we introduce the MI-RKHS framework as well as a MI-KRR regression for estimating the mean functional. In § 4.2 , we derive the convergence rate and asymptotic distribution for the proposed MI-KRR estimator. An example on latent Gaussian process model is provided to examine the proposed MI-RKHS framework. Proofs of the theorems and lemmas are deferred to § 5 .
In-Context Learning in Measure-Indexed RKHS Framework
Let X ⊆ R d \mathcal{X} \subseteq\mathbb{R}^{d} X ⊆ R d d d d H X \mathcal{H}_{X} H X X \mathcal{X} X ⟨ ⋅ , ⋅ ⟩ H X \langle\cdot,\cdot\rangle_{\mathcal{H}_{X}} ⟨ ⋅ , ⋅ ⟩ H X R ⟨ X ⟩ : X × X → R R^{\langle X\rangle}:\mathcal{X}\times\mathcal{X}\to\mathbb{R} R ⟨ X ⟩ : X × X → R μ \mu μ X \mathcal{X} X R ⟨ X ⟩ R^{\langle X\rangle} R ⟨ X ⟩
Assumption 1. C X ≜ sup x ∈ X R ⟨ X ⟩ ( x , x ) < ∞ C_{X}\triangleq\sup_{x\in\mathcal{X}}R^{\langle X\rangle}(x,x)<\infty C X ≜ sup x ∈ X R ⟨ X ⟩ ( x , x ) < ∞
The Assumption 1 is standard in the nonparametric literature which is satisfied for commonly used kernels such as Sobolev kernel and Gaussian kernels [28 , 40 ] . Let f f f F F F F F F H X \mathcal{H}_{X} H X X 1 , … , X n + 1 X_{1},\ldots,X_{n+1} X 1 , … , X n + 1 μ \mu μ f f f Y i Y_{i} Y i
Y i = f ( X i ) + ε i , i = 1 , … , n + 1 , (1) Y_{i}=f(X_{i})+\varepsilon_{i},\qquad i=1,\ldots,n+1, \tag*{(1)} Y i = f ( X i ) + ε i , i = 1 , … , n + 1 , (1) where ε i \varepsilon_i ε i f , X 1 , … , X n + 1 f,X_{1},\ldots,X_{n+1} f , X 1 , … , X n + 1
M ∗ ( D n , x ) = E [ Y n + 1 ∣ D n , X n + 1 = x ] , x ∈ X , (2) M_{*}(D_{n},x)=\mathbb{E}[Y_{n+1}\mid D_{n},X_{n+1}=x],\qquad x\in\mathcal{X}, \tag*{(2)} M ∗ ( D n , x ) = E [ Y n + 1 ∣ D n , X n + 1 = x ] , x ∈ X , (2) where D n = { Z 1 , … , Z n } D_{n}=\{Z_{1},\ldots,Z_{n}\} D n = { Z 1 , … , Z n } Z i = ( X i , Y i ) Z_{i}=(X_{i},Y_{i}) Z i = ( X i , Y i ) M ∗ M_{*} M ∗ D n D_{n} D n Z n + 1 Z_{n+1} Z n + 1 [14 , 35 , 41 , 24 , 16 , 2 , 34 , 19 , 37 , 10 ] . The target is to estimate M ∗ M_{*} M ∗ Y n + 1 Y_{n+1} Y n + 1 D n D_{n} D n X n + 1 X_{n+1} X n + 1 M ∗ M_{*} M ∗ D n D_{n} D n n ( d + 1 ) n(d+1) n ( d + 1 ) n n n
An observation is that, although the ambient dimension of D n D_{n} D n n n n M ∗ ( D n , x ) M_{*}(D_{n},x) M ∗ ( D n , x ) D n D_{n} D n M ∗ M_{*} M ∗ D n D_{n} D n D n D_{n} D n
Lemma 1. Under the conditional i.i.d. assumption on Z 1 , … , Z n + 1 Z_{1},\ldots,Z_{n+1} Z 1 , … , Z n + 1 f f f
E [ Y n + 1 ∣ D n , X n + 1 ] = E [ Y n + 1 ∣ π ^ n , X n + 1 ] a.s. , \mathbb{E}[Y_{n+1}\mid D_{n},X_{n+1}]=\mathbb{E}[Y_{n+1}\mid\widehat{\pi}_{n},X_{n+1}]\quad\text{a.s.}, E [ Y n + 1 ∣ D n , X n + 1 ] = E [ Y n + 1 ∣ π n , X n + 1 ] a.s. , where π ^ n = n − 1 ∑ i = 1 n δ Z i \widehat{\pi}_{n}=n^{-1}\sum_{i=1}^{n}\delta_{Z_{i}} π n = n − 1 ∑ i = 1 n δ Z i Z 1 , … , Z n Z_{1},\ldots,Z_{n} Z 1 , … , Z n
Lemma 1 shows that, for the purpose of predicting Y n + 1 Y_{n+1} Y n + 1 D n D_{n} D n π ^ n \widehat{\pi}_{n} π n D n D_{n} D n M ∗ M_{*} M ∗ π ^ n \widehat{\pi}_{n} π n x x x M ∗ M_{*} M ∗ ( π ^ n , x ) (\widehat{\pi}_{n},x) ( π n , x )
M ∗ ( π ^ n , x ) ≡ M ∗ ( D n , x ) . M_{*}(\widehat{\pi}_{n},x)\equiv M_{*}(D_{n},x). M ∗ ( π n , x ) ≡ M ∗ ( D n , x ) . Therefore, M ∗ M_{*} M ∗ P ^ ( Z ) × X \widehat{\mathcal{P}}(\mathcal{Z})\times\mathcal{X} P ( Z ) × X
P ^ ( Z ) = { n − 1 ∑ i = 1 n δ z i : z 1 , … , z n ∈ Z , n ≥ 1 } . \widehat{\mathcal{P}}(\mathcal{Z})=\left\{n^{-1}\sum_{i=1}^{n}\delta_{z_i}:z_1,\ldots,z_n\in\mathcal{Z},n\geq1\right\}. P ( Z ) = { n − 1 i = 1 ∑ n δ z i : z 1 , … , z n ∈ Z , n ≥ 1 } . is the space of empirical measures and Z = X × R \mathcal{Z} = \mathcal{X} \times\mathbb{R} Z = X × R M ∗ M_* M ∗ π ^ n \widehat{\pi}_n π n M ∗ M_* M ∗ [20 ] , where empirical measures are used as a permutation-invariant representation of the context data to facilitate a unified analysis across different context lengths and transformer architectures. In contrast, Lemma 1 shows that the use of π ^ n \widehat{\pi}_n π n M ∗ M_* M ∗ D n D_n D n
A General MI-RKHS Framework
In this section, we establish a general MI-RKHS framework that characterizes the regularity of M ∗ M_* M ∗ f f f F F F f ∈ H X f \in\mathcal{H}_X f ∈ H X ∥ f ∥ H X \|f\|_{\mathcal{H}_X} ∥ f ∥ H X
Assumption 2. E [ ∥ f ∥ H X 2 ] < ∞ \mathbb{E}[\|f\|_{\mathcal{H}_X}^2] < \infty E [ ∥ f ∥ H X 2 ] < ∞
Assumption 2 says that f f f ∥ ⋅ ∥ H X \|\cdot\|_{\mathcal{H}_X} ∥ ⋅ ∥ H X f f f (1) , (2) and independence between X n + 1 X_{n+1} X n + 1 ( f , D n ) (f, D_n) ( f , D n )
M ∗ ( π ^ n , x ) = E [ f ( X n + 1 ) + ϵ n + 1 ∣ π ^ n , X n + 1 = x ] = E [ f ∣ π ^ n ] ( x ) . (3) M_*(\widehat{\pi}_n, x) = \mathbb{E}[f(X_{n+1}) + \epsilon_{n+1} \mid\widehat{\pi}_n, X_{n+1} = x] = \mathbb{E}[f \mid\widehat{\pi}_n](x). \tag*{(3)} M ∗ ( π n , x ) = E [ f ( X n + 1 ) + ϵ n + 1 ∣ π n , X n + 1 = x ] = E [ f ∣ π n ] ( x ) . (3) Let H ∗ ( π ^ n ) = E [ f ∣ π ^ n ] H_*(\widehat{\pi}_n) = \mathbb{E}[f \mid\widehat{\pi}_n] H ∗ ( π n ) = E [ f ∣ π n ]
E [ ∥ H ∗ ( π ^ n ) ∥ H X 2 ] ≤ E [ ∥ f ∥ H X 2 ] < ∞ , \mathbb{E}[\|H_*(\widehat{\pi}_n)\|_{\mathcal{H}_X}^2] \leq\mathbb{E}[\|f\|_{\mathcal{H}_X}^2] < \infty, E [ ∥ H ∗ ( π n ) ∥ H X 2 ] ≤ E [ ∥ f ∥ H X 2 ] < ∞ , so H ∗ ( π ^ n ) ∈ H X H_*(\widehat{\pi}_n) \in\mathcal{H}_X H ∗ ( π n ) ∈ H X H ∗ H_* H ∗ P ^ ( Z ) \widehat{\mathcal{P}}(\mathcal{Z}) P ( Z ) H X \mathcal{H}_X H X H ∗ H_* H ∗
P ( Z ) = { π : π is a probability measure on Z with ∫ Z y 2 d π ( x , y ) < ∞ } . \mathcal{P}(\mathcal{Z}) = \left\{ \pi: \pi\text{ is a probability measure on } \mathcal{Z} \text{ with } \int_{\mathcal{Z}} y^2 \,\mathrm{d}\pi(x,y) < \infty\right\}. P ( Z ) = { π : π is a probability measure on Z with ∫ Z y 2 d π ( x , y ) < ∞ } . Any π ∈ P ( Z ) \pi\in\mathcal{P}(\mathcal{Z}) π ∈ P ( Z ) ∫ Z y 2 d π ( x , y ) < ∞ \int_{\mathcal{Z}} y^2 \,\mathrm{d}\pi(x,y) < \infty ∫ Z y 2 d π ( x , y ) < ∞ π \pi π Z \mathcal{Z} Z y y y P ^ ( Z ) ⊂ P ( Z ) \widehat{\mathcal{P}}(\mathcal{Z}) \subset\mathcal{P}(\mathcal{Z}) P ( Z ) ⊂ P ( Z ) H ∗ : P ^ ( Z ) → H X H_* : \widehat{\mathcal{P}}(\mathcal{Z}) \to\mathcal{H}_X H ∗ : P ( Z ) → H X P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) H ∗ H_* H ∗ H ∗ ∈ H Π H_* \in\mathcal{H}_{\Pi} H ∗ ∈ H Π H Π \mathcal{H}_{\Pi} H Π P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) H X \mathcal{H}_X H X H Π \mathcal{H}_{\Pi} H Π ⟨ ⋅ , ⋅ ⟩ H Π \langle\cdot,\cdot\rangle_{\mathcal{H}_{\Pi}} ⟨ ⋅ , ⋅ ⟩ H Π R ⟨ Π ⟩ : P ( Z ) × P ( Z ) → L ( H X ) R^{\langle\Pi\rangle} : \mathcal{P}(\mathcal{Z}) \times\mathcal{P}(\mathcal{Z}) \to\mathcal{L}(\mathcal{H}_X) R ⟨ Π ⟩ : P ( Z ) × P ( Z ) → L ( H X ) L ( H X ) \mathcal{L}(\mathcal{H}_X) L ( H X ) H X \mathcal{H}_X H X π ∈ P ( Z ) \pi\in\mathcal{P}(\mathcal{Z}) π ∈ P ( Z ) R π ⟨ Π ⟩ ( ⋅ ) = R ⟨ Π ⟩ ( ⋅ , π ) R^{\langle\Pi\rangle}_{\pi}(\cdot) = R^{\langle\Pi\rangle}(\cdot,\pi) R π ⟨ Π ⟩ ( ⋅ ) = R ⟨ Π ⟩ ( ⋅ , π ) R π ⟨ Π ⟩ R^{\langle\Pi\rangle}_{\pi} R π ⟨ Π ⟩ P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) L ( H X ) \mathcal{L}(\mathcal{H}_X) L ( H X ) g ∈ H X g \in\mathcal{H}_X g ∈ H X ( R π ⟨ Π ⟩ g ) ( ⋅ ) ≔ R ⟨ Π ⟩ ( ⋅ , π ) g ∈ H X (R^{\langle\Pi\rangle}_{\pi}g)(\cdot) \coloneqq R^{\langle\Pi\rangle}(\cdot,\pi)g \in\mathcal{H}_X ( R π ⟨ Π ⟩ g ) ( ⋅ ) : = R ⟨ Π ⟩ ( ⋅ , π ) g ∈ H X π \pi π g g g R π ⟨ Π ⟩ g R^{\langle\Pi\rangle}_{\pi}g R π ⟨ Π ⟩ g P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) H X \mathcal{H}_X H X H Π \mathcal{H}_{\Pi} H Π H ∈ H Π H \in\mathcal{H}_{\Pi} H ∈ H Π g ∈ H X g \in\mathcal{H}_X g ∈ H X π ∈ P ( Z ) \pi\in\mathcal{P}(\mathcal{Z}) π ∈ P ( Z )
Property (a): R π ⟨ Π ⟩ g ∈ H Π R^{\langle\Pi\rangle}_{\pi}g \in\mathcal{H}_{\Pi} R π ⟨ Π ⟩ g ∈ H Π
Property (b): ⟨ H ( π ) , g ⟩ H X = ⟨ H , R π ⟨ Π ⟩ g ⟩ H Π \langle H(\pi),g\rangle_{\mathcal{H}_X} = \langle H,R^{\langle\Pi\rangle}_{\pi}g\rangle_{\mathcal{H}_{\Pi}} ⟨ H ( π ) , g ⟩ H X = ⟨ H , R π ⟨ Π ⟩ g ⟩ H Π
The object R π ⟨ Π ⟩ g R^{\langle\Pi\rangle}_{\pi}g R π ⟨ Π ⟩ g R ⟨ Π ⟩ R^{\langle\Pi\rangle} R ⟨ Π ⟩ π \pi π g ∈ H X g \in\mathcal{H}_X g ∈ H X R π ⟨ Π ⟩ g R^{\langle\Pi\rangle}_{\pi}g R π ⟨ Π ⟩ g H Π \mathcal{H}_{\Pi} H Π H X \mathcal{H}_X H X H H H π \pi π g ∈ H X g \in\mathcal{H}_X g ∈ H X H Π \mathcal{H}_{\Pi} H Π [22 , 44 ] for further discussions.
We will use the above RKHS’s H X \mathcal{H}_{X} H X H Π \mathcal{H}_{\Pi} H Π M \mathcal{M} M P ( Z ) × X \mathcal{P}(\mathcal{Z}) \times\mathcal{X} P ( Z ) × X M ∗ M_* M ∗ P ( Z ) × X \mathcal{P}(\mathcal{Z}) \times\mathcal{X} P ( Z ) × X hyper-kernels defined on distributions [13 , 23 ] . For H ∈ H Π H \in\mathcal{H}_{\Pi} H ∈ H Π ( π , x ) ∈ P ( Z ) × X (\pi,x) \in\mathcal{P}(\mathcal{Z}) \times\mathcal{X} ( π , x ) ∈ P ( Z ) × X M H ( π , x ) = ⟨ H ( π ) , R x ⟨ X ⟩ ⟩ H X M_H(\pi,x) = \langle H(\pi), R_x^{\langle X\rangle}\rangle_{\mathcal{H}_X} M H ( π , x ) = ⟨ H ( π ) , R x ⟨ X ⟩ ⟩ H X
Define
M = { M H : H ∈ H Π } , (4) \mathcal{M} = \{M_H : H \in\mathcal{H}_{\Pi}\}, \tag*{(4)} M = { M H : H ∈ H Π } , (4) and inner product ⟨ M H , M H ′ ⟩ M ≜ ⟨ H , H ′ ⟩ H Π \langle M_H, M_{H'}\rangle_{\mathcal{M}} \triangleq\langle H,H'\rangle_{\mathcal{H}_{\Pi}} ⟨ M H , M H ′ ⟩ M ≜ ⟨ H , H ′ ⟩ H Π H , H ′ ∈ H Π H,H' \in\mathcal{H}_{\Pi} H , H ′ ∈ H Π H ∗ ∈ H Π H_* \in\mathcal{H}_{\Pi} H ∗ ∈ H Π
M ∗ ( π , x ) = ⟨ H ∗ ( π ) , R x ⟨ X ⟩ ⟩ H X = M H ∗ ( π , x ) . (5) M_*(\pi,x) = \langle H_*(\pi), R_x^{\langle X\rangle}\rangle_{\mathcal{H}_X} = M_{H_*}(\pi,x). \tag*{(5)} M ∗ ( π , x ) = ⟨ H ∗ ( π ) , R x ⟨ X ⟩ ⟩ H X = M H ∗ ( π , x ) . (5) So M ∗ = M H ∗ ∈ M M_* = M_{H_*} \in\mathcal{M} M ∗ = M H ∗ ∈ M M \mathcal{M} M H ↦ M H H \mapsto M_H H ↦ M H H Π \mathcal{H}_{\Pi} H Π M \mathcal{M} M
R ( ( π , x ) , ( π ′ , x ′ ) ) = ⟨ R π ⟨ Π ⟩ R x ⟨ X ⟩ , R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ ⟩ H Π (6) R((\pi,x),(\pi',x')) = \langle R_{\pi}^{\langle\Pi\rangle}R_x^{\langle X\rangle}, R_{\pi'}^{\langle\Pi\rangle}R_{x'}^{\langle X\rangle}\rangle_{\mathcal{H}_{\Pi}} \tag*{(6)} R (( π , x ) , ( π ′ , x ′ )) = ⟨ R π ⟨ Π ⟩ R x ⟨ X ⟩ , R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ ⟩ H Π (6) for ( π , x ) , ( π ′ , x ′ ) ∈ P ( Z ) × X (\pi,x),(\pi',x') \in\mathcal{P}(\mathcal{Z}) \times\mathcal{X} ( π , x ) , ( π ′ , x ′ ) ∈ P ( Z ) × X R x ⟨ X ⟩ , R x ′ ⟨ X ⟩ ∈ H X R_x^{\langle X\rangle}, R_{x'}^{\langle X\rangle} \in\mathcal{H}_X R x ⟨ X ⟩ , R x ′ ⟨ X ⟩ ∈ H X R π ⟨ Π ⟩ R x ⟨ X ⟩ R_{\pi}^{\langle\Pi\rangle}R_x^{\langle X\rangle} R π ⟨ Π ⟩ R x ⟨ X ⟩ R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ R_{\pi'}^{\langle\Pi\rangle}R_{x'}^{\langle X\rangle} R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ H Π \mathcal{H}_{\Pi} H Π R ( π , x ) ( ⋅ , ⋅ ) = R ( ( ⋅ , ⋅ ) , ( π , x ) ) R_{(\pi,x)}(\cdot,\cdot) = R((\cdot,\cdot),(\pi,x)) R ( π , x ) ( ⋅ , ⋅ ) = R (( ⋅ , ⋅ ) , ( π , x )) R ( π , x ) ∈ M R_{(\pi,x)} \in\mathcal{M} R ( π , x ) ∈ M H π , x = R π ⟨ Π ⟩ R x ⟨ X ⟩ H_{\pi,x} = R_{\pi}^{\langle\Pi\rangle}R_x^{\langle X\rangle} H π , x = R π ⟨ Π ⟩ R x ⟨ X ⟩ H Π \mathcal{H}_{\Pi} H Π
M H π , x ( π ′ , x ′ ) = ⟨ H π , x ( π ′ ) , R x ′ ⟨ X ⟩ ⟩ H X = ⟨ H π , x , R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ ⟩ H Π = ⟨ R π ⟨ Π ⟩ R x ⟨ X ⟩ , R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ ⟩ H Π = R ( π , x ) ( π ′ , x ′ ) , \begin{aligned}
M_{H_{\pi,x}}(\pi',x') &= \langle H_{\pi,x}(\pi'),R_{x'}^{\langle X\rangle}\rangle_{\mathcal{H}_X} = \langle H_{\pi,x},R_{\pi'}^{\langle\Pi\rangle}R_{x'}^{\langle X\rangle}\rangle_{\mathcal{H}_{\Pi}} \\
&= \langle R_{\pi}^{\langle\Pi\rangle}R_x^{\langle X\rangle},R_{\pi'}^{\langle\Pi\rangle}R_{x'}^{\langle X\rangle}\rangle_{\mathcal{H}_{\Pi}} = R_{(\pi,x)}(\pi',x'),
\end{aligned} M H π , x ( π ′ , x ′ ) = ⟨ H π , x ( π ′ ) , R x ′ ⟨ X ⟩ ⟩ H X = ⟨ H π , x , R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ ⟩ H Π = ⟨ R π ⟨ Π ⟩ R x ⟨ X ⟩ , R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ ⟩ H Π = R ( π , x ) ( π ′ , x ′ ) , implying that R ( π , x ) = M H π , x ∈ M R_{(\pi,x)} = M_{H_{\pi,x}} \in\mathcal{M} R ( π , x ) = M H π , x ∈ M M H ∈ M M_H \in\mathcal{M} M H ∈ M
⟨ M H , R ( π , x ) ⟩ M = ⟨ H , R π ⟨ Π ⟩ R x ⟨ X ⟩ ⟩ H Π = ⟨ H ( π ) , R x ⟨ X ⟩ ⟩ H X = M H ( π , x ) . (7) \langle M_H,R_{(\pi,x)}\rangle_{\mathcal{M}} = \langle H,R_{\pi}^{\langle\Pi\rangle}R_x^{\langle X\rangle}\rangle_{\mathcal{H}_{\Pi}} = \langle H(\pi),R_x^{\langle X\rangle}\rangle_{\mathcal{H}_X} = M_H(\pi,x). \tag*{(7)} ⟨ M H , R ( π , x ) ⟩ M = ⟨ H , R π ⟨ Π ⟩ R x ⟨ X ⟩ ⟩ H Π = ⟨ H ( π ) , R x ⟨ X ⟩ ⟩ H X = M H ( π , x ) . (7) In summarizing the above, we have proved the following result.
Proposition 1. M \mathcal{M} M P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) ⟨ ⋅ , ⋅ ⟩ M \langle\cdot,\cdot\rangle_{\mathcal{M}} ⟨ ⋅ , ⋅ ⟩ M R R R M ∗ M_* M ∗ (2) is an element of M \mathcal{M} M
Measure-Indexed Kernel Ridge Regression
Suppose that, in practice, we have B B B f 1 , … , f B f_1,\ldots,f_B f 1 , … , f B F F F b b b n b n_b n b D n b b = ( Z 1 b , … , Z n b b ) D_{n_b}^b = (Z_1^b,\ldots,Z_{n_b}^b) D n b b = ( Z 1 b , … , Z n b b ) N b N_b N b Z n b + 1 b , … , Z n b + N b b Z_{n_b+1}^b,\ldots,Z_{n_b+N_b}^b Z n b + 1 b , … , Z n b + N b b Z i b = ( X i b , Y i b ) Z_i^b = (X_i^b,Y_i^b) Z i b = ( X i b , Y i b ) X 1 b , … , X n b + N b b ∼ i . i . d . μ X_1^b,\ldots,X_{n_b+N_b}^b \overset{\mathrm{i.i.d.}}{\sim} \mu X 1 b , … , X n b + N b b ∼ i.i.d. μ
Y i b = f b ( X i b ) + ε i b , i = 1 , … , n b + N b . (8) Y_i^b = f_b(X_i^b) + \varepsilon_i^b,\qquad i=1,\ldots,n_b+N_b. \tag*{(8)} Y i b = f b ( X i b ) + ε i b , i = 1 , … , n b + N b . (8) The errors ε i b \varepsilon_i^b ε i b σ ε 2 \sigma_\varepsilon^2 σ ε 2 f b f_b f b X 1 b , … , X n b + N b b X_1^b,\ldots,X_{n_b+N_b}^b X 1 b , … , X n b + N b b π ^ n b b = n b − 1 ∑ i = 1 n b δ Z i b \widehat{\pi}_{n_b}^b = n_b^{-1}\sum_{i=1}^{n_b}\delta_{Z_i^b} π n b b = n b − 1 ∑ i = 1 n b δ Z i b D n b b D_{n_b}^b D n b b (3) , it can be shown that
E [ Y n b + i b ∣ π ^ n b b , X n b + i b = x ] = M ∗ ( π ^ n b b , x ) . \mathbb{E}[Y_{n_b+i}^b\mid\widehat{\pi}_{n_b}^b,X_{n_b+i}^b=x] = M_*(\widehat{\pi}_{n_b}^b,x). E [ Y n b + i b ∣ π n b b , X n b + i b = x ] = M ∗ ( π n b b , x ) . That is, M ∗ M_* M ∗ Y n b + i b Y^b_{n_b+i} Y n b + i b π ^ n b b , X n b + i b \widehat{\pi}^b_{n_b}, X^b_{n_b+i} π n b b , X n b + i b M ∗ M_* M ∗
M ^ n , N , B , λ = arg min M ∈ M ℓ n , N , B , λ ( M ) ≡ arg min M ∈ M { 1 2 N ∑ b = 1 B ∑ i = 1 N b ∣ Y n b + i b − M ( π ^ n b b , X n b + i b ) ∣ 2 + λ 2 ∥ M ∥ M 2 } . (9) \begin{aligned}
\widehat{M}_{\mathbf{n},\mathbf{N},B,\lambda}=\underset{M\in\mathcal{M}}{\arg\min}\ \ell_{\mathbf{n},\mathbf{N},B,\lambda}(M) \\
&\equiv\underset{M\in\mathcal{M}}{\arg\min}\left\{\frac{1}{2N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left|Y^b_{n_b+i}-M\left(\widehat{\pi}^b_{n_b},X^b_{n_b+i}\right)\right|^2+\frac{\lambda}{2}\lVert M\rVert_{\mathcal{M}}^2\right\}. \tag*{(9)}
\end{aligned} M n , N , B , λ = M ∈ M arg min ℓ n , N , B , λ ( M ) ≡ M ∈ M arg min { 2 N 1 b = 1 ∑ B i = 1 ∑ N b Y n b + i b − M ( π n b b , X n b + i b ) 2 + 2 λ ∥ M ∥ M 2 } . (9) where λ > 0 \lambda>0 λ > 0 ∥ ⋅ ∥ M \lVert\cdot\rVert_{\mathcal{M}} ∥ ⋅ ∥ M M \mathcal{M} M n = ( n 1 , … , n B ) \mathbf{n}=(n_1,\ldots,n_B) n = ( n 1 , … , n B ) N = ( N 1 , … , N B ) \mathbf{N}=(N_1,\ldots,N_B) N = ( N 1 , … , N B ) N = N 1 + ⋯ + N B N=N_1+\cdots+N_B N = N 1 + ⋯ + N B π ^ \widehat{\pi} π x x x M ^ n , N , B , λ ( π ^ , x ) \widehat{M}_{\mathbf{n},\mathbf{N},B,\lambda}(\widehat{\pi},x) M n , N , B , λ ( π , x )
By representer theorem [36 ] , M ^ n , N , B , λ \widehat{M}_{\mathbf{n},\mathbf{N},B,\lambda} M n , N , B , λ
M ^ n , N , B , λ ( π ^ , x ) = ∑ b = 1 B ∑ i = 1 N b c i b R ( ( π ^ , x ) , ( π ^ n b b , X n b + i b ) ) . \widehat{M}_{\mathbf{n},\mathbf{N},B,\lambda}(\widehat{\pi},x)=\sum_{b=1}^{B}\sum_{i=1}^{N_b}c_i^bR\left((\widehat{\pi},x),(\widehat{\pi}^b_{n_b},X^b_{n_b+i})\right). M n , N , B , λ ( π , x ) = b = 1 ∑ B i = 1 ∑ N b c i b R ( ( π , x ) , ( π n b b , X n b + i b ) ) . where the coefficient vector c = ( c 1 1 , … , c N 1 1 , … , c 1 B , … , c N B B ) ⊤ \mathbf{c}=(c_1^1,\ldots,c_{N_1}^1,\ldots,c_1^B,\ldots,c_{N_B}^B)^\top c = ( c 1 1 , … , c N 1 1 , … , c 1 B , … , c N B B ) ⊤
c = ( R N × N + N λ I N × N ) − 1 Y , \mathbf{c}=\left(\mathbf{R}_{N\times N}+N\lambda\mathbf{I}_{N\times N}\right)^{-1}\mathbf{Y}, c = ( R N × N + N λ I N × N ) − 1 Y , where R N × N \mathbf{R}_{N\times N} R N × N N × N N\times N N × N R R R M \mathcal{M} M ( ( b , i ) , ( b ′ , i ′ ) ) ((b,i),(b',i')) (( b , i ) , ( b ′ , i ′ )) R N × N \mathbf{R}_{N\times N} R N × N R ( ( π ^ n b b , X n b + i b ) , ( π ^ n b ′ b ′ , X n b ′ + i ′ b ′ ) ) R((\widehat{\pi}^b_{n_b},X^b_{n_b+i}),(\widehat{\pi}^{b'}_{n_{b'}},X^{b'}_{n_{b'}+i'})) R (( π n b b , X n b + i b ) , ( π n b ′ b ′ , X n b ′ + i ′ b ′ )) Y \mathbf{Y} Y Y = ( Y n 1 + 1 1 , … , Y n 1 + N 1 1 , … , Y n B + 1 B , … , Y n B + N B B ) ⊤ \mathbf{Y}=(Y^1_{n_1+1},\ldots,Y^1_{n_1+N_1},\ldots,Y^B_{n_B+1},\ldots,Y^B_{n_B+N_B})^\top Y = ( Y n 1 + 1 1 , … , Y n 1 + N 1 1 , … , Y n B + 1 B , … , Y n B + N B B ) ⊤ [15 ] , we suggest to select λ \lambda λ
H λ = R N × N ( R N × N + N λ I N × N ) − 1 . \mathbf{H}_{\lambda}=\mathbf{R}_{N\times N}\left(\mathbf{R}_{N\times N}+N\lambda\mathbf{I}_{N\times N}\right)^{-1}. H λ = R N × N ( R N × N + N λ I N × N ) − 1 . The GCV criterion is
GCV ( λ ) = N − 1 ∣ ( I N × N − H λ ) Y ∣ 2 ( 1 − N − 1 trace ( H λ ) ) 2 , λ > 0. \operatorname{GCV}(\lambda)=\frac{N^{-1}\left|(\mathbf{I}_{N\times N}-\mathbf{H}_{\lambda})\mathbf{Y}\right|^2}{\left(1-N^{-1}\operatorname{trace}(\mathbf{H}_{\lambda})\right)^2},\quad\lambda>0. GCV ( λ ) = ( 1 − N − 1 trace ( H λ ) ) 2 N − 1 ∣ ( I N × N − H λ ) Y ∣ 2 , λ > 0. We then choose λ ^ G C V = arg min λ GCV ( λ ) \widehat{\lambda}_{\mathrm{GCV}}=\arg\min_{\lambda}\operatorname{GCV}(\lambda) λ GCV = arg min λ GCV ( λ )
Asymptotic Properties
In this section, we will derive the convergence rate and asymptotic distribution for M ^ n , N , B , λ \widehat{M}_{\mathbf{n},\mathbf{N},B,\lambda} M n , N , B , λ
Spectral Property of MI-RKHS
In this section, we will establish the spectral property of M \mathcal{M} M
Assumption 3. C Π ≜ sup π ∈ P ( Z ) ∥ R ( Π ) ( π , π ) ∥ o p < ∞ C_{\Pi}\triangleq\sup_{\pi\in\mathcal{P}(\mathcal{Z})}\lVert R^{(\Pi)}(\pi,\pi)\rVert_{\mathrm{op}}<\infty C Π ≜ sup π ∈ P ( Z ) ∥ R ( Π ) ( π , π ) ∥ op < ∞ ∥ ⋅ ∥ o p \lVert\cdot\rVert_{\mathrm{op}} ∥ ⋅ ∥ op
Assumption 3 bounds the operator norm of the kernel R ⟨ Π ⟩ R^{\langle\Pi\rangle} R ⟨ Π ⟩ P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) M , M ′ ∈ M M, M' \in\mathcal{M} M , M ′ ∈ M
V ( M , M ′ ) = 1 N ∑ b = 1 B N b E [ M ( π ^ n b b , X n b + 1 b ) M ′ ( π ^ n b b , X n b + 1 b ) ] . V(M,M')=\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}\left[M(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})M'(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})\right]. V ( M , M ′ ) = N 1 b = 1 ∑ B N b E [ M ( π n b b , X n b + 1 b ) M ′ ( π n b b , X n b + 1 b ) ] . By (7) , we have the following
V ( M , M ′ ) = 1 N ∑ b = 1 B N b E [ ⟨ M , R ( π ^ n b b , X n b + 1 b ) ⟩ M ⟨ M ′ , R ( π ^ n b b , X n b + 1 b ) ⟩ M ] = ⟨ 1 N ∑ b = 1 B N b E [ R ( π ^ n b b , X n b + 1 b ) ⊗ R ( π ^ n b b , X n b + 1 b ) ] M , M ′ ⟩ M ≡ ⟨ C M , M ′ ⟩ M , \begin{aligned}
V(M,M')&=\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}\left[\left\langle M,R_{(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})}\right\rangle_{\mathcal{M}}\left\langle M',R_{(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})}\right\rangle_{\mathcal{M}}\right]\\
&=\left\langle\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}\left[R_{(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})}\otimes R_{(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})}\right]M,M'\right\rangle_{\mathcal{M}}\\
&\equiv\left\langle CM,M'\right\rangle_{\mathcal{M}},
\end{aligned} V ( M , M ′ ) = N 1 b = 1 ∑ B N b E [ ⟨ M , R ( π n b b , X n b + 1 b ) ⟩ M ⟨ M ′ , R ( π n b b , X n b + 1 b ) ⟩ M ] = ⟨ N 1 b = 1 ∑ B N b E [ R ( π n b b , X n b + 1 b ) ⊗ R ( π n b b , X n b + 1 b ) ] M , M ′ ⟩ M ≡ ⟨ C M , M ′ ⟩ M , where
C n , N , B ≜ 1 N ∑ b = 1 B N b E [ R ( π ^ n b b , X n b + 1 b ) ⊗ R ( π ^ n b b , X n b + 1 b ) ] C_{n,N,B}\triangleq\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}\left[R_{(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})}\otimes R_{(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})}\right] C n , N , B ≜ N 1 b = 1 ∑ B N b E [ R ( π n b b , X n b + 1 b ) ⊗ R ( π n b b , X n b + 1 b ) ] is a nonnegative definite rank-one operator from M \mathcal{M} M M \mathcal{M} M C = C n , N , B C=C_{n,N,B} C = C n , N , B (6) and (7) , for any ( π , x ) ∈ P ( Z ) × X (\pi,x)\in\mathcal{P}(\mathcal{Z})\times\mathcal{X} ( π , x ) ∈ P ( Z ) × X
∥ R ( π , x ) ∥ M 2 = R ( ( π , x ) , ( π , x ) ) (by xref‘(7)‘) = ⟨ R π ⟨ Π ⟩ R x ⟨ X ⟩ , R π ⟨ Π ⟩ R x ⟨ X ⟩ ⟩ H Π (by xref‘(6)‘) = ⟨ R x ⟨ X ⟩ , R ⟨ Π ⟩ ( π , π ) R x ⟨ X ⟩ ⟩ H X (by Property (b)) ≤ ∥ R ⟨ Π ⟩ ( π , π ) ∥ o p ∥ R x ⟨ X ⟩ ∥ H X 2 = ∥ R ⟨ Π ⟩ ( π , π ) ∥ o p R ⟨ X ⟩ ( x , x ) ≤ C Π C X < ∞ (by Assumptions 1 and 3). (10) \begin{aligned}
\left\|R_{(\pi,x)}\right\|_{\mathcal{M}}^{2}
&=R((\pi,x),(\pi,x)) &&\text{(by {xref}`(7)`)}\\
&=\left\langle R_{\pi}^{\langle\Pi\rangle}R_{x}^{\langle X\rangle},R_{\pi}^{\langle\Pi\rangle}R_{x}^{\langle X\rangle}\right\rangle_{\mathcal{H}_{\Pi}} &&\text{(by {xref}`(6)`)}\\
&=\left\langle R_{x}^{\langle X\rangle},R^{\langle\Pi\rangle}(\pi,\pi)R_{x}^{\langle X\rangle}\right\rangle_{\mathcal{H}_{X}} &&\text{(by Property (b))}\\
&\leq\left\|R^{\langle\Pi\rangle}(\pi,\pi)\right\|_{\mathrm{op}}\left\|R_{x}^{\langle X\rangle}\right\|_{\mathcal{H}_{X}}^{2}\\
&=\left\|R^{\langle\Pi\rangle}(\pi,\pi)\right\|_{\mathrm{op}}R^{\langle X\rangle}(x,x)\leq C_{\Pi}C_{X}<\infty&&\text{(by Assumptions 1 and 3).} \tag*{(10)}
\end{aligned} R ( π , x ) M 2 = R (( π , x ) , ( π , x )) = ⟨ R π ⟨ Π ⟩ R x ⟨ X ⟩ , R π ⟨ Π ⟩ R x ⟨ X ⟩ ⟩ H Π = ⟨ R x ⟨ X ⟩ , R ⟨ Π ⟩ ( π , π ) R x ⟨ X ⟩ ⟩ H X ≤ R ⟨ Π ⟩ ( π , π ) op R x ⟨ X ⟩ H X 2 = R ⟨ Π ⟩ ( π , π ) op R ⟨ X ⟩ ( x , x ) ≤ C Π C X < ∞ (by xref ‘(7)‘) (by xref ‘(6)‘) (by Property (b)) (by Assumptions 1 and 3). (10) Consequently,
trace ( C ) = 1 N ∑ b = 1 B N b E [ ∥ R ( π ^ n b b , X n b + 1 b ) ∥ M 2 ] ≤ C Π C X , \operatorname{trace}(C)=\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}\left[\left\|R_{(\widehat{\pi}_{n_b}^{b},X_{n_b+1}^{b})}\right\|_{\mathcal{M}}^{2}\right]\leq C_{\Pi}C_{X}, trace ( C ) = N 1 b = 1 ∑ B N b E [ R ( π n b b , X n b + 1 b ) M 2 ] ≤ C Π C X , i.e., C C C ker ( C ) = { M ∈ M : C M = 0 } \operatorname{ker}(C)=\{M\in\mathcal{M}:CM=0\} ker ( C ) = { M ∈ M : C M = 0 } C C C ker ( C ) ⊥ \operatorname{ker}(C)^{\perp} ker ( C ) ⊥ ⟨ ⋅ , ⋅ ⟩ M \langle\cdot,\cdot\rangle_{\mathcal{M}} ⟨ ⋅ , ⋅ ⟩ M π ^ n b b \widehat{\pi}_{n_b}^{b} π n b b P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) C M = 0 CM=0 C M = 0 M M M ker ( C ) ≠ { 0 } \operatorname{ker}(C)\ne\{0\} ker ( C ) = { 0 } [11 ] ), there exist a positive nondecreasing sequence { ρ k } k ≥ 1 \{\rho_k\}_{k\geq1} { ρ k } k ≥ 1 { φ k } k ≥ 1 ⊆ ker ( C ) ⊥ \{\varphi_k\}_{k\geq1}\subseteq\operatorname{ker}(C)^{\perp} { φ k } k ≥ 1 ⊆ ker ( C ) ⊥
V ( φ k , φ l ) = ⟨ C φ k , φ l ⟩ M = δ k l , ⟨ φ k , φ l ⟩ M = ρ k δ k l , k , l ≥ 1 , (11) V(\varphi_k,\varphi_l)=\langle C\varphi_k,\varphi_l\rangle_{\mathcal{M}}=\delta_{kl},\qquad\langle\varphi_k,\varphi_l\rangle_{\mathcal{M}}=\rho_k\delta_{kl},\quad k,l\geq1, \tag*{(11)} V ( φ k , φ l ) = ⟨ C φ k , φ l ⟩ M = δ k l , ⟨ φ k , φ l ⟩ M = ρ k δ k l , k , l ≥ 1 , (11) where δ k l \delta_{kl} δ k l ker ( C ) = { 0 } \operatorname{ker}(C)=\{0\} ker ( C ) = { 0 } ker ( C ) ⊥ = M \operatorname{ker}(C)^{\perp}=\mathcal{M} ker ( C ) ⊥ = M { φ k } k ≥ 1 \{\varphi_k\}_{k\geq1} { φ k } k ≥ 1 M \mathcal{M} M ker ( C ) \operatorname{ker}(C) ker ( C )
Assumption 4. There exists a constant C φ C_{\varphi} C φ n n n N N N B B B sup k ≥ 1 ∥ φ k ∥ ∞ ≤ C φ \sup_{k\geq1}\|\varphi_k\|_{\infty}\leq C_{\varphi} sup k ≥ 1 ∥ φ k ∥ ∞ ≤ C φ
Assumption 4 requires that the basis functions φ k \varphi_k φ k M \mathcal{M} M C φ C_\varphi C φ n \mathbf{n} n N \mathbf{N} N B \mathbf{B} B
For M , M ′ ∈ M M,M^{\prime}\in\mathcal{M} M , M ′ ∈ M ⟨ M , M ′ ⟩ = V ( M , M ′ ) + λ ⟨ M , M ′ ⟩ M \langle M,M^{\prime}\rangle=V(M,M^{\prime})+\lambda\langle M,M^{\prime}\rangle_{\mathcal{M}} ⟨ M , M ′ ⟩ = V ( M , M ′ ) + λ ⟨ M , M ′ ⟩ M ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ ⟨ ⋅ , ⋅ ⟩ M \langle\cdot,\cdot\rangle_{\mathcal{M}} ⟨ ⋅ , ⋅ ⟩ M M \mathcal{M} M M \mathcal{M} M ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ M ∈ M M\in\mathcal{M} M ∈ M M = M 1 + M 2 M=M_1+M_2 M = M 1 + M 2 M 1 ∈ ker ( C ) ⊥ M_1\in\operatorname{ker}(C)^{\perp} M 1 ∈ ker ( C ) ⊥ M 2 ∈ ker ( C ) M_2\in\operatorname{ker}(C) M 2 ∈ ker ( C ) M 1 = ∑ k ≥ 1 m k φ k M_1=\sum_{k\geq1}m_k\varphi_k M 1 = ∑ k ≥ 1 m k φ k
V ( M 1 , M 2 ) = ∑ k ≥ 1 ρ k m k 2 ≤ ρ 1 − 1 ∑ k ≥ 1 ρ k m k 2 = ρ 1 − 1 ∥ M 1 ∥ M 2 . V(M_1,M_2)=\sum_{k\geq1}\rho_km_k^2\leq\rho_1^{-1}\sum_{k\geq1}\rho_km_k^2=\rho_1^{-1}\lVert M_1\rVert_{\mathcal{M}}^2. V ( M 1 , M 2 ) = k ≥ 1 ∑ ρ k m k 2 ≤ ρ 1 − 1 k ≥ 1 ∑ ρ k m k 2 = ρ 1 − 1 ∥ M 1 ∥ M 2 . Since ∥ M ∥ M 2 = ∥ M 1 ∥ M 2 + ∥ M 2 ∥ M 2 ≥ ∥ M 1 ∥ M 2 \lVert M\rVert_{\mathcal{M}}^2=\lVert M_1\rVert_{\mathcal{M}}^2+\lVert M_2\rVert_{\mathcal{M}}^2\geq\lVert M_1\rVert_{\mathcal{M}}^2 ∥ M ∥ M 2 = ∥ M 1 ∥ M 2 + ∥ M 2 ∥ M 2 ≥ ∥ M 1 ∥ M 2
∥ M ∥ 2 ≜ V ( M , M ) + λ ∥ M ∥ M 2 = V ( M 1 , M 1 ) + λ ∥ M ∥ M 2 ≤ ρ 1 − 1 ∥ M 1 ∥ M 2 + λ ∥ M ∥ M 2 ≤ ( ρ 1 − 1 + λ ) ∥ M ∥ M 2 . \begin{aligned}
\lVert M\rVert^2 &\triangleq V(M,M)+\lambda\lVert M\rVert_{\mathcal{M}}^2=V(M_1,M_1)+\lambda\lVert M\rVert_{\mathcal{M}}^2\\
&\leq\rho_1^{-1}\lVert M_1\rVert_{\mathcal{M}}^2+\lambda\lVert M\rVert_{\mathcal{M}}^2\leq(\rho_1^{-1}+\lambda)\lVert M\rVert_{\mathcal{M}}^2.
\end{aligned} ∥ M ∥ 2 ≜ V ( M , M ) + λ ∥ M ∥ M 2 = V ( M 1 , M 1 ) + λ ∥ M ∥ M 2 ≤ ρ 1 − 1 ∥ M 1 ∥ M 2 + λ ∥ M ∥ M 2 ≤ ( ρ 1 − 1 + λ ) ∥ M ∥ M 2 . On the other hand, ∥ M ∥ 2 ≥ λ ∥ M ∥ M 2 \lVert M\rVert^2\geq\lambda\lVert M\rVert_{\mathcal{M}}^2 ∥ M ∥ 2 ≥ λ ∥ M ∥ M 2 ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ ∥ ⋅ ∥ M \lVert\cdot\rVert_{\mathcal{M}} ∥ ⋅ ∥ M
Let K : ( P ( Z ) × X ) 2 → R K:(\mathcal{P}(\mathcal{Z})\times\mathcal{X})^2\to\mathbb{R} K : ( P ( Z ) × X ) 2 → R M \mathcal{M} M ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ ( π ^ , x ) ∈ P ( Z ) × X (\widehat{\pi},x)\in\mathcal{P}(\mathcal{Z})\times\mathcal{X} ( π , x ) ∈ P ( Z ) × X M ∈ M M\in\mathcal{M} M ∈ M
K ( π ^ , x ) ≜ K ( ( ⋅ , ⋅ ) , ( π ^ , x ) ) ∈ M , ⟨ M , K ( π ^ , x ) ⟩ = M ( π ^ , x ) . K_{(\widehat{\pi},x)}\triangleq K((\cdot,\cdot),(\widehat{\pi},x))\in\mathcal{M},\quad\langle M,K_{(\widehat{\pi},x)}\rangle=M(\widehat{\pi},x). K ( π , x ) ≜ K (( ⋅ , ⋅ ) , ( π , x )) ∈ M , ⟨ M , K ( π , x ) ⟩ = M ( π , x ) . For any M ∈ M M\in\mathcal{M} M ∈ M λ ⟨ M , ⋅ ⟩ M \lambda\langle M,\cdot\rangle_{\mathcal{M}} λ ⟨ M , ⋅ ⟩ M ( M , ∥ ⋅ ∥ ) (\mathcal{M},\lVert\cdot\rVert) ( M , ∥ ⋅ ∥) P λ : M → M P_\lambda:\mathcal{M}\to\mathcal{M} P λ : M → M M , M ′ ∈ M M,M^{\prime}\in\mathcal{M} M , M ′ ∈ M
⟨ P λ M , M ′ ⟩ = λ ⟨ M , M ′ ⟩ M . \langle P_\lambda M,M^{\prime}\rangle=\lambda\langle M,M^{\prime}\rangle_{\mathcal{M}}. ⟨ P λ M , M ′ ⟩ = λ ⟨ M , M ′ ⟩ M . Convergence Rate and Asymptotic Normality
In this section, we derive the convergence rate and asymptotic distribution for M ^ n , N , B , λ \widehat{M}_{n,N,B,\lambda} M n , N , B , λ b = 1 , … , B b=1,\ldots,B b = 1 , … , B
V b 2 ( n b ) = E [ ∣ f b ( X n b + 1 b ) − E [ f b ∣ π ^ n b b ] ( X n b + 1 b ) ∣ 2 ] , V_b^2(n_b)=\mathbb{E}\left[\left|f_b(X_{n_b+1}^b)-\mathbb{E}[f_b\mid\widehat{\pi}_{n_b}^b](X_{n_b+1}^b)\right|^2\right], V b 2 ( n b ) = E [ f b ( X n b + 1 b ) − E [ f b ∣ π n b b ] ( X n b + 1 b ) 2 ] , and V 2 ( n ) = max 1 ≤ b ≤ B V b 2 ( n b ) V^2(\mathbf{n})=\max_{1\leq b\leq B}V_b^2(n_b) V 2 ( n ) = max 1 ≤ b ≤ B V b 2 ( n b ) V b 2 ( n b ) V_b^2(n_b) V b 2 ( n b ) f b f_b f b b b b V 2 ( n ) V^2(\mathbf{n}) V 2 ( n ) B B B ϵ n b , i b = Y n b + i b − M ∗ ( π ^ n b b , X n b + i b ) \epsilon_{n_b,i}^b=Y_{n_b+i}^b-M_*(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b) ϵ n b , i b = Y n b + i b − M ∗ ( π n b b , X n b + i b ) i = 1 , … , N b i=1,\ldots,N_b i = 1 , … , N b ε i b \varepsilon_i^b ε i b (8) . We further assume that these error terms have finite second moments which is a mild condition commonly used in nonparametric literature.
Assumption 5. There are positive constants σ e 2 \sigma_e^2 σ e 2 σ ε 2 \sigma_\varepsilon^2 σ ε 2 b = 1 , … , B b=1,\ldots,B b = 1 , … , B
E [ ∣ ϵ n b , i b ∣ 2 ∣ π ^ n b b , X n b + i b ] ≤ σ e 2 almost surely , and E [ ∣ ε i b ∣ 2 ] = σ ε 2 . \mathbb{E}\left[\left|\epsilon_{n_b,i}^b\right|^2\mid\widehat{\pi}_{n_b}^b,X_{n_b+i}^b\right]\leq\sigma_e^2\ \text{almost surely},\quad\text{and}\quad\mathbb{E}\left[\left|\varepsilon_i^b\right|^2\right]=\sigma_\varepsilon^2. E [ ϵ n b , i b 2 ∣ π n b b , X n b + i b ] ≤ σ e 2 almost surely , and E [ ε i b 2 ] = σ ε 2 . Let
M h = { M ∈ M : ∥ M ∥ ≤ C φ − 1 h 1 / 2 } and a h = ∫ 0 1 log ( 1 + D ( M h , ϵ , ∥ ⋅ ∥ ∞ ) ) d ϵ , \mathcal{M}_h=\left\{M\in\mathcal{M}:\lVert M\rVert\leq C_\varphi^{-1}h^{1/2}\right\}\ \text{and}\ a_h=\int_0^1\sqrt{\log(1+D(\mathcal{M}_h,\epsilon,\lVert\cdot\rVert_\infty))}\,\mathrm{d}\epsilon, M h = { M ∈ M : ∥ M ∥ ≤ C φ − 1 h 1/2 } and a h = ∫ 0 1 log ( 1 + D ( M h , ϵ , ∥ ⋅ ∥ ∞ )) d ϵ , where h = ( ∑ k ≥ 1 1 k + λ ρ k ) − 1 h = \left(\sum_{k\geq1} \frac{1}{k+\lambda\rho_k}\right)^{-1} h = ( ∑ k ≥ 1 k + λ ρ k 1 ) − 1 D ( M h , ϵ , ∥ ⋅ ∥ ∞ ) D(\mathcal{M}_h,\epsilon,\lVert\cdot\rVert_\infty) D ( M h , ϵ , ∥ ⋅ ∥ ∞ ) ϵ \epsilon ϵ M h \mathcal{M}_h M h ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ M ^ n , N , B , λ \widehat{M}_{n,N,B,\lambda} M n , N , B , λ
Theorem 2. Suppose that Assumptions 1-5 are satisfied and as n , N , B → ∞ n,N,B\to\infty n , N , B → ∞ n b n_b n b N b N_b N b B B B
a h ∑ b = 1 B N b 2 = o ( N h ) . (12) a_h\sqrt{\sum_{b=1}^{B}N_b^2}=o(Nh). \tag*{(12)} a h b = 1 ∑ B N b 2 = o ( N h ) . (12) Then ∥ M ^ n , N , B , λ − M ∗ ∥ = O P ( λ 1 / 2 + U ( n , N , B , λ ) ( N h ) − 1 / 2 ) \lVert\widehat{M}_{n,N,B,\lambda}-M_*\rVert=O_{\mathbb{P}}\left(\lambda^{1/2}+U(n,N,B,\lambda)(Nh)^{-1/2}\right) ∥ M n , N , B , λ − M ∗ ∥ = O P ( λ 1/2 + U ( n , N , B , λ ) ( N h ) − 1/2 )
U ( n , N , B , λ ) = max { 1 , ( λ max 1 ≤ b ≤ B N b ) 1 / 2 , V ( n ) ( max 1 ≤ b ≤ B N b ) 1 / 2 } . U(n,N,B,\lambda)=\max\left\{1,\left(\lambda\max_{1\leq b\leq B}N_b\right)^{1/2},V(n)\left(\max_{1\leq b\leq B}N_b\right)^{1/2}\right\}. U ( n , N , B , λ ) = max { 1 , ( λ 1 ≤ b ≤ B max N b ) 1/2 , V ( n ) ( 1 ≤ b ≤ B max N b ) 1/2 } . We will demonstrate the rate of convergence of Theorem 2 under a concrete setting. For technical convenience, consider the balanced case n b = n n_b=n n b = n N b = T N_b=T N b = T b = 1 , … , B b=1,\ldots,B b = 1 , … , B N = T B N=TB N = T B ρ k ≍ k 2 γ \rho_k\asymp k^{2\gamma} ρ k ≍ k 2 γ c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 γ > 1 / 2 \gamma>1/2 γ > 1/2
c 1 k 2 γ ≤ ρ k ≤ c 2 k 2 γ , k ≥ 1. (13) c_1 k^{2\gamma}\leq\rho_k\leq c_2 k^{2\gamma},\quad k\geq1. \tag*{(13)} c 1 k 2 γ ≤ ρ k ≤ c 2 k 2 γ , k ≥ 1. (13) Under (13) , it can be checked that λ ≍ h 2 γ \lambda\asymp h^{2\gamma} λ ≍ h 2 γ C C C k − 2 γ k^{-2\gamma} k − 2 γ
log D ( B M , ϵ , ∥ ⋅ ∥ ∞ ) ≲ ϵ − 1 / γ , for ϵ > 0 , (14) \log D(\mathbb{B}_{\mathcal{M}},\epsilon,\lVert\cdot\rVert_\infty)\lesssim\epsilon^{-1/\gamma},\quad\text{for }\epsilon>0, \tag*{(14)} log D ( B M , ϵ , ∥ ⋅ ∥ ∞ ) ≲ ϵ − 1/ γ , for ϵ > 0 , (14) where B M = { M ∈ M : ∥ M ∥ M ≤ 1 } \mathbb{B}_{\mathcal{M}}=\{M\in\mathcal{M}:\lVert M\rVert_{\mathcal{M}}\leq1\} B M = { M ∈ M : ∥ M ∥ M ≤ 1 } M \mathcal{M} M 1 / γ 1/\gamma 1/ γ (14) is consistent with the eigenvalue decay of C C C [9 ] ). Since M h ⊂ C φ − 1 ( h / λ ) 1 / 2 B M \mathcal{M}_h\subset C_\varphi^{-1}(h/\lambda)^{1/2}\mathbb{B}_{\mathcal{M}} M h ⊂ C φ − 1 ( h / λ ) 1/2 B M
log D ( M h , ϵ , ∥ ⋅ ∥ ∞ ) ≤ log D ( C φ − 1 ( h / λ ) 1 / 2 B M , ϵ , ∥ ⋅ ∥ ∞ ) = log D ( B M , C φ ( h / λ ) − 1 / 2 ϵ , ∥ ⋅ ∥ ∞ ) ≲ ( h / λ ) 1 / ( 2 γ ) ϵ − 1 / γ . \begin{aligned}
\log D(\mathcal{M}_h,\epsilon,\lVert\cdot\rVert_\infty)\leq\log D(C_\varphi^{-1}(h/\lambda)^{1/2}\mathbb{B}_{\mathcal{M}},\epsilon,\lVert\cdot\rVert_\infty) \\
&=\log D(\mathbb{B}_{\mathcal{M}},C_\varphi(h/\lambda)^{-1/2}\epsilon,\lVert\cdot\rVert_\infty)\lesssim(h/\lambda)^{1/(2\gamma)}\epsilon^{-1/\gamma}.
\end{aligned} log D ( M h , ϵ , ∥ ⋅ ∥ ∞ ) ≤ log D ( C φ − 1 ( h / λ ) 1/2 B M , ϵ , ∥ ⋅ ∥ ∞ ) = log D ( B M , C φ ( h / λ ) − 1/2 ϵ , ∥ ⋅ ∥ ∞ ) ≲ ( h / λ ) 1/ ( 2 γ ) ϵ − 1/ γ . Consequently, a h = O ( h ( 1 − 2 γ ) / ( 4 γ ) ) a_h=O(h^{(1-2\gamma)/(4\gamma)}) a h = O ( h ( 1 − 2 γ ) / ( 4 γ ) ) h ≍ N − 1 / ( 2 γ + 1 ) h\asymp N^{-1/(2\gamma+1)} h ≍ N − 1/ ( 2 γ + 1 ) T = o ( B ( 2 γ − 1 ) 2 / ( 6 γ − 1 ) ) T=o(B^{(2\gamma-1)^2/(6\gamma-1)}) T = o ( B ( 2 γ − 1 ) 2 / ( 6 γ − 1 ) ) B B B T T T (12) is satisfied. Meanwhile, it can be checked that
λ max 1 ≤ b ≤ B N b = λ T ≍ T N − 2 γ / ( 2 γ + 1 ) = ( T B − 2 γ ) 1 / ( 2 γ + 1 ) = O ( 1 ) . \lambda\max_{1\leq b\leq B}N_b=\lambda T\asymp TN^{-2\gamma/(2\gamma+1)}=(TB^{-2\gamma})^{1/(2\gamma+1)}=O(1). λ 1 ≤ b ≤ B max N b = λ T ≍ T N − 2 γ / ( 2 γ + 1 ) = ( T B − 2 γ ) 1/ ( 2 γ + 1 ) = O ( 1 ) . If we further assume that V ( n ) = O ( T − 1 / 2 ) V(n)=O(T^{-1/2}) V ( n ) = O ( T − 1/2 ) f b f_b f b T T T U ( n , N , B , λ ) = O ( 1 ) U(n,N,B,\lambda)=O(1) U ( n , N , B , λ ) = O ( 1 )
∥ M ^ n , N , B , λ − M ∗ ∥ = O P ( λ 1 / 2 + ( N h ) − 1 / 2 ) = O P ( N − γ / ( 2 γ + 1 ) ) . (15) \lVert\widehat{M}_{n,N,B,\lambda}-M_*\rVert=O_{\mathbb{P}}\left(\lambda^{1/2}+(Nh)^{-1/2}\right)=O_{\mathbb{P}}\left(N^{-\gamma/(2\gamma+1)}\right). \tag*{(15)} ∥ M n , N , B , λ − M ∗ ∥ = O P ( λ 1/2 + ( N h ) − 1/2 ) = O P ( N − γ / ( 2 γ + 1 ) ) . (15) The rate is minimax optimal since there are N = T B N=TB N = T B (9) . In the classic distributed learning ([42 , 29 , 30 , 5 , 21 , 32 ] (Xu et al. 2019)), all task-specific regression functions coincide, i.e., f 1 = ⋯ = f B = f ∗ f_1=\cdots=f_B=f_* f 1 = ⋯ = f B = f ∗ f ∗ f_* f ∗ f b f_b f b F = δ f ∗ F = \delta_{f_*} F = δ f ∗ V ( n ) = 0 V(\mathbf{n}) = 0 V ( n ) = 0 n \mathbf{n} n n ≍ T \mathbf{n} \asymp T n ≍ T N = T B N = TB N = T B 2 N 2N 2 N (15) is minimax optimal even when accounting for the context data. More generally, V ( n ) ≠ 0 V(\mathbf{n}) \ne0 V ( n ) = 0 f b f_b f b [26 , 27 , 33 , 8 , 7 , 31 ] (Li et al. 2021).
Suppose that the reproducing kernel R ⟨ X ⟩ R^{\langle X\rangle} R ⟨ X ⟩
R ⟨ X ⟩ ( x , x ′ ) = ∑ k ≥ 1 φ k ⟨ X ⟩ ( x ) φ k ⟨ X ⟩ ( x ′ ) ρ k ⟨ X ⟩ , x , x ′ ∈ X , (16) R^{\langle X\rangle}(x,x') = \sum_{k\ge1} \frac{\varphi_k^{\langle X\rangle}(x)\varphi_k^{\langle X\rangle}(x')}{\rho_k^{\langle X\rangle}},\quad x,x'\in\mathcal{X}, \tag*{(16)} R ⟨ X ⟩ ( x , x ′ ) = k ≥ 1 ∑ ρ k ⟨ X ⟩ φ k ⟨ X ⟩ ( x ) φ k ⟨ X ⟩ ( x ′ ) , x , x ′ ∈ X , (16) where φ k ⟨ X ⟩ \varphi_k^{\langle X\rangle} φ k ⟨ X ⟩ L 2 ( μ ) L^2(\mu) L 2 ( μ ) C φ ⟨ X ⟩ = sup k ≥ 1 ∥ φ k ⟨ X ⟩ ∥ ∞ < ∞ C_{\varphi\langle X\rangle}=\sup_{k\ge1}\|\varphi_k^{\langle X\rangle}\|_\infty<\infty C φ ⟨ X ⟩ = sup k ≥ 1 ∥ φ k ⟨ X ⟩ ∥ ∞ < ∞ ρ k ⟨ X ⟩ ≍ k 2 β \rho_k^{\langle X\rangle}\asymp k^{2\beta} ρ k ⟨ X ⟩ ≍ k 2 β β > 1 / 2 \beta>1/2 β > 1/2 V b ( n b ) V_b(n_b) V b ( n b )
Lemma 2. Suppose that ε i b \varepsilon_i^b ε i b R ⟨ X ⟩ R^{\langle X\rangle} R ⟨ X ⟩ (16) , then
V b ( n b ) 2 ≤ C 1 ( 1 + E [ ∥ f b ∥ X 2 ] ) n b − 2 β 2 β + 1 , b = 1 , … , B , V_b(n_b)^2 \le C_1\left(1+\mathbb{E}\left[\|f_b\|_{\mathcal{X}}^2\right]\right)n_b^{-\frac{2\beta}{2\beta+1}},\quad b=1,\ldots,B, V b ( n b ) 2 ≤ C 1 ( 1 + E [ ∥ f b ∥ X 2 ] ) n b − 2 β + 1 2 β , b = 1 , … , B , where C 1 C_1 C 1
By Lemma 2, V ( n ) = O ( n − β / ( 2 β + 1 ) ) V(\mathbf{n})=O(n^{-\beta/(2\beta+1)}) V ( n ) = O ( n − β / ( 2 β + 1 ) ) V ( n ) T 1 / 2 = O ( 1 ) V(\mathbf{n})T^{1/2}=O(1) V ( n ) T 1/2 = O ( 1 ) T = O ( n 2 β / ( 2 β + 1 ) ) T=O(n^{2\beta/(2\beta+1)}) T = O ( n 2 β / ( 2 β + 1 ) ) β → ∞ \beta\to\infty β → ∞ 2 β / ( 2 β + 1 ) → 1 2\beta/(2\beta+1)\to1 2 β / ( 2 β + 1 ) → 1 T T T O ( n ) O(n) O ( n ) 2 N 2N 2 N (15) remains nearly minimax optimal even after accounting for the context observations.
In the end of this section, we establish the asymptotic normality for M ^ n , N , B , λ \widehat{M}_{n,N,B,\lambda} M n , N , B , λ π ^ ∈ P ( Z ) \widehat{\pi}\in\mathcal{P}(\mathcal{Z}) π ∈ P ( Z ) x ∈ X x\in\mathcal{X} x ∈ X ( π ^ , x ) (\widehat{\pi},x) ( π , x ) M ∗ M_* M ∗ ( π ^ , x ) (\widehat{\pi},x) ( π , x ) M ∗ ( π ^ , x ) M_*(\widehat{\pi},x) M ∗ ( π , x ) M ^ n , N , B , λ ( π ^ , x ) \widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x) M n , N , B , λ ( π , x )
Theorem 3. Suppose that Assumptions 1-5 are satisfied and, as n , N , B → ∞ \mathbf{n},N,B\to\infty n , N , B → ∞ (12) holds and
U ( n , N , B , λ ) = O ( 1 ) , (17) U(\mathbf{n},N,B,\lambda)=O(1), \tag*{(17)} U ( n , N , B , λ ) = O ( 1 ) , (17) 1 σ ( π ^ , x ) 2 ∑ k = 1 ∞ h ∣ φ k ( π ^ , x ) ∣ 2 ( 1 + λ ρ k ) 2 → 1 , (18) \frac{1}{\sigma^2_{(\widehat{\pi},x)}}\sum_{k=1}^{\infty}\frac{h|\varphi_k(\widehat{\pi},x)|^2}{(1+\lambda\rho_k)^2}\to1, \tag*{(18)} σ ( π , x ) 2 1 k = 1 ∑ ∞ ( 1 + λ ρ k ) 2 h ∣ φ k ( π , x ) ∣ 2 → 1 , (18) N h ( P λ M ∗ ) ( π ^ , x ) = o ( 1 ) , (19) \sqrt{Nh}(P_\lambda M_*)(\widehat{\pi},x)=o(1), \tag*{(19)} N h ( P λ M ∗ ) ( π , x ) = o ( 1 ) , (19) a h ∑ b = 1 B N b 2 N h ( λ 1 / 2 + ( N h ) − 1 ) = o ( 1 ) , (20) \frac{a_h\sqrt{\sum_{b=1}^{B}N_b^2}}{\sqrt{Nh}}\left(\lambda^{1/2}+(Nh)^{-1}\right)=o(1), \tag*{(20)} N h a h ∑ b = 1 B N b 2 ( λ 1/2 + ( N h ) − 1 ) = o ( 1 ) , (20) ∑ b = 1 B N b 2 V ( n ) 2 = o ( N h ) , (21) \sum_{b=1}^{B}N_b^2V(n)^2=o(Nh), \tag*{(21)} b = 1 ∑ B N b 2 V ( n ) 2 = o ( N h ) , (21) where σ π ^ , x 2 \sigma_{\hat{\pi},x}^{2} σ π ^ , x 2 π ^ , x \hat{\pi},x π ^ , x
N h σ ε σ π ^ , x ( M ^ n , N , B , λ ( π ^ , x ) − M ∗ ( π ^ , x ) ) ⟶ d N ( 0 , 1 ) . \frac{\sqrt{Nh}}{\sigma_{\varepsilon}\sigma_{\hat{\pi},x}}\left(\widehat{M}_{n,N,B,\lambda}(\hat{\pi},x)-M_{*}(\hat{\pi},x)\right)\mathrel{\mathop{\longrightarrow}\limits^{d}}N(0,1). σ ε σ π ^ , x N h ( M n , N , B , λ ( π ^ , x ) − M ∗ ( π ^ , x ) ) ⟶ d N ( 0 , 1 ) . Theorem 3 establishes a pointwise CLT for M ^ n , N , B , λ ( π ^ , x ) \widehat{M}_{n,N,B,\lambda}(\hat{\pi},x) M n , N , B , λ ( π ^ , x ) M ∗ ( π ^ , x ) M_{*}(\hat{\pi},x) M ∗ ( π ^ , x ) π ^ \hat{\pi} π ^ M ^ n , N , B , λ ( π ^ , x ) \widehat{M}_{n,N,B,\lambda}(\hat{\pi},x) M n , N , B , λ ( π ^ , x ) σ π ^ , x 2 \sigma_{\hat{\pi},x}^{2} σ π ^ , x 2 M ^ n , N , B , λ ( π ^ , x ) \widehat{M}_{n,N,B,\lambda}(\hat{\pi},x) M n , N , B , λ ( π ^ , x ) M ∗ = ∑ k = 1 ∞ m k φ k M_{*}=\sum_{k=1}^{\infty}m_{k}\varphi_{k} M ∗ = ∑ k = 1 ∞ m k φ k
P λ M ∗ = ∑ k = 1 ∞ m k P λ φ k = ∑ k = 1 ∞ λ ρ k m k 1 + λ ρ k φ k . P_{\lambda}M_{*}=\sum_{k=1}^{\infty}m_{k}P_{\lambda}\varphi_{k}=\sum_{k=1}^{\infty}\frac{\lambda\rho_{k}m_{k}}{1+\lambda\rho_{k}}\varphi_{k}. P λ M ∗ = k = 1 ∑ ∞ m k P λ φ k = k = 1 ∑ ∞ 1 + λ ρ k λ ρ k m k φ k . If ∑ k = 1 ∞ m k 2 ρ k 1 + ϵ < ∞ \sum_{k=1}^{\infty}m_{k}^{2}\rho_{k}^{1+\epsilon}<\infty ∑ k = 1 ∞ m k 2 ρ k 1 + ϵ < ∞ ϵ ∈ ( 1 / ( 2 γ ) , 1 ) \epsilon\in(1/(2\gamma),1) ϵ ∈ ( 1/ ( 2 γ ) , 1 ) M ∗ M_{*} M ∗ H X \mathcal{H}_{X} H X
∥ P λ M ∗ ∥ 2 = ∑ k = 1 ∞ m k 2 ( λ ρ k ) 2 1 + λ ρ k = ∑ k = 1 ∞ m k 2 ( λ ρ k ) 1 + ϵ ( λ ρ k ) 1 − ϵ 1 + λ ρ k = O ( λ 1 + ϵ ) . \lVert P_{\lambda}M_{*}\rVert^{2}=\sum_{k=1}^{\infty}m_{k}^{2}\frac{(\lambda\rho_{k})^{2}}{1+\lambda\rho_{k}}=\sum_{k=1}^{\infty}m_{k}^{2}(\lambda\rho_{k})^{1+\epsilon}\frac{(\lambda\rho_{k})^{1-\epsilon}}{1+\lambda\rho_{k}}=O(\lambda^{1+\epsilon}). ∥ P λ M ∗ ∥ 2 = k = 1 ∑ ∞ m k 2 1 + λ ρ k ( λ ρ k ) 2 = k = 1 ∑ ∞ m k 2 ( λ ρ k ) 1 + ϵ 1 + λ ρ k ( λ ρ k ) 1 − ϵ = O ( λ 1 + ϵ ) . When h ≍ N − 1 / ( 2 γ + 1 ) h\asymp N^{-1/(2\gamma+1)} h ≍ N − 1/ ( 2 γ + 1 )
N h ∣ ( P λ M ∗ ) ( π ^ , x ) ∣ ≤ N h C ϵ h − 1 / 2 ∥ P λ M ∗ ∥ = O ( N N − γ ( 1 + ϵ ) / ( 2 γ + 1 ) ) = o ( 1 ) , \sqrt{Nh}\lvert(P_{\lambda}M_{*})(\hat{\pi},x)\rvert\leq\sqrt{Nh}C_{\epsilon}h^{-1/2}\lVert P_{\lambda}M_{*}\rVert=O\left(\sqrt{N}N^{-\gamma(1+\epsilon)/(2\gamma+1)}\right)=o(1), N h ∣( P λ M ∗ ) ( π ^ , x )∣ ≤ N h C ϵ h − 1/2 ∥ P λ M ∗ ∥ = O ( N N − γ ( 1 + ϵ ) / ( 2 γ + 1 ) ) = o ( 1 ) , which verifies Condition (19). It can be directly checked that Condition (20) holds when γ > ( 3 + 5 ) / 4 \gamma>(3+\sqrt{5})/4 γ > ( 3 + 5 ) /4 T = o ( B ( 4 γ 2 − 6 γ + 1 ) / ( 8 γ − 1 ) ) T=o\left(B^{(4\gamma^{2}-6\gamma+1)/(8\gamma-1)}\right) T = o ( B ( 4 γ 2 − 6 γ + 1 ) / ( 8 γ − 1 ) ) γ \gamma γ M \mathcal{M} M T 2 ( γ + 1 ) B = o ( n 2 β ( 2 γ + 1 ) / ( 2 β + 1 ) ) T^{2(\gamma+1)}B=o\left(n^{2\beta(2\gamma+1)/(2\beta+1)}\right) T 2 ( γ + 1 ) B = o ( n 2 β ( 2 γ + 1 ) / ( 2 β + 1 ) )
In practice, we suggest to estimate σ π ^ , x 2 \sigma_{\hat{\pi},x}^{2} σ π ^ , x 2
σ ^ ( π ^ , x ) 2 = ∑ k = 1 ∞ h ∣ φ k ( π ^ , x ) ∣ 2 ( 1 + λ ρ k ) 2 . \widehat{\sigma}_{(\hat{\pi},x)}^{2}=\sum_{k=1}^{\infty}\frac{h\lvert\varphi_{k}(\hat{\pi},x)\rvert^{2}}{(1+\lambda\rho_{k})^{2}}. σ ( π ^ , x ) 2 = k = 1 ∑ ∞ ( 1 + λ ρ k ) 2 h ∣ φ k ( π ^ , x ) ∣ 2 . Suppose D n n e w = { Z 1 n e w , … , Z n n e w } D_{n}^{\mathrm{new}}=\{Z_{1}^{\mathrm{new},\ldots,Z_{n}^{\mathrm{new}}\}} D n new = { Z 1 new , … , Z n new } Z i n e w = ( X i n e w , Y i n e w ) Z_{i}^{\mathrm{new}}=(X_{i}^{\mathrm{new}},Y_{i}^{\mathrm{new}}) Z i new = ( X i new , Y i new )
Y i n e w = f n e w ( X i n e w ) + ϵ i n e w , i = 1 , … , n . Y_{i}^{\mathrm{new}}=f_{\mathrm{new}}(X_{i}^{\mathrm{new}})+\epsilon_{i}^{\mathrm{new}},\quad i=1,\ldots,n. Y i new = f new ( X i new ) + ϵ i new , i = 1 , … , n . Let π ^ n n e w \hat{\pi}_{n}^{\mathrm{new}} π ^ n new D n n e w D_{n}^{\mathrm{new}} D n new M ∗ ( π ^ , x ) = E [ Y n e w ∣ π ^ n n e w = π ^ , X n e w = x ] M_{*}(\hat{\pi},x)=\mathbb{E}[Y^{\mathrm{new}}\mid\hat{\pi}_{n}^{\mathrm{new}}=\hat{\pi},X^{\mathrm{new}}=x] M ∗ ( π ^ , x ) = E [ Y new ∣ π ^ n new = π ^ , X new = x ] 100 % ( 1 − α ) 100\%(1-\alpha) 100% ( 1 − α ) M ∗ ( π ^ , x ) M_{*}(\hat{\pi},x) M ∗ ( π ^ , x )
M ^ n , N , B , λ ( π ^ , x ) ± z 1 − α / 2 σ ^ ( π ^ , x ) σ ^ ε N h , \widehat{M}_{n,N,B,\lambda}(\hat{\pi},x)\pm z_{1-\alpha/2}\frac{\widehat{\sigma}_{(\hat{\pi},x)}\widehat{\sigma}_{\varepsilon}}{\sqrt{Nh}}, M n , N , B , λ ( π ^ , x ) ± z 1 − α /2 N h σ ( π ^ , x ) σ ε , where z 1 − α / 2 z_{1-\alpha/2} z 1 − α /2 ( 1 − α / 2 ) (1-\alpha/2) ( 1 − α /2 )
σ ^ ε 2 = N − 1 ∑ b = 1 B ∑ i = 1 N b ( Y n b + i b − M ^ n , N , B , λ ( π ^ n b b , X n b + i b ) ) 2 \widehat{\sigma}_{\varepsilon}^{2}=N^{-1}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left(Y_{n_b+i}^{b}-\widehat{M}_{n,N,B,\lambda}\left(\widehat{\pi}_{n_b}^{b},X_{n_b+i}^{b}\right)\right)^{2} σ ε 2 = N − 1 b = 1 ∑ B i = 1 ∑ N b ( Y n b + i b − M n , N , B , λ ( π n b b , X n b + i b ) ) 2 is a consistent estimator of σ ε 2 \sigma_{\varepsilon}^{2} σ ε 2 π ^ n n e w = π ^ \widehat{\pi}_{n}^{\mathrm{new}}=\widehat{\pi} π n new = π X n e w = x X^{\mathrm{new}}=x X new = x
Y n e w − M ^ n , N , B , λ ( π ^ , x ) = f n e w ( x ) − M ^ n , N , B , λ ( π ^ , x ) + ε n e w = f n e w ( x ) − M ∗ ( π ^ , x ) + M ∗ ( π ^ , x ) − M ^ n , N , B , λ ( π ^ , x ) + ε n e w = O P ( n − β / ( 2 β + 1 ) ) + M ∗ ( π ^ , x ) − M ^ n , N , B , λ ( π ^ , x ) + ε n e w , \begin{aligned}
Y^{\mathrm{new}}-\widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x)
&=f_{\mathrm{new}}(x)-\widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x)+\varepsilon^{\mathrm{new}}\\
&=f_{\mathrm{new}}(x)-M_{*}(\widehat{\pi},x)+M_{*}(\widehat{\pi},x)-\widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x)+\varepsilon^{\mathrm{new}}\\
&=O_{\mathbb{P}}\left(n^{-\beta/(2\beta+1)}\right)+M_{*}(\widehat{\pi},x)-\widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x)+\varepsilon^{\mathrm{new}},
\end{aligned} Y new − M n , N , B , λ ( π , x ) = f new ( x ) − M n , N , B , λ ( π , x ) + ε new = f new ( x ) − M ∗ ( π , x ) + M ∗ ( π , x ) − M n , N , B , λ ( π , x ) + ε new = O P ( n − β / ( 2 β + 1 ) ) + M ∗ ( π , x ) − M n , N , B , λ ( π , x ) + ε new , where the O P ( n − β / ( 2 β + 1 ) ) O_{\mathbb{P}}\left(n^{-\beta/(2\beta+1)}\right) O P ( n − β / ( 2 β + 1 ) ) ε n e w \varepsilon^{\mathrm{new}} ε new M ^ n , N , B , λ ( π ^ , x ) \widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x) M n , N , B , λ ( π , x ) ε n e w ∼ N ( 0 , σ ε 2 ) \varepsilon^{\mathrm{new}}\sim N(0,\sigma_{\varepsilon}^{2}) ε new ∼ N ( 0 , σ ε 2 ) Theorem 3 ,
M ∗ ( π ^ , x ) − M ^ n , N , B , λ ( π ^ , x ) + ε n e w ∼ a . d . N ( 0 , σ ε 2 σ π ^ , x 2 N h + σ ε 2 ) , M_{*}(\widehat{\pi},x)-\widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x)+\varepsilon^{\mathrm{new}}\mathrel{\overset{\mathrm{a.d.}}{\sim}}N\left(0,\frac{\sigma_{\varepsilon}^{2}\sigma_{\widehat{\pi},x}^{2}}{Nh}+\sigma_{\varepsilon}^{2}\right), M ∗ ( π , x ) − M n , N , B , λ ( π , x ) + ε new ∼ a.d. N ( 0 , N h σ ε 2 σ π , x 2 + σ ε 2 ) , where “a.d. ∼ \sim ∼ n − β / ( 2 β + 1 ) = o ( ( N h ) − 1 / 2 ) n^{-\beta/(2\beta+1)}=o((Nh)^{-1/2}) n − β / ( 2 β + 1 ) = o (( N h ) − 1/2 ) O P ( n − β / ( 2 β + 1 ) ) O_{\mathbb{P}}\left(n^{-\beta/(2\beta+1)}\right) O P ( n − β / ( 2 β + 1 ) )
Y n e w − M ^ n , N , B , λ ( π ^ , x ) ∼ a . d . N ( 0 , σ ε 2 σ π ^ , x 2 N h + σ ε 2 ) . Y^{\mathrm{new}}-\widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x)\mathrel{\overset{\mathrm{a.d.}}{\sim}}N\left(0,\frac{\sigma_{\varepsilon}^{2}\sigma_{\widehat{\pi},x}^{2}}{Nh}+\sigma_{\varepsilon}^{2}\right). Y new − M n , N , B , λ ( π , x ) ∼ a.d. N ( 0 , N h σ ε 2 σ π , x 2 + σ ε 2 ) . Therefore, we suggest the following 100 % ( 1 − α ) 100\%(1-\alpha) 100% ( 1 − α ) Y n e w Y_{\mathrm{new}} Y new
M ^ n , N , B , λ ( π ^ , x ) ± z 1 − α / 2 σ ^ π ^ , x 2 σ ^ ε 2 N h + σ ^ ε 2 . \widehat{M}_{n,N,B,\lambda}(\widehat{\pi},x)\pm z_{1-\alpha/2}\sqrt{\frac{\widehat{\sigma}_{\widehat{\pi},x}^{2}\widehat{\sigma}_{\varepsilon}^{2}}{Nh}+\widehat{\sigma}_{\varepsilon}^{2}}. M n , N , B , λ ( π , x ) ± z 1 − α /2 N h σ π , x 2 σ ε 2 + σ ε 2 . Example: Latent Gaussian Process Model
In this section, we consider a latent Gaussian process model which will validate the proposed MI-RKHS framework. Specifically, we will describe an explicit construction of the RKHS H Π \mathcal{H}_{\Pi} H Π R ⟨ Π ⟩ R^{\langle\Pi\rangle} R ⟨ Π ⟩ H ∗ H_{*} H ∗ R ⟨ Π ⟩ R^{\langle\Pi\rangle} R ⟨ Π ⟩ R ⟨ X ⟩ R^{\langle X\rangle} R ⟨ X ⟩ R R R (6) , hence, such explicit construction is useful for the application of the proposed MI-KRR (9) . Suppose that H X \mathcal{H}_{X} H X C d C_{d} C d f 1 , f 2 ∈ H X f_{1},f_{2}\in\mathcal{H}_{X} f 1 , f 2 ∈ H X
∥ f 1 f 2 ∥ H X ≤ C d ∥ f 1 ∥ H X ∥ f 2 ∥ H X . (22) \lVert f_{1}f_{2}\rVert_{\mathcal{H}_{X}}\leq C_{d}\lVert f_{1}\rVert_{\mathcal{H}_{X}}\lVert f_{2}\rVert_{\mathcal{H}_{X}}. \tag*{(22)} ∥ f 1 f 2 ∥ H X ≤ C d ∥ f 1 ∥ H X ∥ f 2 ∥ H X . (22) This holds, for example, when H X \mathcal{H}_{X} H X d / 2 d/2 d /2 X ⊂ R d \mathcal{X}\subset\mathbb{R}^{d} X ⊂ R d [1 ] ). Suppose that the latent random function f f f f = f ∗ + W f=f_{*}+W f = f ∗ + W f ∗ ∈ H X f_{*}\in\mathcal{H}_{X} f ∗ ∈ H X W W W C ∗ C_{*} C ∗ E [ ∥ W ∥ H X 2 ] < ∞ \mathbb{E}[\lVert W\rVert_{\mathcal{H}_{X}}^{2}]<\infty E [∥ W ∥ H X 2 ] < ∞ f ∈ H X f\in\mathcal{H}_{X} f ∈ H X x ∈ X x\in\mathcal{X} x ∈ X
∥ C ∗ ( ⋅ , x ) ∥ H X = ∥ E [ W ( ⋅ ) W ( x ) ] ∥ H X ≤ E [ ∣ W ( x ) ∣ ⋅ ∥ W ∥ H X ] ≤ E [ ∥ W ∥ H X 2 R ⟨ X ⟩ ( x , x ) 1 / 2 ] ≤ C X 1 / 2 E [ ∥ W ∥ H X 2 ] < ∞ . (23) \begin{aligned}
\lVert C_{*}(\cdot,x)\rVert_{\mathcal{H}_{X}}
&=\lVert\mathbb{E}[W(\cdot)W(x)]\rVert_{\mathcal{H}_{X}}\leq\mathbb{E}[|W(x)|\cdot\lVert W\rVert_{\mathcal{H}_{X}}]\\
&\leq\mathbb{E}[\lVert W\rVert_{\mathcal{H}_{X}}^{2}R^{\langle X\rangle}(x,x)^{1/2}]
\leq C_{X}^{1/2}\mathbb{E}[\lVert W\rVert_{\mathcal{H}_{X}}^{2}]<\infty. \tag*{(23)}
\end{aligned} ∥ C ∗ ( ⋅ , x ) ∥ H X = ∥ E [ W ( ⋅ ) W ( x )] ∥ H X ≤ E [ ∣ W ( x ) ∣ ⋅ ∥ W ∥ H X ] ≤ E [∥ W ∥ H X 2 R ⟨ X ⟩ ( x , x ) 1/2 ] ≤ C X 1/2 E [∥ W ∥ H X 2 ] < ∞. (23) therefore, C ∗ ( ⋅ , x ) C_*(\cdot,x) C ∗ ( ⋅ , x ) H X \mathcal{H}_X H X Γ ∗ = E [ W ⊗ W ] \Gamma_*=\mathbb{E}[W\otimes W] Γ ∗ = E [ W ⊗ W ] trace ( Γ ∗ ) = E [ ∥ W ∥ H X 2 ] < ∞ \operatorname{trace}(\Gamma_*)=\mathbb{E}[\lVert W\rVert_{\mathcal{H}_X}^2]<\infty trace ( Γ ∗ ) = E [∥ W ∥ H X 2 ] < ∞ Γ ∗ \Gamma_* Γ ∗ H X \mathcal{H}_X H X H X \mathcal{H}_X H X
C ∗ ( ⋅ , x ) = E [ W ( ⋅ ) W ( x ) ] = E [ ⟨ W , R x ⟨ X ⟩ ⟩ H X ⟨ W , R x ⟨ X ⟩ ⟩ H X ] = ⟨ Γ ∗ R x ⟨ X ⟩ , R x ⟨ X ⟩ ⟩ H X = ( Γ ∗ R x ⟨ X ⟩ ) ( ⋅ ) . (24) C_*(\cdot,x)=\mathbb{E}[W(\cdot)W(x)]=\mathbb{E}[\langle W,R_x^{\langle X\rangle}\rangle_{\mathcal{H}_X}\langle W,R_x^{\langle X\rangle}\rangle_{\mathcal{H}_X}]=\langle\Gamma_*R_x^{\langle X\rangle},R_x^{\langle X\rangle}\rangle_{\mathcal{H}_X}=(\Gamma_*R_x^{\langle X\rangle})(\cdot). \tag*{(24)} C ∗ ( ⋅ , x ) = E [ W ( ⋅ ) W ( x )] = E [⟨ W , R x ⟨ X ⟩ ⟩ H X ⟨ W , R x ⟨ X ⟩ ⟩ H X ] = ⟨ Γ ∗ R x ⟨ X ⟩ , R x ⟨ X ⟩ ⟩ H X = ( Γ ∗ R x ⟨ X ⟩ ) ( ⋅ ) . (24) For π ∈ P ( Z ) \pi\in\mathcal{P}(\mathcal{Z}) π ∈ P ( Z )
G π = ∫ Z ( y − f ∗ ( x ) ) Γ ∗ 1 / 2 R x ⟨ X ⟩ d π ( x , y ) , J π = ∫ X ( Γ ∗ 1 / 2 R x ⟨ X ⟩ ) ⊗ ( Γ ∗ 1 / 2 R x ⟨ X ⟩ ) d π X ( x ) , g ∈ H X , (25) G_\pi=\int_{\mathcal{Z}}(y-f_*(x))\Gamma_*^{1/2}R_x^{\langle X\rangle}d\pi(x,y),\qquad J_\pi=\int_{\mathcal{X}}(\Gamma_*^{1/2}R_x^{\langle X\rangle})\otimes(\Gamma_*^{1/2}R_x^{\langle X\rangle})d\pi^X(x),\quad g\in\mathcal{H}_X, \tag*{(25)} G π = ∫ Z ( y − f ∗ ( x )) Γ ∗ 1/2 R x ⟨ X ⟩ d π ( x , y ) , J π = ∫ X ( Γ ∗ 1/2 R x ⟨ X ⟩ ) ⊗ ( Γ ∗ 1/2 R x ⟨ X ⟩ ) d π X ( x ) , g ∈ H X , (25) where π X \pi^X π X X X X π \pi π
∥ G π ∥ H X 2 ≤ ∫ Z ∣ y − f ∗ ( x ) ∣ 2 d π ( x , y ) ∥ Γ ∗ 1 / 2 R x ⟨ X ⟩ ∥ H X 2 ≤ 2 ( ∫ Z y 2 d π ( x , y ) + ∥ f ∗ ∥ 2 2 ) × C X ∥ Γ ∗ ∥ o p < ∞ , \begin{aligned}
\lVert G_\pi\rVert_{\mathcal{H}_X}^2
&\le\int_{\mathcal{Z}}|y-f_*(x)|^2d\pi(x,y)\lVert\Gamma_*^{1/2}R_x^{\langle X\rangle}\rVert_{\mathcal{H}_X}^2\\
&\le2\left(\int_{\mathcal{Z}}y^2d\pi(x,y)+\lVert f_*\rVert_2^2\right)\times C_X\lVert\Gamma_*\rVert_{\mathrm{op}}<\infty,
\end{aligned} ∥ G π ∥ H X 2 ≤ ∫ Z ∣ y − f ∗ ( x ) ∣ 2 d π ( x , y ) ∥ Γ ∗ 1/2 R x ⟨ X ⟩ ∥ H X 2 ≤ 2 ( ∫ Z y 2 d π ( x , y ) + ∥ f ∗ ∥ 2 2 ) × C X ∥ Γ ∗ ∥ op < ∞ , and for any g ∈ H X g\in\mathcal{H}_X g ∈ H X
∥ J π g ∥ H X ≤ ∫ X ∥ ( Γ ∗ 1 / 2 R x ⟨ X ⟩ ) ⊗ ( Γ ∗ 1 / 2 R x ⟨ X ⟩ ) g ∥ H X d π X ( x ) ≤ ∫ X ∥ Γ ∗ 1 / 2 R x ⟨ X ⟩ ∥ H X 2 d π X ( x ) ∥ g ∥ H X ≤ C X ∥ Γ ∗ ∥ o p ∥ g ∥ H X . \begin{aligned}
\lVert J_\pi g\rVert_{\mathcal{H}_X}
&\le\int_{\mathcal{X}}\lVert(\Gamma_*^{1/2}R_x^{\langle X\rangle})\otimes(\Gamma_*^{1/2}R_x^{\langle X\rangle})g\rVert_{\mathcal{H}_X}d\pi^X(x)\\
&\le\int_{\mathcal{X}}\lVert\Gamma_*^{1/2}R_x^{\langle X\rangle}\rVert_{\mathcal{H}_X}^2d\pi^X(x)\lVert g\rVert_{\mathcal{H}_X}\le C_X\lVert\Gamma_*\rVert_{\mathrm{op}}\lVert g\rVert_{\mathcal{H}_X}.
\end{aligned} ∥ J π g ∥ H X ≤ ∫ X ∥( Γ ∗ 1/2 R x ⟨ X ⟩ ) ⊗ ( Γ ∗ 1/2 R x ⟨ X ⟩ ) g ∥ H X d π X ( x ) ≤ ∫ X ∥ Γ ∗ 1/2 R x ⟨ X ⟩ ∥ H X 2 d π X ( x ) ∥ g ∥ H X ≤ C X ∥ Γ ∗ ∥ op ∥ g ∥ H X . Consequently, G π ∈ H X G_\pi\in\mathcal{H}_X G π ∈ H X J π ∈ L ( H X ) J_\pi\in\mathcal{L}(\mathcal{H}_X) J π ∈ L ( H X ) ∥ J π ∥ o p ≤ C X ∥ Γ ∗ ∥ o p \lVert J_\pi\rVert_{\mathrm{op}}\le C_X\lVert\Gamma_*\rVert_{\mathrm{op}} ∥ J π ∥ op ≤ C X ∥ Γ ∗ ∥ op J π J_\pi J π T π = ( J π + σ ε 2 Id ) − 1 T_\pi=(J_\pi+\sigma_\varepsilon^2\operatorname{Id})^{-1} T π = ( J π + σ ε 2 Id ) − 1
H ∗ ( π ) = f ∗ + Γ ∗ 1 / 2 T π G π , π ∈ P ( Z ) . H_*(\pi)=f_*+\Gamma_*^{1/2}T_\pi G_\pi,\quad\pi\in\mathcal{P}(\mathcal{Z}). H ∗ ( π ) = f ∗ + Γ ∗ 1/2 T π G π , π ∈ P ( Z ) . Clearly, H ∗ H_* H ∗ P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) H X \mathcal{H}_X H X R ⟨ Z ⟩ ( z , z ′ ) = ( 1 + y y ′ ) R ⟨ X ⟩ ( x , x ′ ) R^{\langle Z\rangle}(z,z')=(1+yy')R^{\langle X\rangle}(x,x') R ⟨ Z ⟩ ( z , z ′ ) = ( 1 + y y ′ ) R ⟨ X ⟩ ( x , x ′ )
R ⟨ Π , X ⟩ ( π , π ′ ) = ∫ X × X R ⟨ X ⟩ ( x , x ′ ) d π X ( x ) d π ′ X ( x ′ ) , R^{\langle\Pi,X\rangle}(\pi,\pi')=\int_{\mathcal{X}\times\mathcal{X}}R^{\langle X\rangle}(x,x')d\pi^X(x)d\pi^{\prime X}(x'), R ⟨ Π , X ⟩ ( π , π ′ ) = ∫ X × X R ⟨ X ⟩ ( x , x ′ ) d π X ( x ) d π ′ X ( x ′ ) , R ⟨ Π , Z ⟩ ( π , π ′ ) = ∫ Z × Z R ⟨ Z ⟩ ( z , z ′ ) d π ( z ) d π ′ ( z ′ ) . (26) R^{\langle\Pi,Z\rangle}(\pi,\pi')=\int_{\mathcal{Z}\times\mathcal{Z}}R^{\langle Z\rangle}(z,z')d\pi(z)d\pi'(z'). \tag*{(26)} R ⟨ Π , Z ⟩ ( π , π ′ ) = ∫ Z × Z R ⟨ Z ⟩ ( z , z ′ ) d π ( z ) d π ′ ( z ′ ) . (26) Define
R ⟨ Π ⟩ ( π , π ′ ) = ( 1 + R ⟨ Π , Z ⟩ ( π , π ′ ) 1 − c ε R ⟨ Π , X ⟩ ( π , π ′ ) ) ⋅ id H X , π , π ′ ∈ P ( Z ) , (27) R^{\langle\Pi\rangle}(\pi,\pi')=\left(1+\frac{R^{\langle\Pi,Z\rangle}(\pi,\pi')}{1-c_\varepsilon R^{\langle\Pi,X\rangle}(\pi,\pi')}\right)\cdot\operatorname{id}_{\mathcal{H}_X},\quad\pi,\pi'\in\mathcal{P}(\mathcal{Z}), \tag*{(27)} R ⟨ Π ⟩ ( π , π ′ ) = ( 1 + 1 − c ε R ⟨ Π , X ⟩ ( π , π ′ ) R ⟨ Π , Z ⟩ ( π , π ′ ) ) ⋅ id H X , π , π ′ ∈ P ( Z ) , (27) where 0 < c ε < C X − 1 0<c_\varepsilon<C_X^{-1} 0 < c ε < C X − 1 ∣ R ⟨ Π , X ⟩ ( π , π ′ ) ∣ ≤ C X \lvert R^{\langle\Pi,X\rangle}(\pi,\pi')\rvert\le C_X ∣ R ⟨ Π , X ⟩ ( π , π ′ )∣ ≤ C X π , π ′ ∈ P ( Z ) \pi,\pi'\in\mathcal{P}(\mathcal{Z}) π , π ′ ∈ P ( Z ) c ε c_\varepsilon c ε R ⟨ Π ⟩ R^{\langle\Pi\rangle} R ⟨ Π ⟩ H Π \mathcal{H}_\Pi H Π P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) H X \mathcal{H}_X H X R ⟨ Π ⟩ R^{\langle\Pi\rangle} R ⟨ Π ⟩ M ∗ M_* M ∗ H ∗ H_* H ∗ H ∗ H_* H ∗ H Π \mathcal{H}_\Pi H Π
Lemma 3. Under Assumption 1, (22) and the following series condition
∑ k = 1 ∞ ( E [ ⟨ W , φ k ⟨ X ⟩ ⟩ H X 2 ] ρ k ⟨ X ⟩ ) 1 / 2 < ∞ , (28) \sum_{k=1}^{\infty}\left(\frac{\mathbb{E}[\langle W,\varphi_k^{\langle X\rangle}\rangle_{\mathcal{H}_X}^2]}{\rho_k^{\langle X\rangle}}\right)^{1/2}<\infty, \tag*{(28)} k = 1 ∑ ∞ ( ρ k ⟨ X ⟩ E [⟨ W , φ k ⟨ X ⟩ ⟩ H X 2 ] ) 1/2 < ∞ , (28) it holds that H ∗ ∈ H Π H_* \in\mathcal{H}_{\Pi} H ∗ ∈ H Π M ∗ ( π n , ⋅ ) = H ∗ ( π n ) ( ⋅ ) M_*(\pi_n,\cdot)=H_*(\pi_n)(\cdot) M ∗ ( π n , ⋅ ) = H ∗ ( π n ) ( ⋅ )
Note that (28) is slightly stronger than E [ ∥ W ∥ H X 2 ] < ∞ \mathbb{E}[\lVert W\rVert_{\mathcal{H}_X}^{2}]<\infty E [∥ W ∥ H X 2 ] < ∞
∑ k = 1 ∞ E [ ⟨ W , φ k ⟨ X ⟩ ⟩ H X 2 ] / ρ k ⟨ X ⟩ < ∞ . \sum_{k=1}^{\infty}\mathbb{E}[\langle W,\varphi_k^{\langle X\rangle}\rangle_{\mathcal{H}_X}^{2}]/\rho_k^{\langle X\rangle}<\infty. k = 1 ∑ ∞ E [⟨ W , φ k ⟨ X ⟩ ⟩ H X 2 ] / ρ k ⟨ X ⟩ < ∞. The additional strength of (28) ensures that the covariance function C ∗ C_* C ∗ H X \mathcal{H}_X H X H ∗ H_* H ∗ H Π \mathcal{H}_{\Pi} H Π R ⟨ Π ⟩ R^{\langle\Pi\rangle} R ⟨ Π ⟩ (27) together with (6) leads to the construction of R R R ( π , x ) , ( π ′ , x ′ ) ∈ P ( Z ) × X (\pi,x),(\pi',x')\in\mathcal{P}(\mathcal{Z})\times\mathcal{X} ( π , x ) , ( π ′ , x ′ ) ∈ P ( Z ) × X
R ( ( π , x ) , ( π ′ , x ′ ) ) = ⟨ R π ⟨ Π ⟩ R x ⟨ X ⟩ , R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ ⟩ H Π = ⟨ ( R π ⟨ Π ⟩ R x ⟨ X ⟩ ) ( π ′ ) , R x ′ ⟨ X ⟩ ⟩ H X (by Property(b)) = ⟨ R ⟨ Π ⟩ ( π , π ′ ) R x ⟨ X ⟩ , R x ′ ⟨ X ⟩ ⟩ H X = ( 1 + R ⟨ Π , Z ⟩ ( π , π ′ ) 1 − c E R ⟨ Π , X ⟩ ( π , π ′ ) ) ⟨ R x ⟨ X ⟩ , R x ′ ⟨ X ⟩ ⟩ H X = ( 1 + R ⟨ Π , Z ⟩ ( π , π ′ ) 1 − c E R ⟨ Π , X ⟩ ( π , π ′ ) ) R ⟨ X ⟩ ( x , x ′ ) . \begin{aligned}
R((\pi,x),(\pi',x'))&=\langle R_{\pi}^{\langle\Pi\rangle}R_x^{\langle X\rangle},R_{\pi'}^{\langle\Pi\rangle}R_{x'}^{\langle X\rangle}\rangle_{\mathcal{H}_{\Pi}}\\
&=\langle(R_{\pi}^{\langle\Pi\rangle}R_x^{\langle X\rangle})(\pi'),R_{x'}^{\langle X\rangle}\rangle_{\mathcal{H}_X}\quad\text{(by Property(b))}\\
&=\langle R^{\langle\Pi\rangle}(\pi,\pi')R_x^{\langle X\rangle},R_{x'}^{\langle X\rangle}\rangle_{\mathcal{H}_X}\\
&=\left(1+\frac{R^{\langle\Pi,Z\rangle}(\pi,\pi')}{1-c_{\mathcal{E}}R^{\langle\Pi,X\rangle}(\pi,\pi')}\right)\langle R_x^{\langle X\rangle},R_{x'}^{\langle X\rangle}\rangle_{\mathcal{H}_X}\\
&=\left(1+\frac{R^{\langle\Pi,Z\rangle}(\pi,\pi')}{1-c_{\mathcal{E}}R^{\langle\Pi,X\rangle}(\pi,\pi')}\right)R^{\langle X\rangle}(x,x').
\end{aligned} R (( π , x ) , ( π ′ , x ′ )) = ⟨ R π ⟨ Π ⟩ R x ⟨ X ⟩ , R π ′ ⟨ Π ⟩ R x ′ ⟨ X ⟩ ⟩ H Π = ⟨( R π ⟨ Π ⟩ R x ⟨ X ⟩ ) ( π ′ ) , R x ′ ⟨ X ⟩ ⟩ H X (by Property(b)) = ⟨ R ⟨ Π ⟩ ( π , π ′ ) R x ⟨ X ⟩ , R x ′ ⟨ X ⟩ ⟩ H X = ( 1 + 1 − c E R ⟨ Π , X ⟩ ( π , π ′ ) R ⟨ Π , Z ⟩ ( π , π ′ ) ) ⟨ R x ⟨ X ⟩ , R x ′ ⟨ X ⟩ ⟩ H X = ( 1 + 1 − c E R ⟨ Π , X ⟩ ( π , π ′ ) R ⟨ Π , Z ⟩ ( π , π ′ ) ) R ⟨ X ⟩ ( x , x ′ ) . where the second last equality holds due to
R ⟨ Π ⟩ ( π , π ′ ) R x ⟨ X ⟩ = ( 1 + R ⟨ Π , Z ⟩ ( π , π ′ ) 1 − c E R ⟨ Π , X ⟩ ( π , π ′ ) ) R x ⟨ X ⟩ . R^{\langle\Pi\rangle}(\pi,\pi')R_x^{\langle X\rangle}=\left(1+\frac{R^{\langle\Pi,Z\rangle}(\pi,\pi')}{1-c_{\mathcal{E}}R^{\langle\Pi,X\rangle}(\pi,\pi')}\right)R_x^{\langle X\rangle}. R ⟨ Π ⟩ ( π , π ′ ) R x ⟨ X ⟩ = ( 1 + 1 − c E R ⟨ Π , X ⟩ ( π , π ′ ) R ⟨ Π , Z ⟩ ( π , π ′ ) ) R x ⟨ X ⟩ . Appendix: Proofs
Proof of Lemma 1
Let
H ( z 1 , … , z n , x ) = E [ Y n + 1 ∣ D n = ( z 1 , … , z n ) , X n + 1 = x ] , z 1 , … , z n ∈ Z , x ∈ X . H(z_1,\ldots,z_n,x)=\mathbb{E}[Y_{n+1}\mid D_n=(z_1,\ldots,z_n),X_{n+1}=x],\quad z_1,\ldots,z_n\in\mathcal{Z},x\in\mathcal{X}. H ( z 1 , … , z n , x ) = E [ Y n + 1 ∣ D n = ( z 1 , … , z n ) , X n + 1 = x ] , z 1 , … , z n ∈ Z , x ∈ X . Since Z 1 , … , Z n + 1 Z_1,\ldots,Z_{n+1} Z 1 , … , Z n + 1
H ~ ( { z 1 , … , z n } , x ) = H ( z 1 , … , z n , x ) , \widetilde{H}(\{z_1,\ldots,z_n\},x)=H(z_1,\ldots,z_n,x), H ({ z 1 , … , z n } , x ) = H ( z 1 , … , z n , x ) , which is a measurable mapping that maps an unordered set { z 1 , … , z n } \{z_1,\ldots,z_n\} { z 1 , … , z n } H ( z 1 , … , z n , x ) H(z_1,\ldots,z_n,x) H ( z 1 , … , z n , x ) { z 1 , … , z n } \{z_1,\ldots,z_n\} { z 1 , … , z n } { z 1 ′ , … , z n ′ } \{z'_1,\ldots,z'_n\} { z 1 ′ , … , z n ′ } τ \tau τ 1 , … , n 1,\ldots,n 1 , … , n z i ′ = z τ ( i ) z'_i=z_{\tau(i)} z i ′ = z τ ( i ) i = 1 , … , n i=1,\ldots,n i = 1 , … , n
H ~ ( { z 1 ′ , … , z n ′ } , x ) = H ( z 1 ′ , … , z n ′ , x ) = E [ Y n + 1 ∣ D n = ( z 1 ′ , … , z n ′ ) , X n + 1 = x ] = E [ Y n + 1 ∣ D n = ( z τ ( 1 ) , … , z τ ( n ) ) , X n + 1 = x ] = E [ Y n + 1 ∣ D n = ( z 1 , … , z n ) , X n + 1 = x ] = H ( z 1 , … , z n , x ) = H ~ ( { z 1 , … , z n } , x ) . \begin{aligned}
\widetilde{H}(\{z'_1,\ldots,z'_n\},x)&=H(z'_1,\ldots,z'_n,x)=\mathbb{E}[Y_{n+1}\mid D_n=(z'_1,\ldots,z'_n),X_{n+1}=x]\\
&=\mathbb{E}[Y_{n+1}\mid D_n=(z_{\tau(1)},\ldots,z_{\tau(n)}),X_{n+1}=x]\\
&=\mathbb{E}[Y_{n+1}\mid D_n=(z_1,\ldots,z_n),X_{n+1}=x]=H(z_1,\ldots,z_n,x)\\
&=\widetilde{H}(\{z_1,\ldots,z_n\},x).
\end{aligned} H ({ z 1 ′ , … , z n ′ } , x ) = H ( z 1 ′ , … , z n ′ , x ) = E [ Y n + 1 ∣ D n = ( z 1 ′ , … , z n ′ ) , X n + 1 = x ] = E [ Y n + 1 ∣ D n = ( z τ ( 1 ) , … , z τ ( n ) ) , X n + 1 = x ] = E [ Y n + 1 ∣ D n = ( z 1 , … , z n ) , X n + 1 = x ] = H ( z 1 , … , z n , x ) = H ({ z 1 , … , z n } , x ) . Let
s ( n − 1 ∑ i = 1 n δ z i ) = { z 1 , … , z n } . s\left(n^{-1}\sum_{i=1}^{n}\delta_{z_i}\right)=\{z_1,\ldots,z_n\}. s ( n − 1 i = 1 ∑ n δ z i ) = { z 1 , … , z n } . which maps the empirical measure to a unique set of the data points. Then we have
H ( z 1 , … , z n , x ) = H ~ ( ( s ( n − 1 ∑ i = 1 n δ z i ) , x ) ) H(z_1,\ldots,z_n,x)=\widetilde{H}\left(\left(s\left(n^{-1}\sum_{i=1}^{n}\delta_{z_i}\right),x\right)\right) H ( z 1 , … , z n , x ) = H ( ( s ( n − 1 i = 1 ∑ n δ z i ) , x ) ) which leads to
E [ Y n + 1 ∣ D n , X n + 1 = x ] = H ~ ( s ( π ~ n ) , x ) , \mathbb{E}[Y_{n+1}\mid D_n,X_{n+1}=x]=\widetilde{H}(s(\widetilde{\pi}_n),x), E [ Y n + 1 ∣ D n , X n + 1 = x ] = H ( s ( π n ) , x ) , which means E [ Y n + 1 ∣ D n , X n + 1 = x ] \mathbb{E}[Y_{n+1}\mid D_n,X_{n+1}=x] E [ Y n + 1 ∣ D n , X n + 1 = x ] ( π ~ n , x ) (\widetilde{\pi}_n,x) ( π n , x ) σ \sigma σ σ ( π ~ n , X n + 1 ) ⊆ σ ( D n , X n + 1 ) \sigma(\widetilde{\pi}_n,X_{n+1})\subseteq\sigma(D_n,X_{n+1}) σ ( π n , X n + 1 ) ⊆ σ ( D n , X n + 1 ) π ~ n \widetilde{\pi}_n π n D n D_n D n
E [ Y n + 1 ∣ π ~ n , X n + 1 ] = E [ E [ Y n + 1 ∣ D n , X n + 1 ] ∣ π ~ n , X n + 1 ] = E [ H ~ ( s ( π ~ n ) , X n + 1 ) ∣ π ~ n , X n + 1 ] = H ~ ( s ( π ~ n ) , X n + 1 ) = E [ Y n + 1 ∣ D n , X n + 1 ] . \begin{aligned}
\mathbb{E}[Y_{n+1}\mid\widetilde{\pi}_n,X_{n+1}]
&=\mathbb{E}[\mathbb{E}[Y_{n+1}\mid D_n,X_{n+1}]\mid\widetilde{\pi}_n,X_{n+1}]\\
&=\mathbb{E}[\widetilde{H}(s(\widetilde{\pi}_n),X_{n+1})\mid\widetilde{\pi}_n,X_{n+1}]
=\widetilde{H}(s(\widetilde{\pi}_n),X_{n+1})
=\mathbb{E}[Y_{n+1}\mid D_n,X_{n+1}].
\end{aligned} E [ Y n + 1 ∣ π n , X n + 1 ] = E [ E [ Y n + 1 ∣ D n , X n + 1 ] ∣ π n , X n + 1 ] = E [ H ( s ( π n ) , X n + 1 ) ∣ π n , X n + 1 ] = H ( s ( π n ) , X n + 1 ) = E [ Y n + 1 ∣ D n , X n + 1 ] . Proof is completed.
Proofs in Section 3.2
We calculate the Fréchet derivatives of ℓ n , N , B , λ ( M ) \ell_{n,N,B,\lambda}(M) ℓ n , N , B , λ ( M )
S n , N , B , λ ( M ) ≡ D ℓ n , N , B , λ ( M ) = − 1 N ∑ b = 1 B ∑ i = 1 N b ( Y n b + i b − M ( π ~ n b b , X n b + i b ) ) K ( π ~ n b b , X n b + i b ) + P λ M , \begin{aligned}
S_{n,N,B,\lambda}(M)\equiv D\ell_{n,N,B,\lambda}(M) \\
&=\frac{-1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left(Y_{n_b+i}^{b}-M(\widetilde{\pi}_{n_b}^{b},X_{n_b+i}^{b})\right)K(\widetilde{\pi}_{n_b}^{b},X_{n_b+i}^{b})+P_{\lambda}M,
\end{aligned} S n , N , B , λ ( M ) ≡ D ℓ n , N , B , λ ( M ) = N − 1 b = 1 ∑ B i = 1 ∑ N b ( Y n b + i b − M ( π n b b , X n b + i b ) ) K ( π n b b , X n b + i b ) + P λ M , D S n , N , B , λ ( M ) Δ M = 1 N ∑ b = 1 B ∑ i = 1 N b Δ M ( π ~ n b b , X n b + i b ) K ( π ~ n b b , X n b + i b ) + P λ Δ M , DS_{n,N,B,\lambda}(M)\Delta M
=\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\Delta M(\widetilde{\pi}_{n_b}^{b},X_{n_b+i}^{b})K(\widetilde{\pi}_{n_b}^{b},X_{n_b+i}^{b})+P_{\lambda}\Delta M, D S n , N , B , λ ( M ) Δ M = N 1 b = 1 ∑ B i = 1 ∑ N b Δ M ( π n b b , X n b + i b ) K ( π n b b , X n b + i b ) + P λ Δ M , D 2 S n , N , B , λ ( M ) ≡ 0. D^{2}S_{n,N,B,\lambda}(M)\equiv0. D 2 S n , N , B , λ ( M ) ≡ 0. By exchangeability, ( π ~ n b b , Z n b + i b ) (\widetilde{\pi}_{n_b}^{b},Z_{n_b+i}^{b}) ( π n b b , Z n b + i b ) ( π ~ n b b , Z n b + 1 b ) (\widetilde{\pi}_{n_b}^{b},Z_{n_b+1}^{b}) ( π n b b , Z n b + 1 b )
E [ S n , N , B , λ ( M ) ] = 1 N ∑ b = 1 B N b E [ ( M − M ∗ ) ( π ~ n b b , X n b + 1 b ) K ( π ~ n b b , X n b + 1 b ) ] + P λ M ≡ S ‾ n , B , λ ( M ) , \mathbb{E}[S_{n,N,B,\lambda}(M)]
=\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}[(M-M_*)(\widetilde{\pi}_{n_b}^{b},X_{n_b+1}^{b})K(\widetilde{\pi}_{n_b}^{b},X_{n_b+1}^{b})]+P_{\lambda}M\equiv\overline{S}_{n,B,\lambda}(M), E [ S n , N , B , λ ( M )] = N 1 b = 1 ∑ B N b E [( M − M ∗ ) ( π n b b , X n b + 1 b ) K ( π n b b , X n b + 1 b )] + P λ M ≡ S n , B , λ ( M ) , E [ D S n , N , B , λ ( M ) Δ M ] = 1 N ∑ b = 1 B N b E [ Δ M ( π ~ n b b , X n b + 1 b ) K ( π ~ n b b , X n b + 1 b ) ] + P λ Δ M ≡ D S ‾ n , B , λ ( M ) Δ M . \mathbb{E}[DS_{n,N,B,\lambda}(M)\Delta M]
=\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}[\Delta M(\widetilde{\pi}_{n_b}^{b},X_{n_b+1}^{b})K(\widetilde{\pi}_{n_b}^{b},X_{n_b+1}^{b})]+P_{\lambda}\Delta M\equiv D\overline{S}_{n,B,\lambda}(M)\Delta M. E [ D S n , N , B , λ ( M ) Δ M ] = N 1 b = 1 ∑ B N b E [ Δ M ( π n b b , X n b + 1 b ) K ( π n b b , X n b + 1 b )] + P λ Δ M ≡ D S n , B , λ ( M ) Δ M . Obviously, for any Δ M , Δ M ′ ∈ M \Delta M,\Delta M'\in\mathcal{M} Δ M , Δ M ′ ∈ M
⟨ D S ‾ n , B , λ ( M ) Δ M , Δ M ′ ⟩ M = ⟨ Δ M , Δ M ′ ⟩ M , \left\langle D\overline{S}_{n,B,\lambda}(M)\Delta M,\Delta M'\right\rangle_{\mathcal{M}}=\left\langle\Delta M,\Delta M'\right\rangle_{\mathcal{M}}, ⟨ D S n , B , λ ( M ) Δ M , Δ M ′ ⟩ M = ⟨ Δ M , Δ M ′ ⟩ M , so D S ‾ n , B , λ ( M ) = id D\overline{S}_{n,B,\lambda}(M)=\operatorname{id} D S n , B , λ ( M ) = id M \mathcal{M} M
S ‾ n , B , λ ( M ) = 1 N ∑ b = 1 B N b E [ ( M − M ∗ ) ( π ~ n b b , X n b + 1 b ) K ( π ~ n b b , X n b + 1 b ) ] + P λ ( M − M ∗ ) + P λ M ∗ = M − M ∗ + P λ M ∗ . \begin{aligned}
\overline{S}_{n,B,\lambda}(M)
&=\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}[(M-M_*)(\widetilde{\pi}_{n_b}^{b},X_{n_b+1}^{b})K(\widetilde{\pi}_{n_b}^{b},X_{n_b+1}^{b})]+P_{\lambda}(M-M_*)+P_{\lambda}M_*\\
&=M-M_*+P_{\lambda}M_*.
\end{aligned} S n , B , λ ( M ) = N 1 b = 1 ∑ B N b E [( M − M ∗ ) ( π n b b , X n b + 1 b ) K ( π n b b , X n b + 1 b )] + P λ ( M − M ∗ ) + P λ M ∗ = M − M ∗ + P λ M ∗ . Let M λ = ( id − P λ ) M ∗ M_\lambda=(\operatorname{id}-P_\lambda)M_* M λ = ( id − P λ ) M ∗ S ˉ n , B , λ ( M λ ) = 0 \bar{S}_{n,B,\lambda}(M_\lambda)=0 S ˉ n , B , λ ( M λ ) = 0
Lemma 4. It holds almost surely that K ( π ^ n b b , X n b + 1 b ) ∈ ker ( C ) ⊥ K(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)\in\operatorname{ker}(C)^\perp K ( π ^ n b b , X n b + 1 b ) ∈ ker ( C ) ⊥ ∥ K ( π ^ n b b , X n b + 1 b ) ∥ ≤ C φ h − 1 / 2 \left\|K(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)\right\|\le C_\varphi h^{-1/2} K ( π ^ n b b , X n b + 1 b ) ≤ C φ h − 1/2
Proof of Lemma 4. Decompose
K ( π ^ n b b , X n b + 1 b ) = M 1 + M 2 , for M 1 ∈ ker ( C ) and M 2 ∈ ker ( C ) ⊥ . K(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)=M_1+M_2,\text{ for }M_1\in\operatorname{ker}(C)\text{ and }M_2\in\operatorname{ker}(C)^\perp. K ( π ^ n b b , X n b + 1 b ) = M 1 + M 2 , for M 1 ∈ ker ( C ) and M 2 ∈ ker ( C ) ⊥ . Clearly,
1 N ∑ b = 1 B N b E [ ∣ M 1 ( π ^ n b b , X n b + 1 b ) ∣ 2 ] = V ( M 1 , M 1 ) = ⟨ C M 1 , M 1 ⟩ M = 0 , \frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}[|M_1(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)|^2]=V(M_1,M_1)=\langle CM_1,M_1\rangle_M=0, N 1 b = 1 ∑ B N b E [ ∣ M 1 ( π ^ n b b , X n b + 1 b ) ∣ 2 ] = V ( M 1 , M 1 ) = ⟨ C M 1 , M 1 ⟩ M = 0 , hence, M 1 ( π ^ n b b , X n b + 1 b ) = 0 M_1(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)=0 M 1 ( π ^ n b b , X n b + 1 b ) = 0
⟨ M 1 , K ( π ^ n b b , X n b + 1 b ) ⟩ = M 1 ( π ^ n b b , X n b + 1 b ) = 0 , almost surely . \langle M_1,K(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)\rangle=M_1(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)=0,\text{ almost surely}. ⟨ M 1 , K ( π ^ n b b , X n b + 1 b )⟩ = M 1 ( π ^ n b b , X n b + 1 b ) = 0 , almost surely . Meanwhile,
⟨ M 1 , M 2 ⟩ = V ( M 1 , M 2 ) + λ ⟨ M 1 , M 2 ⟩ M = ⟨ C M 1 , M 2 ⟩ M + λ ⟨ M 1 , M 2 ⟩ M = 0. \langle M_1,M_2\rangle=V(M_1,M_2)+\lambda\langle M_1,M_2\rangle_M=\langle CM_1,M_2\rangle_M+\lambda\langle M_1,M_2\rangle_M=0. ⟨ M 1 , M 2 ⟩ = V ( M 1 , M 2 ) + λ ⟨ M 1 , M 2 ⟩ M = ⟨ C M 1 , M 2 ⟩ M + λ ⟨ M 1 , M 2 ⟩ M = 0. So we have ∥ M 1 ∥ = 0 \|M_1\|=0 ∥ M 1 ∥ = 0 K ( π ^ n b b , X n b + 1 b ) = M 2 ∈ ker ( C n ) ⊥ K(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)=M_2\in\operatorname{ker}(C_n)^\perp K ( π ^ n b b , X n b + 1 b ) = M 2 ∈ ker ( C n ) ⊥ K ( π ^ n b b , X n b + 1 b ) = ∑ k ≥ 1 a k φ k K(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)=\sum_{k\ge1}a_k\varphi_k K ( π ^ n b b , X n b + 1 b ) = ∑ k ≥ 1 a k φ k
φ k ( π ^ n b b , X n b + 1 b ) = ⟨ φ k , K ( π ^ n b b , X n b + 1 b ) ⟩ = a k ( 1 + λ ρ k ) , (29) \varphi_k(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)=\langle\varphi_k,K(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)\rangle=a_k(1+\lambda\rho_k), \tag*{(29)} φ k ( π ^ n b b , X n b + 1 b ) = ⟨ φ k , K ( π ^ n b b , X n b + 1 b )⟩ = a k ( 1 + λ ρ k ) , (29) we have a k = φ k ( π ^ n b b , X n b + 1 b ) / ( 1 + λ ρ k ) a_k=\varphi_k(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)/(1+\lambda\rho_k) a k = φ k ( π ^ n b b , X n b + 1 b ) / ( 1 + λ ρ k )
∥ K ( π ^ n b b , X n b + 1 b ) ∥ 2 = ∑ k ≥ 1 ∣ φ k ( π ^ n b b , X n b + 1 b ) ∣ 2 1 + λ ρ k ≤ C φ 2 h − 1 . \left\|K(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)\right\|^2=\sum_{k\ge1}\frac{|\varphi_k(\hat{\pi}_{n_b}^b,X_{n_b+1}^b)|^2}{1+\lambda\rho_k}\le C_\varphi^2h^{-1}. K ( π ^ n b b , X n b + 1 b ) 2 = k ≥ 1 ∑ 1 + λ ρ k ∣ φ k ( π ^ n b b , X n b + 1 b ) ∣ 2 ≤ C φ 2 h − 1 . Proof is completed.
Proof of Theorem 2. Define T ( M ) = M − S n , N , B , λ ( M λ + M ) T(M)=M-S_{n,N,B,\lambda}(M_\lambda+M) T ( M ) = M − S n , N , B , λ ( M λ + M ) M ∈ M M\in\mathcal{M} M ∈ M [40 ] , T ( M ) T(M) T ( M )
T ( M ) = − Γ ( M ) − S n , N , B , λ ( M λ ) , T(M)=-\Gamma(M)-S_{n,N,B,\lambda}(M_\lambda), T ( M ) = − Γ ( M ) − S n , N , B , λ ( M λ ) , where
Γ ( M ) = 1 N ∑ b = 1 B ∑ i = 1 N b [ M ( π ^ n b b , X n b + i b ) K ( π ^ n b b , X n b + i b ) − E [ M ( π ^ n b b , X n b + i b ) K ( π ^ n b b , X n b + i b ) ] ] = 1 N ∑ b = 1 B ∑ i = 1 N b [ W i b ( M ) − E [ W i b ( M ) ] ] = 1 N ∑ b = 1 B ∑ i = 1 N b [ W i b ( M ) − E [ W i b ( M ) ∣ Π ] ] + 1 N ∑ b = 1 B ∑ i = 1 N b [ E [ W i b ( M ) ∣ Π ] − E [ W i b ( M ) ] ] ≡ Γ 1 ( M ) + Γ 2 ( M ) . \begin{aligned}
\Gamma(M)&=\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left[M(\hat{\pi}_{n_b}^b,X_{n_b+i}^b)K(\hat{\pi}_{n_b}^b,X_{n_b+i}^b)-\mathbb{E}\left[M(\hat{\pi}_{n_b}^b,X_{n_b+i}^b)K(\hat{\pi}_{n_b}^b,X_{n_b+i}^b)\right]\right]\\
&=\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left[W_i^b(M)-\mathbb{E}\left[W_i^b(M)\right]\right]\\
&=\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left[W_i^b(M)-\mathbb{E}\left[W_i^b(M)\mid\Pi\right]\right]+\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left[\mathbb{E}\left[W_i^b(M)\mid\Pi\right]-\mathbb{E}\left[W_i^b(M)\right]\right]\\
&\equiv\Gamma_1(M)+\Gamma_2(M).
\end{aligned} Γ ( M ) = N 1 b = 1 ∑ B i = 1 ∑ N b [ M ( π ^ n b b , X n b + i b ) K ( π ^ n b b , X n b + i b ) − E [ M ( π ^ n b b , X n b + i b ) K ( π ^ n b b , X n b + i b ) ] ] = N 1 b = 1 ∑ B i = 1 ∑ N b [ W i b ( M ) − E [ W i b ( M ) ] ] = N 1 b = 1 ∑ B i = 1 ∑ N b [ W i b ( M ) − E [ W i b ( M ) ∣ Π ] ] + N 1 b = 1 ∑ B i = 1 ∑ N b [ E [ W i b ( M ) ∣ Π ] − E [ W i b ( M ) ] ] ≡ Γ 1 ( M ) + Γ 2 ( M ) . In the above, we have set W i b ( M ) ≜ M ( π ^ n b b , X n b + i b ) K ( π ^ n b b , X n b + i b ) W_i^b(M) \triangleq M(\hat{\pi}_{n_b}^b, X_{n_b+i}^b)K(\hat{\pi}_{n_b}^b,X_{n_b+i}^b) W i b ( M ) ≜ M ( π ^ n b b , X n b + i b ) K ( π ^ n b b , X n b + i b ) Π = { D n 1 1 , … , D n B B } \boldsymbol{\Pi}=\{D_{n_1}^1,\ldots,D_{n_B}^B\} Π = { D n 1 1 , … , D n B B } D n b b = ( Z 1 b , … , Z N b b ) D_{n_b}^b=(Z_1^b,\ldots,Z_{N_b}^b) D n b b = ( Z 1 b , … , Z N b b ) b b b
Since W 1 b ( M ) , … , W N b b ( M ) W_1^b(M),\ldots,W_{N_b}^b(M) W 1 b ( M ) , … , W N b b ( M ) Π \boldsymbol{\Pi} Π ( W i b ( M ) , π ^ n b b ) (W_i^b(M),\hat{\pi}_{n_b}^b) ( W i b ( M ) , π ^ n b b ) { D n b ′ b ′ : b ′ ≠ b } \{D_{n_{b'}}^{b'}:b'\ne b\} { D n b ′ b ′ : b ′ = b }
Γ 2 ( M ) = 1 N ∑ b = 1 B ∑ i = 1 N b [ E [ W i b ( M ) ∣ D n b b ] − E [ W i b ( M ) ] ] = 1 N ∑ b = 1 B N b [ E [ W 1 b ( M ) ∣ D n b b ] − E [ W 1 b ( M ) ] ] . \Gamma_2(M)=\frac{1}{N}\sum_{b=1}^B\sum_{i=1}^{N_b}\left[\mathbb{E}\left[W_i^b(M)\mid D_{n_b}^b\right]-\mathbb{E}\left[W_i^b(M)\right]\right]=\frac{1}{N}\sum_{b=1}^B N_b\left[\mathbb{E}\left[W_1^b(M)\mid D_{n_b}^b\right]-\mathbb{E}\left[W_1^b(M)\right]\right]. Γ 2 ( M ) = N 1 b = 1 ∑ B i = 1 ∑ N b [ E [ W i b ( M ) ∣ D n b b ] − E [ W i b ( M ) ] ] = N 1 b = 1 ∑ B N b [ E [ W 1 b ( M ) ∣ D n b b ] − E [ W 1 b ( M ) ] ] . Since D n 1 1 , … , D n B B D_{n_1}^1,\ldots,D_{n_B}^B D n 1 1 , … , D n B B Lemma 4 it has ∥ W i b ( M ) ∥ ≤ C φ h − 1 / 2 ∥ M ∥ ∞ \|W_i^b(M)\|\le C_\varphi h^{-1/2}\|M\|_\infty ∥ W i b ( M ) ∥ ≤ C φ h − 1/2 ∥ M ∥ ∞ [25 ] that, for any M , M ′ ∈ M M,M'\in\mathcal{M} M , M ′ ∈ M
P ( ∥ N h ∑ b = 1 B N b 2 [ Γ 2 ( M ) − Γ 2 ( M ′ ) ] ∥ ≥ t ) ≤ 2 e − t 2 8 C φ 2 ∥ M − M ′ ∥ ∞ 2 , \mathbb{P}\left(\left\|N\sqrt{\frac{h}{\sum_{b=1}^B N_b^2}}\left[\Gamma_2(M)-\Gamma_2(M')\right]\right\|\ge t\right)\le2e^{-\frac{t^2}{8C_\varphi^2\|M-M'\|_\infty^2}}, P ( N ∑ b = 1 B N b 2 h [ Γ 2 ( M ) − Γ 2 ( M ′ ) ] ≥ t ) ≤ 2 e − 8 C φ 2 ∥ M − M ′ ∥ ∞ 2 t 2 , and so
∥ N h ∑ b = 1 B N b 2 [ Γ 2 ( M ) − Γ 2 ( M ′ ) ] ∥ ψ 2 ≤ 3 C φ ∥ M − M ′ ∥ ∞ , \left\|N\sqrt{\frac{h}{\sum_{b=1}^B N_b^2}}\left[\Gamma_2(M)-\Gamma_2(M')\right]\right\|_{\psi_2}\le3C_\varphi\|M-M'\|_\infty, N ∑ b = 1 B N b 2 h [ Γ 2 ( M ) − Γ 2 ( M ′ ) ] ψ 2 ≤ 3 C φ ∥ M − M ′ ∥ ∞ , where ∥ ⋅ ∥ ψ 2 \|\cdot\|_{\psi_2} ∥ ⋅ ∥ ψ 2 ψ 2 ( s ) ≜ e s 2 − 1 \psi_2(s)\triangleq e^{s^2}-1 ψ 2 ( s ) ≜ e s 2 − 1 [17 ] we have
N h ∑ b = 1 B N b 2 sup M ∈ M h ∥ Γ 2 ( M ) ∥ ≲ a h ≔ ∫ 0 1 ψ 2 − 1 ( D ( M h , ϵ , ∥ ⋅ ∥ ∞ ) ) d ϵ . N\sqrt{\frac{h}{\sum_{b=1}^B N_b^2}}\sup_{M\in\mathcal{M}_h}\|\Gamma_2(M)\|\lesssim a_h\coloneqq\int_0^1\psi_2^{-1}(D(\mathcal{M}_h,\epsilon,\|\cdot\|_\infty))d\epsilon. N ∑ b = 1 B N b 2 h M ∈ M h sup ∥ Γ 2 ( M ) ∥ ≲ a h : = ∫ 0 1 ψ 2 − 1 ( D ( M h , ϵ , ∥ ⋅ ∥ ∞ )) d ϵ . By Lemma 8.1 of Kosorok [17 ] , we have for any t ≥ 0 t\ge0 t ≥ 0
P ( sup M ∈ M h ∥ Γ 2 ( M ) ∥ ≥ t ) ≤ 2 e − h t 2 a h 2 ∑ b = 1 B N b 2 . (30) \mathbb{P}\left(\sup_{M\in\mathcal{M}_h}\|\Gamma_2(M)\|\ge t\right)\le2e^{-\frac{ht^2}{a_h^2\sum_{b=1}^B N_b^2}}. \tag*{(30)} P ( M ∈ M h sup ∥ Γ 2 ( M ) ∥ ≥ t ) ≤ 2 e − a h 2 ∑ b = 1 B N b 2 h t 2 . (30) So we have
sup M ∈ M h ∥ Γ 2 ( M ) ∥ = O P ( a h ∑ b = 1 B N b 2 N 2 h ) . \sup_{M\in\mathcal{M}_h}\|\Gamma_2(M)\|=O_{\mathbb{P}}\left(a_h\sqrt{\frac{\sum_{b=1}^B N_b^2}{N^2h}}\right). M ∈ M h sup ∥ Γ 2 ( M ) ∥ = O P a h N 2 h ∑ b = 1 B N b 2 . For any M , M ′ ∈ H M,M'\in\mathcal{H} M , M ′ ∈ H C φ − 1 h 1 / 2 ( M − M ′ ) / ∥ M − M ′ ∥ ∈ M h C_\varphi^{-1}h^{1/2}(M-M')/\|M-M'\|\in\mathcal{M}_h C φ − 1 h 1/2 ( M − M ′ ) /∥ M − M ′ ∥ ∈ M h
sup M , M ′ ∈ M M ≠ M ′ ∥ Γ 2 ( M ) − Γ 2 ( M ′ ) ∥ ∥ M − M ′ ∥ = sup M , M ′ ∈ M M ≠ M ′ ∥ Γ 2 ( C φ − 1 h 1 / 2 ( M − M ′ ) ∥ M − M ′ ∥ ) ∥ C φ h − 1 / 2 ≤ sup M ∈ M h ∥ Γ 2 ( M ) ∥ C φ h − 1 / 2 = O P ( a h ∑ b = 1 B N b 2 N h ) = o P ( 1 ) , (31) \begin{aligned}
\sup_{\substack{M,M'\in\mathcal{M}\\M\ne M'}}\frac{\|\Gamma_2(M)-\Gamma_2(M')\|}{\|M-M'\|}=\sup_{\substack{M,M'\in\mathcal{M}\\M\ne M'}}\left\|\Gamma_2\left(\frac{C_\varphi^{-1}h^{1/2}(M-M')}{\|M-M'\|}\right)\right\|C_\varphi h^{-1/2} \\
&\le\sup_{M\in\mathcal{M}_h}\|\Gamma_2(M)\|C_\varphi h^{-1/2}=O_{\mathbb{P}}\left(\frac{a_h\sqrt{\sum_{b=1}^B N_b^2}}{Nh}\right)=o_{\mathbb{P}}(1), \tag*{(31)}
\end{aligned} M , M ′ ∈ M M = M ′ sup ∥ M − M ′ ∥ ∥ Γ 2 ( M ) − Γ 2 ( M ′ ) ∥ = M , M ′ ∈ M M = M ′ sup Γ 2 ( ∥ M − M ′ ∥ C φ − 1 h 1/2 ( M − M ′ ) ) C φ h − 1/2 ≤ M ∈ M h sup ∥ Γ 2 ( M ) ∥ C φ h − 1/2 = O P N h a h ∑ b = 1 B N b 2 = o P ( 1 ) , (31) where the last o P ( 1 ) o_{\mathbb{P}}(1) o P ( 1 ) a h ∑ b = 1 B N b 2 = o ( N h ) a_h\sqrt{\sum_{b=1}^B N_b^2}=o(Nh) a h ∑ b = 1 B N b 2 = o ( N h )
To deal with Γ 1 \Gamma_1 Γ 1 Π \boldsymbol{\Pi} Π W i b ( M ) W_i^b(M) W i b ( M ) C φ h − 1 / 2 ∥ M − M ′ ∥ ∞ C_{\varphi}h^{-1/2}\|M-M'\|_{\infty} C φ h − 1/2 ∥ M − M ′ ∥ ∞ [25 ] , for any M , M ′ ∈ M M,M'\in\mathcal{M} M , M ′ ∈ M
P ( ∥ N h [ Γ 1 ( M ) − Γ 1 ( M ′ ) ] ∥ ≥ t ∣ Π ) ≤ 2 e − t 2 8 C φ 2 ∥ M − M ′ ∥ ∞ 2 . \mathbb{P}\left(\left\|\sqrt{Nh}[\Gamma_1(M)-\Gamma_1(M')]\right\|\geq t\mid\boldsymbol{\Pi}\right)\leq2e^{-\frac{t^2}{8C_{\varphi}^2\|M-M'\|_{\infty}^2}}. P ( N h [ Γ 1 ( M ) − Γ 1 ( M ′ )] ≥ t ∣ Π ) ≤ 2 e − 8 C φ 2 ∥ M − M ′ ∥ ∞ 2 t 2 . By Theorem 8.4 of Kosorok [17 ] , we have
∥ sup M ∈ M h N h ∥ Γ 1 ( M ) ∥ ∥ ψ 2 ∣ Π ≲ a h : = ∫ 0 1 ψ 2 − 1 ( D ( M h , ε , ∥ ⋅ ∥ ∞ ) ) d ε . \left\|\sup_{M\in\mathcal{M}_h}\sqrt{Nh}\|\Gamma_1(M)\|\right\|_{\psi_2\mid\boldsymbol{\Pi}}\lesssim a_h:=\int_0^1\psi_2^{-1}\left(D(\mathcal{M}_h,\varepsilon,\|\cdot\|_{\infty})\right)d\varepsilon. M ∈ M h sup N h ∥ Γ 1 ( M ) ∥ ψ 2 ∣ Π ≲ a h := ∫ 0 1 ψ 2 − 1 ( D ( M h , ε , ∥ ⋅ ∥ ∞ ) ) d ε . where ∥ ⋅ ∥ ψ 2 ∣ Π \|\cdot\|_{\psi_2\mid\boldsymbol{\Pi}} ∥ ⋅ ∥ ψ 2 ∣ Π Π \boldsymbol{\Pi} Π [17 ] , we have for any t ≥ 0 t\geq0 t ≥ 0
P ( sup M ∈ M h ∥ N h Γ 1 ( M ) ∥ ≥ t ∣ Π ) ≤ 2 e − t 2 a h 2 . (32) \mathbb{P}\left(\sup_{M\in\mathcal{M}_h}\left\|\sqrt{Nh}\Gamma_1(M)\right\|\geq t\mid\boldsymbol{\Pi}\right)\leq2e^{-\frac{t^2}{a_h^2}}. \tag*{(32)} P ( M ∈ M h sup N h Γ 1 ( M ) ≥ t ∣ Π ) ≤ 2 e − a h 2 t 2 . (32) Then it holds that
sup M ∈ M h ∥ Γ 1 ( M ) ∥ = O P ( a h N h ) . \sup_{M\in\mathcal{M}_h}\|\Gamma_1(M)\|=O_{\mathbb{P}}\left(\frac{a_h}{\sqrt{Nh}}\right). M ∈ M h sup ∥ Γ 1 ( M ) ∥ = O P ( N h a h ) . By condition a h ∑ b = 1 B N b 2 = o ( N h ) a_h\sqrt{\sum_{b=1}^B N_b^2}=o(Nh) a h ∑ b = 1 B N b 2 = o ( N h ) a h = o ( N h 2 ) a_h=o(\sqrt{Nh^2}) a h = o ( N h 2 )
sup M , M ′ ∈ M M ≠ M ′ ∥ Γ 1 ( M ) − Γ 1 ( M ′ ) ∥ ∥ M − M ′ ∥ = O P ( a h N h 2 ) = o P ( 1 ) . (33) \sup_{\substack{M,M'\in\mathcal{M}\\M\ne M'}}\frac{\|\Gamma_1(M)-\Gamma_1(M')\|}{\|M-M'\|}=O_{\mathbb{P}}\left(\frac{a_h}{\sqrt{Nh^2}}\right)=o_{\mathbb{P}}(1). \tag*{(33)} M , M ′ ∈ M M = M ′ sup ∥ M − M ′ ∥ ∥ Γ 1 ( M ) − Γ 1 ( M ′ ) ∥ = O P ( N h 2 a h ) = o P ( 1 ) . (33) Combining (31) and (33) , we get
sup M , M ′ ∈ M M ≠ M ′ ∥ T ( M ) − T ( M ′ ) ∥ ∥ M − M ′ ∥ = sup M , M ′ ∈ M M ≠ M ′ ∥ Γ ( M ) − Γ ( M ′ ) ∥ ∥ M − M ′ ∥ ≤ sup M , M ′ ∈ M M ≠ M ′ ∥ Γ 1 ( M ) − Γ 1 ( M ′ ) ∥ ∥ M − M ′ ∥ + sup M , M ′ ∈ M M ≠ M ′ ∥ Γ 2 ( M ) − Γ 2 ( M ′ ) ∥ ∥ M − M ′ ∥ = o P ( 1 ) . \begin{aligned}
\sup_{\substack{M,M'\in\mathcal{M}\\M\ne M'}}\frac{\|T(M)-T(M')\|}{\|M-M'\|}
&=\sup_{\substack{M,M'\in\mathcal{M}\\M\ne M'}}\frac{\|\Gamma(M)-\Gamma(M')\|}{\|M-M'\|}\\
&\leq\sup_{\substack{M,M'\in\mathcal{M}\\M\ne M'}}\frac{\|\Gamma_1(M)-\Gamma_1(M')\|}{\|M-M'\|}
+\sup_{\substack{M,M'\in\mathcal{M}\\M\ne M'}}\frac{\|\Gamma_2(M)-\Gamma_2(M')\|}{\|M-M'\|}\\
&=o_{\mathbb{P}}(1).
\end{aligned} M , M ′ ∈ M M = M ′ sup ∥ M − M ′ ∥ ∥ T ( M ) − T ( M ′ ) ∥ = M , M ′ ∈ M M = M ′ sup ∥ M − M ′ ∥ ∥Γ ( M ) − Γ ( M ′ ) ∥ ≤ M , M ′ ∈ M M = M ′ sup ∥ M − M ′ ∥ ∥ Γ 1 ( M ) − Γ 1 ( M ′ ) ∥ + M , M ′ ∈ M M = M ′ sup ∥ M − M ′ ∥ ∥ Γ 2 ( M ) − Γ 2 ( M ′ ) ∥ = o P ( 1 ) . Therefore, with probability approaching one, for M ∈ M M\in\mathcal{M} M ∈ M ∥ M ∥ ≤ 2 ∥ T ( 0 ) ∥ = 2 ∥ S n , N , B , λ ( M λ ) ∥ \|M\|\leq2\|T(0)\|=2\|S_{n,N,B,\lambda}(M_{\lambda})\| ∥ M ∥ ≤ 2∥ T ( 0 ) ∥ = 2∥ S n , N , B , λ ( M λ ) ∥ ∥ T ( M ) ∥ ≤ 2 ∥ T ( 0 ) ∥ \|T(M)\|\leq2\|T(0)\| ∥ T ( M ) ∥ ≤ 2∥ T ( 0 ) ∥ T T T Δ M \Delta M Δ M ∥ Δ M ∥ ≤ 2 ∥ T ( 0 ) ∥ \|\Delta M\|\leq2\|T(0)\| ∥Δ M ∥ ≤ 2∥ T ( 0 ) ∥ T ( Δ M ) = Δ M T(\Delta M)=\Delta M T ( Δ M ) = Δ M
∥ M ^ n , N , B , λ − M ∗ ∥ ≤ ∥ M λ − M ∗ ∥ + ∥ M ^ n , N , B , λ − M λ ∥ ≤ ∥ M λ − M ∗ ∥ + 2 ∥ S n , N , B , λ ( M λ ) ∥ . \|\widehat{M}_{n,N,B,\lambda}-M_*\|\leq\|M_{\lambda}-M_*\|+\|\widehat{M}_{n,N,B,\lambda}-M_{\lambda}\|\leq\|M_{\lambda}-M_*\|+2\|S_{n,N,B,\lambda}(M_{\lambda})\|. ∥ M n , N , B , λ − M ∗ ∥ ≤ ∥ M λ − M ∗ ∥ + ∥ M n , N , B , λ − M λ ∥ ≤ ∥ M λ − M ∗ ∥ + 2∥ S n , N , B , λ ( M λ ) ∥. We need to bound the second moment of ∥ S n , N , B , λ ( M λ ) ∥ \|S_{n,N,B,\lambda}(M_{\lambda})\| ∥ S n , N , B , λ ( M λ ) ∥ e n b , i b = Y n b + i b − M ∗ ( π ^ n b b , X n b + i b ) e^b_{n_b,i}=Y^b_{n_b+i}-M_*(\widehat{\pi}^b_{n_b},X^b_{n_b+i}) e n b , i b = Y n b + i b − M ∗ ( π n b b , X n b + i b )
Since E [ S n , N , B , λ ( M λ ) ] = 0 \mathbb{E}[S_{n,N,B,\lambda}(M_\lambda)]=0 E [ S n , N , B , λ ( M λ )] = 0
S n , N , B , λ ( M λ ) = S n , N , B , λ ( M λ ) − E [ S n , N , B , λ ( M λ ) ] = 1 N ∑ b = 1 B ∑ i = 1 N b [ W i b ( M λ − M ∗ ) − E [ W i b ( M λ − M ∗ ) ] ] − 1 N ∑ b = 1 B ∑ i = 1 N b e n b , i b K ( π ^ n b b , X n b + i b ) ≡ 1 N ∑ b = 1 B ∑ i = 1 N b [ W i b ( M λ − M ∗ ) − E [ W i b ( M λ − M ∗ ) ∣ D n b b ] ] + 1 N ∑ b = 1 B N b [ E [ W 1 b ( M λ − M ∗ ) ∣ D n b b ] − E [ W 1 b ( M λ − M ∗ ) ] ] − 1 N ∑ b = 1 B ∑ i = 1 N b e n b , i b K ( π ^ n b b , X n b + i b ) ≜ I 1 , 1 + I 1 , 2 − I 2 . \begin{aligned}
S_{n,N,B,\lambda}(M_\lambda)
&=S_{n,N,B,\lambda}(M_\lambda)-\mathbb{E}[S_{n,N,B,\lambda}(M_\lambda)]\\
&=\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left[W_i^b(M_\lambda-M_*)-\mathbb{E}[W_i^b(M_\lambda-M_*)]\right]
-\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}e_{n_b,i}^bK(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b)\\
&\equiv\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left[W_i^b(M_\lambda-M_*)-\mathbb{E}\left[W_i^b(M_\lambda-M_*)\mid D_{n_b}^b\right]\right]\\
&\quad+\frac{1}{N}\sum_{b=1}^{B}N_b\left[\mathbb{E}\left[W_1^b(M_\lambda-M_*)\mid D_{n_b}^b\right]-\mathbb{E}\left[W_1^b(M_\lambda-M_*)\right]\right]\\
&\quad-\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}e_{n_b,i}^bK(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b)\\
&\triangleq I_{1,1}+I_{1,2}-I_2.
\end{aligned} S n , N , B , λ ( M λ ) = S n , N , B , λ ( M λ ) − E [ S n , N , B , λ ( M λ )] = N 1 b = 1 ∑ B i = 1 ∑ N b [ W i b ( M λ − M ∗ ) − E [ W i b ( M λ − M ∗ )] ] − N 1 b = 1 ∑ B i = 1 ∑ N b e n b , i b K ( π n b b , X n b + i b ) ≡ N 1 b = 1 ∑ B i = 1 ∑ N b [ W i b ( M λ − M ∗ ) − E [ W i b ( M λ − M ∗ ) ∣ D n b b ] ] + N 1 b = 1 ∑ B N b [ E [ W 1 b ( M λ − M ∗ ) ∣ D n b b ] − E [ W 1 b ( M λ − M ∗ ) ] ] − N 1 b = 1 ∑ B i = 1 ∑ N b e n b , i b K ( π n b b , X n b + i b ) ≜ I 1 , 1 + I 1 , 2 − I 2 . We treat the second moments of the three terms respectively. First, W i b ( M λ − M ∗ ) , i = 1 , … , N b , b = 1 , … , B W_i^b(M_\lambda-M_*), i=1,\ldots,N_b, b=1,\ldots,B W i b ( M λ − M ∗ ) , i = 1 , … , N b , b = 1 , … , B Π \Pi Π Lemma 4 , we get that
E [ ∥ I 1 , 1 ∥ 2 ∣ Π ] = 1 N 2 ∑ b = 1 B ∑ i = 1 N b ( E [ ∥ W i b ( M λ − M ∗ ) ∥ 2 ∣ Π ] − ∥ E [ W i b ( M λ − M ∗ ) ∣ Π ] ∥ 2 ) ≤ 1 N 2 ∑ b = 1 B ∑ i = 1 N b E [ ∥ W i b ( M λ − M ∗ ) ∥ 2 ∣ Π ] ≤ C φ 2 N 2 h ∑ b = 1 B N b E [ ∣ ( P λ M ∗ ) ( π ^ n b b , X n b + 1 b ) ∣ 2 ∣ Π ] . \begin{aligned}
\mathbb{E}\left[\|I_{1,1}\|^2\mid\Pi\right]
&=\frac{1}{N^2}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left(\mathbb{E}\left[\|W_i^b(M_\lambda-M_*)\|^2\mid\Pi\right]-\|\mathbb{E}[W_i^b(M_\lambda-M_*)\mid\Pi]\|^2\right)\\
&\leq\frac{1}{N^2}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\mathbb{E}\left[\|W_i^b(M_\lambda-M_*)\|^2\mid\Pi\right]\\
&\leq\frac{C_\varphi^2}{N^2h}\sum_{b=1}^{B}N_b\mathbb{E}\left[|(P_\lambda M_*)(\widehat{\pi}_{n_b}^b,X_{n_b+1}^b)|^2\mid\Pi\right].
\end{aligned} E [ ∥ I 1 , 1 ∥ 2 ∣ Π ] = N 2 1 b = 1 ∑ B i = 1 ∑ N b ( E [ ∥ W i b ( M λ − M ∗ ) ∥ 2 ∣ Π ] − ∥ E [ W i b ( M λ − M ∗ ) ∣ Π ] ∥ 2 ) ≤ N 2 1 b = 1 ∑ B i = 1 ∑ N b E [ ∥ W i b ( M λ − M ∗ ) ∥ 2 ∣ Π ] ≤ N 2 h C φ 2 b = 1 ∑ B N b E [ ∣ ( P λ M ∗ ) ( π n b b , X n b + 1 b ) ∣ 2 ∣ Π ] . The last equality follows by the fact that ( π ^ n b b , X n b + i b ) (\widehat{\pi}_{n_b}^b,X_{n_b+i}^b) ( π n b b , X n b + i b ) ( π ^ n b b , X n b + 1 b ) (\widehat{\pi}_{n_b}^b,X_{n_b+1}^b) ( π n b b , X n b + 1 b ) M λ − M ∗ = P λ M ∗ M_\lambda-M_*=P_\lambda M_* M λ − M ∗ = P λ M ∗ P λ M ∗ P_\lambda M_* P λ M ∗
∥ P λ M ∗ ∥ = sup ∥ M ∥ = 1 ⟨ P λ M ∗ , M ⟩ = sup ∥ M ∥ = 1 λ ⟨ M ∗ , M ⟩ M ≤ λ ∥ M ∗ ∥ M , (34) \|P_\lambda M_*\|=\sup_{\|M\|=1}\langle P_\lambda M_*,M\rangle=\sup_{\|M\|=1}\lambda\langle M_*,M\rangle_M\leq\sqrt{\lambda}\|M_*\|_M, \tag*{(34)} ∥ P λ M ∗ ∥ = ∥ M ∥ = 1 sup ⟨ P λ M ∗ , M ⟩ = ∥ M ∥ = 1 sup λ ⟨ M ∗ , M ⟩ M ≤ λ ∥ M ∗ ∥ M , (34) Therefore, taking expectation with respect to Π \Pi Π (34) ,
E [ ∥ I 1 , 1 ∥ 2 ] ≤ C φ 2 N 2 h ∑ b = 1 B N b E [ ∣ ( P λ M ∗ ) ( π ^ n b b , X n b + 1 b ) ∣ 2 ] = C φ 2 N h V ( P λ M ∗ , P λ M ∗ ) ≤ C φ 2 N h ∥ P λ M ∗ ∥ 2 ≤ C φ 2 ∥ M ∗ ∥ M 2 λ N h . (35) \begin{aligned}
\mathbb{E}[\|I_{1,1}\|^2]
&\leq\frac{C_\varphi^2}{N^2h}\sum_{b=1}^{B}N_b\mathbb{E}[|(P_\lambda M_*)(\widehat{\pi}_{n_b}^b,X_{n_b+1}^b)|^2]
=\frac{C_\varphi^2}{Nh}V(P_\lambda M_*,P_\lambda M_*)\\
&\leq\frac{C_\varphi^2}{Nh}\|P_\lambda M_*\|^2
\leq\frac{C_\varphi^2\|M_*\|_M^2\lambda}{Nh}. \tag*{(35)}
\end{aligned} E [ ∥ I 1 , 1 ∥ 2 ] ≤ N 2 h C φ 2 b = 1 ∑ B N b E [ ∣ ( P λ M ∗ ) ( π n b b , X n b + 1 b ) ∣ 2 ] = N h C φ 2 V ( P λ M ∗ , P λ M ∗ ) ≤ N h C φ 2 ∥ P λ M ∗ ∥ 2 ≤ N h C φ 2 ∥ M ∗ ∥ M 2 λ . (35) Second, V b ≜ E [ W 1 b ( M λ − M ∗ ) ∣ D n b b ] V_b\triangleq\mathbb{E}[W_1^b(M_\lambda-M_*)\mid D_{n_b}^b] V b ≜ E [ W 1 b ( M λ − M ∗ ) ∣ D n b b ] b = 1 , … , B b=1,\ldots,B b = 1 , … , B
E [ ∥ I 1 , 2 ∥ 2 ] = 1 N 2 ∑ b = 1 B N b 2 ( E [ ∥ V b ∥ 2 ] − ∥ E [ V b ] ∥ 2 ) ≤ 1 N 2 ∑ b = 1 B N b 2 E [ ∥ V b ∥ 2 ] ≤ 1 N 2 ∑ b = 1 B N b 2 E [ ∥ W 1 b ( M λ − M ∗ ) ∥ 2 ] ≤ C φ 2 max 1 ≤ b ≤ B N b N h × 1 N ∑ b = 1 B N b E [ ∣ ( P λ M ∗ ) ( π ^ n b b , X n b + 1 b ) ∣ 2 ] = C φ 2 max 1 ≤ b ≤ B N b N h V ( P λ M ∗ , P λ M ∗ ) ≤ C φ 2 ∥ M ∗ ∥ 2 max 1 ≤ b ≤ B N b λ N h . (36) \begin{aligned}
\mathbb{E}[\|I_{1,2}\|^2] = \frac{1}{N^2}\sum_{b=1}^{B}N_b^2\left(\mathbb{E}[\|V_b\|^2]-\|\mathbb{E}[V_b]\|^2\right) \le\frac{1}{N^2}\sum_{b=1}^{B}N_b^2\mathbb{E}[\|V_b\|^2] \\
&\le\frac{1}{N^2}\sum_{b=1}^{B}N_b^2\mathbb{E}[\|W_1^b(M_\lambda-M_*)\|^2] \\
&\le\frac{C_\varphi^2\max_{1\le b\le B}N_b}{Nh}\times\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}[|(P_\lambda M_*)(\widehat{\pi}_{n_b}^b,X_{n_b+1}^b)|^2] \\
&= \frac{C_\varphi^2\max_{1\le b\le B}N_b}{Nh}V(P_\lambda M_*,P_\lambda M_*)\le\frac{C_\varphi^2\|M_*\|^2\max_{1\le b\le B}N_b\lambda}{Nh}. \tag*{(36)}
\end{aligned} E [ ∥ I 1 , 2 ∥ 2 ] = N 2 1 b = 1 ∑ B N b 2 ( E [ ∥ V b ∥ 2 ] − ∥ E [ V b ] ∥ 2 ) ≤ N 2 1 b = 1 ∑ B N b 2 E [ ∥ V b ∥ 2 ] ≤ N 2 1 b = 1 ∑ B N b 2 E [ ∥ W 1 b ( M λ − M ∗ ) ∥ 2 ] ≤ N h C φ 2 max 1 ≤ b ≤ B N b × N 1 b = 1 ∑ B N b E [ ∣ ( P λ M ∗ ) ( π n b b , X n b + 1 b ) ∣ 2 ] = N h C φ 2 max 1 ≤ b ≤ B N b V ( P λ M ∗ , P λ M ∗ ) ≤ N h C φ 2 ∥ M ∗ ∥ 2 max 1 ≤ b ≤ B N b λ . (36) Third, we analyze the second moment of I 2 I_2 I 2 e n b , i b e_{n_b,i}^b e n b , i b D n b b , X n b + i b D_{n_b}^b,X_{n_b+i}^b D n b b , X n b + i b
E [ ∥ I 2 ∥ 2 ] = 1 N 2 ∑ b , b ′ = 1 B ∑ i = 1 N b ∑ i ′ = 1 N b ′ E [ e n b , i b e n b ′ , i ′ b ′ ⟨ K ( π ^ n b b , X n b + i b ) , K ( π ^ n b ′ b ′ , X n b ′ + i ′ b ′ ) ⟩ ] = 1 N 2 ∑ b = 1 B ∑ i , i ′ = 1 N b E [ e n b , i b e n b , i ′ b ⟨ K ( π ^ n b b , X n b + i b ) , K ( π ^ n b b , X n b + i ′ b ) ⟩ ] . (37) \begin{aligned}
\mathbb{E}[\|I_2\|^2]=\frac{1}{N^2}\sum_{b,b'=1}^{B}\sum_{i=1}^{N_b}\sum_{i'=1}^{N_{b'}}\mathbb{E}\left[e_{n_b,i}^be_{n_{b'},i'}^{b'}\left\langle K(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),K(\widehat{\pi}_{n_{b'}}^{b'},X_{n_{b'}+i'}^{b'})\right\rangle\right] \\
&=\frac{1}{N^2}\sum_{b=1}^{B}\sum_{i,i'=1}^{N_b}\mathbb{E}\left[e_{n_b,i}^be_{n_b,i'}^b\left\langle K(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),K(\widehat{\pi}_{n_b}^b,X_{n_b+i'}^b)\right\rangle\right]. \tag*{(37)}
\end{aligned} E [ ∥ I 2 ∥ 2 ] = N 2 1 b , b ′ = 1 ∑ B i = 1 ∑ N b i ′ = 1 ∑ N b ′ E [ e n b , i b e n b ′ , i ′ b ′ ⟨ K ( π n b b , X n b + i b ) , K ( π n b ′ b ′ , X n b ′ + i ′ b ′ ) ⟩ ] = N 2 1 b = 1 ∑ B i , i ′ = 1 ∑ N b E [ e n b , i b e n b , i ′ b ⟨ K ( π n b b , X n b + i b ) , K ( π n b b , X n b + i ′ b ) ⟩ ] . (37) For i ≠ i ′ i\ne i' i = i ′ Z n b + i b Z_{n_b+i}^b Z n b + i b Z n b + i ′ b Z_{n_b+i'}^b Z n b + i ′ b ( D n b b , f b ) (D_{n_b}^b,f_b) ( D n b b , f b ) δ n b , i b = e n b , i b K ( π ^ n b b , X n b + i b ) \delta_{n_b,i}^b=e_{n_b,i}^bK(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b) δ n b , i b = e n b , i b K ( π n b b , X n b + i b )
E [ e n b , i b e n b , i ′ b ⟨ K ( π ^ n b b , X n b + i b ) , K ( π ^ n b b , X n b + i ′ b ) ⟩ ] = E [ E [ ⟨ δ n b , i b , δ n b , i ′ b ⟩ ∣ D n b b , f b ] ] = E [ ⟨ E [ δ n b , i b ∣ D n b b , f b ] , E [ δ n b , i ′ b ∣ D n b b , f b ] ⟩ ] = E [ ∥ E [ δ n b , i b ∣ D n b b , f b ] ∥ 2 ] . \begin{aligned}
\mathbb{E}\left[e_{n_b,i}^be_{n_b,i'}^b\left\langle K(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),K(\widehat{\pi}_{n_b}^b,X_{n_b+i'}^b)\right\rangle\right]
=\mathbb{E}\left[\mathbb{E}[\langle\delta_{n_b,i}^b,\delta_{n_b,i'}^b\rangle\mid D_{n_b}^b,f_b]\right] \\
&=\mathbb{E}\left[\left\langle\mathbb{E}[\delta_{n_b,i}^b\mid D_{n_b}^b,f_b],\mathbb{E}[\delta_{n_b,i'}^b\mid D_{n_b}^b,f_b]\right\rangle\right] \\
&=\mathbb{E}\left[\|\mathbb{E}[\delta_{n_b,i}^b\mid D_{n_b}^b,f_b]\|^2\right].
\end{aligned} E [ e n b , i b e n b , i ′ b ⟨ K ( π n b b , X n b + i b ) , K ( π n b b , X n b + i ′ b ) ⟩ ] = E [ E [⟨ δ n b , i b , δ n b , i ′ b ⟩ ∣ D n b b , f b ] ] = E [ ⟨ E [ δ n b , i b ∣ D n b b , f b ] , E [ δ n b , i ′ b ∣ D n b b , f b ] ⟩ ] = E [ ∥ E [ δ n b , i b ∣ D n b b , f b ] ∥ 2 ] . Since Z n b + i b Z_{n_b+i}^b Z n b + i b D n b b D_{n_b}^b D n b b f b f_b f b
E [ Y n b + i b ∣ D n b b , f b , X n b + i b ] = E [ Y n b + i b ∣ f b , X n b + i b ] = f b ( X n b + i b ) . (38) \mathbb{E}[Y_{n_b+i}^b\mid D_{n_b}^b,f_b,X_{n_b+i}^b]=\mathbb{E}[Y_{n_b+i}^b\mid f_b,X_{n_b+i}^b]=f_b(X_{n_b+i}^b). \tag*{(38)} E [ Y n b + i b ∣ D n b b , f b , X n b + i b ] = E [ Y n b + i b ∣ f b , X n b + i b ] = f b ( X n b + i b ) . (38) Also, M ∗ ( π ^ n b b , X n b + i b ) = E [ f b ∣ π ^ n b b ] ( X n b + i b ) M_*(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b)=\mathbb{E}[f_b\mid\widehat{\pi}_{n_b}^b](X_{n_b+i}^b) M ∗ ( π n b b , X n b + i b ) = E [ f b ∣ π n b b ] ( X n b + i b )
E [ δ n b , i b ∣ D n b b , f b , X n b + i b ] = E [ e n b , i b K ( π ^ n b b , X n b + i b ) ∣ D n b b , f b , X n b + i b ] = E [ e n b , i b ∣ D n b b , f b , X n b + i b ] K ( π ^ n b b , X n b + i b ) = ( E [ Y n b + i b ∣ D n b b , f b , X n b + i b ] − M ∗ ( π ^ n b b , X n b + i b ) ) K ( π ^ n b b , X n b + i b ) = ( f b ( X n b + i b ) − E [ f b ∣ π ^ n b b ] ( X n b + i b ) ) K ( π ^ n b b , X n b + i b ) , \begin{aligned}
\mathbb{E}[\delta_{n_b,i}^b\mid D_{n_b}^b,f_b,X_{n_b+i}^b]
=\mathbb{E}[e_{n_b,i}^bK(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b)\mid D_{n_b}^b,f_b,X_{n_b+i}^b]
=\mathbb{E}[e_{n_b,i}^b\mid D_{n_b}^b,f_b,X_{n_b+i}^b]K(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b) \\
&=\left(\mathbb{E}[Y_{n_b+i}^b\mid D_{n_b}^b,f_b,X_{n_b+i}^b]-M_*(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b)\right)K(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b) \\
&=\left(f_b(X_{n_b+i}^b)-\mathbb{E}[f_b\mid\widehat{\pi}_{n_b}^b](X_{n_b+i}^b)\right)K(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),
\end{aligned} E [ δ n b , i b ∣ D n b b , f b , X n b + i b ] = E [ e n b , i b K ( π n b b , X n b + i b ) ∣ D n b b , f b , X n b + i b ] = E [ e n b , i b ∣ D n b b , f b , X n b + i b ] K ( π n b b , X n b + i b ) = ( E [ Y n b + i b ∣ D n b b , f b , X n b + i b ] − M ∗ ( π n b b , X n b + i b ) ) K ( π n b b , X n b + i b ) = ( f b ( X n b + i b ) − E [ f b ∣ π n b b ] ( X n b + i b ) ) K ( π n b b , X n b + i b ) , which implies
E [ δ n b , i b ∣ D n b b , f b ] = E [ ( f b ( X n b + i b ) − E [ f b ∣ π ^ n b b ] ( X n b + i b ) ) K ( π ^ n b b , X n b + i b ) ∣ D n b b , f b ] . \mathbb{E}[\delta_{n_b,i}^b\mid D_{n_b}^b,f_b]
=\mathbb{E}\left[(f_b(X_{n_b+i}^b)-\mathbb{E}[f_b\mid\widehat{\pi}_{n_b}^b](X_{n_b+i}^b))K(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b)\mid D_{n_b}^b,f_b\right]. E [ δ n b , i b ∣ D n b b , f b ] = E [ ( f b ( X n b + i b ) − E [ f b ∣ π n b b ] ( X n b + i b )) K ( π n b b , X n b + i b ) ∣ D n b b , f b ] . By Lemma 4 , we get
∥ E [ δ n b , i b ∣ D n b b , f b ] ∥ ≤ C φ h − 1 / 2 E [ ∣ f b ( X n b + i b ) − E [ f b ∣ π ^ n b b ] ( X n b + i b ) ∣ D n b b , f b ] . \left\|\mathbb{E}[\delta^{b}_{n_b,i}\mid D^{b}_{n_b},f_b]\right\| \le C_{\varphi}h^{-1/2}\mathbb{E}\left[|f_b(X^{b}_{n_b+i})-\mathbb{E}[f_b\mid\widehat{\pi}^{b}_{n_b}](X^{b}_{n_b+i})|D^{b}_{n_b},f_b\right]. E [ δ n b , i b ∣ D n b b , f b ] ≤ C φ h − 1/2 E [ ∣ f b ( X n b + i b ) − E [ f b ∣ π n b b ] ( X n b + i b ) ∣ D n b b , f b ] . hence,
∥ E [ δ n b , i b ∣ D n b b , f b ] ∥ 2 ≤ C φ 2 h − 1 E [ ∣ f b ( X n b + i b ) − E [ f b ∣ π ^ n b b ] ( X n b + i b ) ∣ 2 ∣ D n b b , f b ] . \left\|\mathbb{E}[\delta^{b}_{n_b,i}\mid D^{b}_{n_b},f_b]\right\|^{2}\le C_{\varphi}^{2}h^{-1}\mathbb{E}\left[|f_b(X^{b}_{n_b+i})-\mathbb{E}[f_b\mid\widehat{\pi}^{b}_{n_b}](X^{b}_{n_b+i})|^{2}\mid D^{b}_{n_b},f_b\right]. E [ δ n b , i b ∣ D n b b , f b ] 2 ≤ C φ 2 h − 1 E [ ∣ f b ( X n b + i b ) − E [ f b ∣ π n b b ] ( X n b + i b ) ∣ 2 ∣ D n b b , f b ] . Therefore,
E [ e n b , i b e n b , i ′ b ⟨ K ( π ^ n b b , X n b + i b ) , K ( π ^ n b b , X n b + i ′ b ) ⟩ ] ≤ C φ 2 h − 1 E [ ∣ f b ( X n b + i b ) − E [ f b ∣ π ^ n b b ] ( X n b + i b ) ∣ 2 ] ≡ C φ 2 h − 1 V b 2 ( n ) , \mathbb{E}\left[e^{b}_{n_b,i}e^{b}_{n_b,i'}\left\langle K(\widehat{\pi}^{b}_{n_b},X^{b}_{n_b+i}),K(\widehat{\pi}^{b}_{n_b},X^{b}_{n_b+i'})\right\rangle\right]\le C_{\varphi}^{2}h^{-1}\mathbb{E}\left[|f_b(X^{b}_{n_b+i})-\mathbb{E}[f_b\mid\widehat{\pi}^{b}_{n_b}](X^{b}_{n_b+i})|^{2}\right]\equiv C_{\varphi}^{2}h^{-1}V_b^{2}(\mathbf{n}), E [ e n b , i b e n b , i ′ b ⟨ K ( π n b b , X n b + i b ) , K ( π n b b , X n b + i ′ b ) ⟩ ] ≤ C φ 2 h − 1 E [ ∣ f b ( X n b + i b ) − E [ f b ∣ π n b b ] ( X n b + i b ) ∣ 2 ] ≡ C φ 2 h − 1 V b 2 ( n ) , where recall V b 2 ( n ) = E [ ∣ f b ( X n b + i b ) − E [ f b ∣ π ^ n b b ] ( X n b + i b ) ∣ 2 V_b^{2}(\mathbf{n})=\mathbb{E}[|f_b(X^{b}_{n_b+i})-\mathbb{E}[f_b\mid\widehat{\pi}^{b}_{n_b}](X^{b}_{n_b+i})|^{2} V b 2 ( n ) = E [ ∣ f b ( X n b + i b ) − E [ f b ∣ π n b b ] ( X n b + i b ) ∣ 2 Lemma 4 , for i = i ′ i=i' i = i ′
E [ e n b , i b e n b , i ′ b ⟨ K ( π ^ n b b , X n b + i b ) , K ( π ^ n b b , X n b + i ′ b ) ⟩ ] ≤ C φ 2 h − 1 σ e 2 . \mathbb{E}\left[e^{b}_{n_b,i}e^{b}_{n_b,i'}\left\langle K(\widehat{\pi}^{b}_{n_b},X^{b}_{n_b+i}),K(\widehat{\pi}^{b}_{n_b},X^{b}_{n_b+i'})\right\rangle\right]\le C_{\varphi}^{2}h^{-1}\sigma_{e}^{2}. E [ e n b , i b e n b , i ′ b ⟨ K ( π n b b , X n b + i b ) , K ( π n b b , X n b + i ′ b ) ⟩ ] ≤ C φ 2 h − 1 σ e 2 . Plugging into (37) , we get
E [ ∥ I 2 ∥ 2 ] ≤ C φ 2 ∑ b = 1 B N b 2 V b 2 ( n ) N 2 h + C φ 2 σ e 2 N h ≤ C φ 2 V 2 ( n ) max 1 ≤ b ≤ B N b N h + C φ 2 σ e 2 N h . (39) \mathbb{E}[\|I_2\|^{2}]\le\frac{C_{\varphi}^{2}\sum_{b=1}^{B}N_b^{2}V_b^{2}(\mathbf{n})}{N^{2}h}+\frac{C_{\varphi}^{2}\sigma_e^{2}}{Nh}\le\frac{C_{\varphi}^{2}V^{2}(\mathbf{n})\max_{1\le b\le B}N_b}{Nh}+\frac{C_{\varphi}^{2}\sigma_e^{2}}{Nh}. \tag*{(39)} E [ ∥ I 2 ∥ 2 ] ≤ N 2 h C φ 2 ∑ b = 1 B N b 2 V b 2 ( n ) + N h C φ 2 σ e 2 ≤ N h C φ 2 V 2 ( n ) max 1 ≤ b ≤ B N b + N h C φ 2 σ e 2 . (39) Recalling V 2 ( n ) = max 1 ≤ b ≤ B V b 2 ( n ) V^{2}(\mathbf{n})=\max_{1\le b\le B}V_b^{2}(\mathbf{n}) V 2 ( n ) = max 1 ≤ b ≤ B V b 2 ( n ) (35) , (36) and (39) into S n , B , λ ( M λ ) S_{n,B,\lambda}(M_{\lambda}) S n , B , λ ( M λ )
∥ S n , N , B , λ ( M λ ) ∥ = O P ( λ 1 / 2 ( N h ) − 1 / 2 + ( λ max 1 ≤ b ≤ B N b ) 1 / 2 ( N h ) − 1 / 2 + ( N h ) − 1 / 2 + V ( n ) ( max 1 ≤ b ≤ B N b ) 1 / 2 ( N h ) − 1 / 2 ) = O P ( ( λ max 1 ≤ b ≤ B N b ) 1 / 2 ( N h ) − 1 / 2 + ( N h ) − 1 / 2 + V ( n ) ( max 1 ≤ b ≤ B N b ) 1 / 2 ( N h ) − 1 / 2 ) = O P ( U ( n , N , B , λ ) ( N h ) − 1 / 2 ) , \begin{aligned}
\|S_{n,N,B,\lambda}(M_{\lambda})\|
&=O_{\mathbb{P}}\left(\lambda^{1/2}(Nh)^{-1/2}+(\lambda\max_{1\le b\le B}N_b)^{1/2}(Nh)^{-1/2}+(Nh)^{-1/2}+V(\mathbf{n})(\max_{1\le b\le B}N_b)^{1/2}(Nh)^{-1/2}\right)\\
&=O_{\mathbb{P}}\left((\lambda\max_{1\le b\le B}N_b)^{1/2}(Nh)^{-1/2}+(Nh)^{-1/2}+V(\mathbf{n})(\max_{1\le b\le B}N_b)^{1/2}(Nh)^{-1/2}\right)\\
&=O_{\mathbb{P}}\left(U(\mathbf{n},N,B,\lambda)(Nh)^{-1/2}\right),
\end{aligned} ∥ S n , N , B , λ ( M λ ) ∥ = O P ( λ 1/2 ( N h ) − 1/2 + ( λ 1 ≤ b ≤ B max N b ) 1/2 ( N h ) − 1/2 + ( N h ) − 1/2 + V ( n ) ( 1 ≤ b ≤ B max N b ) 1/2 ( N h ) − 1/2 ) = O P ( ( λ 1 ≤ b ≤ B max N b ) 1/2 ( N h ) − 1/2 + ( N h ) − 1/2 + V ( n ) ( 1 ≤ b ≤ B max N b ) 1/2 ( N h ) − 1/2 ) = O P ( U ( n , N , B , λ ) ( N h ) − 1/2 ) , where U ( n , N , B , λ ) = max { 1 , ( λ max 1 ≤ b ≤ B N b ) 1 / 2 , V ( n ) ( max 1 ≤ b ≤ B N b ) 1 / 2 } U(\mathbf{n},N,B,\lambda)=\max\{1,(\lambda\max_{1\le b\le B}N_b)^{1/2},V(\mathbf{n})(\max_{1\le b\le B}N_b)^{1/2}\} U ( n , N , B , λ ) = max { 1 , ( λ max 1 ≤ b ≤ B N b ) 1/2 , V ( n ) ( max 1 ≤ b ≤ B N b ) 1/2 } (34) ,
∥ M ^ n , N , B , λ − M ∗ ∥ ≤ ∥ P λ M ∗ ∥ + 2 ∥ S n , N , B , λ ( M λ ) ∥ = O P ( λ 1 / 2 + U ( n , N , B , λ ) ( N h ) − 1 / 2 ) . \|\widehat{M}_{n,N,B,\lambda}-M_{*}\|\le\|P_{\lambda}M_{*}\|+2\|S_{n,N,B,\lambda}(M_{\lambda})\|=O_{\mathbb{P}}\left(\lambda^{1/2}+U(\mathbf{n},N,B,\lambda)(Nh)^{-1/2}\right). ∥ M n , N , B , λ − M ∗ ∥ ≤ ∥ P λ M ∗ ∥ + 2∥ S n , N , B , λ ( M λ ) ∥ = O P ( λ 1/2 + U ( n , N , B , λ ) ( N h ) − 1/2 ) . Proof is complete.
The proof of Lemma 2 requires the following Lemma 5 that provides a L 2 L^{2} L 2 f b f_b f b D n b b D^{b}_{n_b} D n b b f b f_b f b f b f_b f b
f ^ b , η = arg min f ∈ H X ℓ b , η ( f ) ≡ arg min f ∈ H X { 1 2 n b ∑ i = 1 n b ∣ Y i b − f ( X i b ) ∣ 2 + η 2 ∥ f ∥ H X 2 } . \widehat{f}_{b,\eta}=\mathop{\arg\min}_{f\in\mathcal{H}_{X}}\ell_{b,\eta}(f)\equiv\mathop{\arg\min}_{f\in\mathcal{H}_{X}}\left\{\frac{1}{2n_b}\sum_{i=1}^{n_b}|Y_i^b-f(X_i^b)|^{2}+\frac{\eta}{2}\|f\|_{\mathcal{H}_{X}}^{2}\right\}. f b , η = arg min f ∈ H X ℓ b , η ( f ) ≡ arg min f ∈ H X { 2 n b 1 i = 1 ∑ n b ∣ Y i b − f ( X i b ) ∣ 2 + 2 η ∥ f ∥ H X 2 } . where η ≍ n b − 2 γ 2 γ + 1 \eta\asymp n_b^{-\frac{2\gamma}{2\gamma+1}} η ≍ n b − 2 γ + 1 2 γ f , f ′ ∈ H X f,f'\in\mathcal{H}_{X} f , f ′ ∈ H X
V X ( f , f ′ ) = E X ∼ μ [ f ( X ) f ′ ( X ) ] = ∫ X f ( x ) f ′ ( x ) d μ ( x ) , V_X(f,f')=\mathbb{E}_{X\sim\mu}[f(X)f'(X)]=\int_{\mathcal{X}}f(x)f'(x)d\mu(x), V X ( f , f ′ ) = E X ∼ μ [ f ( X ) f ′ ( X )] = ∫ X f ( x ) f ′ ( x ) d μ ( x ) , and define a new inner product for H X \mathcal{H}_X H X
⟨ f , f ′ ⟩ X = V X ( f , f ′ ) + η ⟨ f , f ′ ⟩ H X . (40) \langle f, f' \rangle_X = V_X(f, f') + \eta\langle f, f' \rangle_{\mathcal{H}_X}. \tag*{(40)} ⟨ f , f ′ ⟩ X = V X ( f , f ′ ) + η ⟨ f , f ′ ⟩ H X . (40) Under (40) , H X \mathcal{H}_X H X K ⟨ X ⟩ : X × X → R K^{\langle X\rangle}: \mathcal{X} \times\mathcal{X} \to\mathbb{R} K ⟨ X ⟩ : X × X → R P η ⟨ X ⟩ : H X → H X P_{\eta}^{\langle X\rangle}: \mathcal{H}_X \to\mathcal{H}_X P η ⟨ X ⟩ : H X → H X ⟨ P η ⟨ X ⟩ f , f ′ ⟩ X = η ⟨ f , f ′ ⟩ H X \langle P_{\eta}^{\langle X\rangle}f, f' \rangle_X = \eta\langle f, f' \rangle_{\mathcal{H}_X} ⟨ P η ⟨ X ⟩ f , f ′ ⟩ X = η ⟨ f , f ′ ⟩ H X ℓ b , η ( f ) \ell_{b,\eta}(f) ℓ b , η ( f )
S b , η ( f ) ≡ D ℓ b , η ( f ) = − n b − 1 ∑ i = 1 n b ( Y i b − f ( X i b ) ) K X i b ⟨ X ⟩ + P η ⟨ X ⟩ f , S_{b,\eta}(f) \equiv D\ell_{b,\eta}(f) = -n_b^{-1}\sum_{i=1}^{n_b}\left(Y_i^b-f(X_i^b)\right)K_{X_i^b}^{\langle X\rangle}+P_{\eta}^{\langle X\rangle}f, S b , η ( f ) ≡ D ℓ b , η ( f ) = − n b − 1 i = 1 ∑ n b ( Y i b − f ( X i b ) ) K X i b ⟨ X ⟩ + P η ⟨ X ⟩ f , D S b , η ( f ) Δ f = n b − 1 ∑ i = 1 n b Δ f ( X i b ) K X i b ⟨ X ⟩ + P η ⟨ X ⟩ Δ m , DS_{b,\eta}(f)\Delta f = n_b^{-1}\sum_{i=1}^{n_b}\Delta f(X_i^b)K_{X_i^b}^{\langle X\rangle}+P_{\eta}^{\langle X\rangle}\Delta m, D S b , η ( f ) Δ f = n b − 1 i = 1 ∑ n b Δ f ( X i b ) K X i b ⟨ X ⟩ + P η ⟨ X ⟩ Δ m , D 2 S b , η ( f ) ≡ 0. D^2S_{b,\eta}(f)\equiv0. D 2 S b , η ( f ) ≡ 0. The average versions under f b f_b f b
S η ( f ) = E X ∼ μ [ ( f − f b ) ( X ) K X ⟨ X ⟩ ∣ f b ] + P η ⟨ X ⟩ f , S_{\eta}(f) = \mathbb{E}_{X\sim\mu}[(f-f_b)(X)K_X^{\langle X\rangle}\mid f_b]+P_{\eta}^{\langle X\rangle}f, S η ( f ) = E X ∼ μ [( f − f b ) ( X ) K X ⟨ X ⟩ ∣ f b ] + P η ⟨ X ⟩ f , D S η ( f ) Δ f = E X ∼ μ [ Δ f ( X ) K X ⟨ X ⟩ ] + P η ⟨ X ⟩ Δ f . DS_{\eta}(f)\Delta f = \mathbb{E}_{X\sim\mu}[\Delta f(X)K_X^{\langle X\rangle}]+P_{\eta}^{\langle X\rangle}\Delta f. D S η ( f ) Δ f = E X ∼ μ [ Δ f ( X ) K X ⟨ X ⟩ ] + P η ⟨ X ⟩ Δ f . Lemma 5. Suppose that ε i b \varepsilon_i^b ε i b R ⟨ X ⟩ R^{\langle X\rangle} R ⟨ X ⟩ (16) , then
E [ ∥ f ^ n , η b − f b ∥ X ∣ f b 2 ] ≤ C 1 ( 1 + ∥ f b ∥ H X 2 ) n b − 2 β 2 β + 1 , \mathbb{E}\left[\|\widehat{f}_{n,\eta}^b-f_b\|_{X\mid f_b}^2\right]\le C_1(1+\|f_b\|_{\mathcal{H}_X}^2)n_b^{-\frac{2\beta}{2\beta+1}}, E [ ∥ f n , η b − f b ∥ X ∣ f b 2 ] ≤ C 1 ( 1 + ∥ f b ∥ H X 2 ) n b − 2 β + 1 2 β , where C 1 C_1 C 1 f b f_b f b
Proof of Lemma 5. For the subsequent analysis, we take η ≍ n b − 2 β 2 β + 1 \eta\asymp n_b^{-\frac{2\beta}{2\beta+1}} η ≍ n b − 2 β + 1 2 β f ∈ H X f\in\mathcal{H}_X f ∈ H X
⟨ D S η ( f b ) f , f ⟩ X = E X ∼ μ [ f ( X ) 2 ] + η ∥ f ∥ H X 2 . \langle DS_{\eta}(f_b)f,f\rangle_X=\mathbb{E}_{X\sim\mu}[f(X)^2]+\eta\|f\|_{\mathcal{H}_X}^2. ⟨ D S η ( f b ) f , f ⟩ X = E X ∼ μ [ f ( X ) 2 ] + η ∥ f ∥ H X 2 . So D S η ( f b ) = id DS_{\eta}(f_b)=\operatorname{id} D S η ( f b ) = id f b , η = f b − P η ⟨ X ⟩ f b f_{b,\eta}=f_b-P_{\eta}^{\langle X\rangle}f_b f b , η = f b − P η ⟨ X ⟩ f b S η ( f b , η ) = 0 S_{\eta}(f_{b,\eta})=0 S η ( f b , η ) = 0
Γ X ( f ) = n b − 1 ∑ i = 1 n b [ f ( X i b ) K X i b ⟨ X ⟩ − E X ∼ μ [ f ( X ) K X ⟨ X ⟩ ] ] . \Gamma_X(f)=n_b^{-1}\sum_{i=1}^{n_b}\left[f(X_i^b)K_{X_i^b}^{\langle X\rangle}-\mathbb{E}_{X\sim\mu}[f(X)K_X^{\langle X\rangle}]\right]. Γ X ( f ) = n b − 1 i = 1 ∑ n b [ f ( X i b ) K X i b ⟨ X ⟩ − E X ∼ μ [ f ( X ) K X ⟨ X ⟩ ] ] . By boundedness of the eigenfunctions of R ⟨ X ⟩ R^{\langle X\rangle} R ⟨ X ⟩
∥ K x ⟨ X ⟩ ∥ X 2 = K ⟨ X ⟩ ( x , x ) = ∑ k ≥ 1 ∣ φ k ⟨ X ⟩ ( x ) ∣ 2 1 + η ρ k ⟨ X ⟩ ≤ C φ ⟨ X ⟩ 2 ∑ k ≥ 1 1 1 + η ρ k ⟨ X ⟩ ≡ C φ ⟨ X ⟩ 2 g − 1 . (41) \|K_x^{\langle X\rangle}\|_X^2=K^{\langle X\rangle}(x,x)=\sum_{k\ge1}\frac{|\varphi_k^{\langle X\rangle}(x)|^2}{1+\eta\rho_k^{\langle X\rangle}}\le C_{\varphi^{\langle X\rangle}}^2\sum_{k\ge1}\frac{1}{1+\eta\rho_k^{\langle X\rangle}}\equiv C_{\varphi^{\langle X\rangle}}^2g^{-1}. \tag*{(41)} ∥ K x ⟨ X ⟩ ∥ X 2 = K ⟨ X ⟩ ( x , x ) = k ≥ 1 ∑ 1 + η ρ k ⟨ X ⟩ ∣ φ k ⟨ X ⟩ ( x ) ∣ 2 ≤ C φ ⟨ X ⟩ 2 k ≥ 1 ∑ 1 + η ρ k ⟨ X ⟩ 1 ≡ C φ ⟨ X ⟩ 2 g − 1 . (41) Since conditional on f b f_b f b Z 1 b , … , Z n b b Z_1^b,\ldots,Z_{n_b}^b Z 1 b , … , Z n b b ∥ f ( X i b ) K X i b ⟨ X ⟩ ∥ X ≤ C φ ⟨ X ⟩ g − 1 / 2 ∥ f ∥ ∞ \|f(X_i^b)K_{X_i^b}^{\langle X\rangle}\|_X\le C_{\varphi^{\langle X\rangle}}g^{-1/2}\|f\|_\infty ∥ f ( X i b ) K X i b ⟨ X ⟩ ∥ X ≤ C φ ⟨ X ⟩ g − 1/2 ∥ f ∥ ∞ [25 ] , for any f , f ′ ∈ H X f,f'\in\mathcal{H}_X f , f ′ ∈ H X
P ( ∥ n b g [ Γ X ( f ) − Γ X ( f ′ ) ] ∥ X ≥ t ) ≤ 2 e − t 2 8 C φ ⟨ X ⟩ 2 ∥ f − f ′ ∥ ∞ 2 , \mathbb{P}\left(\left\|\sqrt{n_bg}[\Gamma_X(f)-\Gamma_X(f')]\right\|_X\ge t\right)\le2e^{-\frac{t^2}{8C_{\varphi^{\langle X\rangle}}^2\|f-f'\|_\infty^2}}, P ( ∥ n b g [ Γ X ( f ) − Γ X ( f ′ )] ∥ X ≥ t ) ≤ 2 e − 8 C φ ⟨ X ⟩ 2 ∥ f − f ′ ∥ ∞ 2 t 2 , so that
∥ n b g ∥ Γ X ( f ) − Γ X ( f ′ ) ∥ X ∥ ψ 2 ≤ 3 C φ ⟨ X ⟩ ∥ f − f ′ ∥ ∞ . \left\|\sqrt{n_b g}\left\|\Gamma_X(f)-\Gamma_X(f')\right\|_X\right\|_{\psi_2}\leq3C_{\varphi\langle X\rangle}\left\|f-f'\right\|_\infty. ∥ n b g ∥ Γ X ( f ) − Γ X ( f ′ ) ∥ X ∥ ψ 2 ≤ 3 C φ ⟨ X ⟩ ∥ f − f ′ ∥ ∞ . Let H g = { f ∈ H X : ∥ f ∥ X ≤ C φ ⟨ X ⟩ g 1 / 2 } \mathcal{H}_g=\{f\in\mathcal{H}_X:\|f\|_X\leq C_{\varphi\langle X\rangle}g^{1/2}\} H g = { f ∈ H X : ∥ f ∥ X ≤ C φ ⟨ X ⟩ g 1/2 }
a g ≜ ∫ 0 1 ψ 2 − 1 ( D ( H g , ε , ∥ ⋅ ∥ ∞ ) ) d ε . a_g\triangleq\int_0^1\psi_2^{-1}\left(D(\mathcal{H}_g,\varepsilon,\|\cdot\|_\infty)\right)\,d\varepsilon. a g ≜ ∫ 0 1 ψ 2 − 1 ( D ( H g , ε , ∥ ⋅ ∥ ∞ ) ) d ε . By similar calculations in Example 4.1 of Shang et al. [31 ] , we have g ≍ η 1 / ( 2 β ) g\asymp\eta^{1/(2\beta)} g ≍ η 1/ ( 2 β ) a g = O ( ( g / η ) 1 / ( 4 β ) ) = O ( η − ( 2 β − 1 ) / ( 8 β 2 ) ) a_g=O((g/\eta)^{1/(4\beta)})=O(\eta^{-(2\beta-1)/(8\beta^2)}) a g = O (( g / η ) 1/ ( 4 β ) ) = O ( η − ( 2 β − 1 ) / ( 8 β 2 ) ) [17 ] , we have
∥ sup f ∈ H g n b g ∥ Γ X ( f ) ∥ X ∥ ψ 2 ≤ C a g , \left\|\sup_{f\in\mathcal{H}_g}\sqrt{n_b g}\|\Gamma_X(f)\|_X\right\|_{\psi_2}\leq Ca_g, f ∈ H g sup n b g ∥ Γ X ( f ) ∥ X ψ 2 ≤ C a g , where C C C C φ ⟨ X ⟩ C_{\varphi\langle X\rangle} C φ ⟨ X ⟩ ψ 2 \psi_2 ψ 2 f ^ b \widehat f_b f b t ≥ 0 t\geq0 t ≥ 0
P ( sup f ∈ H g n b g ∥ Γ X ( f ) ∥ X ≥ t ) ≤ 2 e − t 2 C 2 a g 2 . \mathbb{P}\left(\sup_{f\in\mathcal{H}_g}\sqrt{n_b g}\|\Gamma_X(f)\|_X\geq t\right)\leq2e^{-\frac{t^2}{C^2a_g^2}}. P ( f ∈ H g sup n b g ∥ Γ X ( f ) ∥ X ≥ t ) ≤ 2 e − C 2 a g 2 t 2 . Let
E = { sup f ∈ H g ∥ Γ X ( f ) ∥ X ≤ 1 2 C φ ⟨ X ⟩ g 1 / 2 } . \mathcal{E}=\left\{\sup_{f\in\mathcal{H}_g}\|\Gamma_X(f)\|_X\leq\frac{1}{2C_{\varphi\langle X\rangle}}g^{1/2}\right\}. E = { f ∈ H g sup ∥ Γ X ( f ) ∥ X ≤ 2 C φ ⟨ X ⟩ 1 g 1/2 } . Since n b g 2 / a g 2 ≳ n b η ( 6 β − 1 ) / ( 4 β 2 ) ≍ n b ( 2 β − 1 ) 2 / ( 2 β ( 2 β + 1 ) ) ⟶ ∞ n_b g^2/a_g^2\gtrsim n_b\eta^{(6\beta-1)/(4\beta^2)}\asymp n_b^{(2\beta-1)^2/(2\beta(2\beta+1))}\longrightarrow\infty n b g 2 / a g 2 ≳ n b η ( 6 β − 1 ) / ( 4 β 2 ) ≍ n b ( 2 β − 1 ) 2 / ( 2 β ( 2 β + 1 )) ⟶ ∞
P ( E c ) ≤ 2 e − n b g 2 C 2 C φ ⟨ X ⟩ 2 a g 2 ⟶ 0. \mathbb{P}(\mathcal{E}^c)\leq2e^{-\frac{n_b g^2}{C^2C_{\varphi\langle X\rangle}^2a_g^2}}\longrightarrow0. P ( E c ) ≤ 2 e − C 2 C φ ⟨ X ⟩ 2 a g 2 n b g 2 ⟶ 0. Let Δ f = f ^ b , η − f ^ b , η \Delta f=\widehat f_{b,\eta}-\widehat f_{b,\eta} Δ f = f b , η − f b , η S b , η ( f ^ b , η ) = 0 S_{b,\eta}(\widehat f_{b,\eta})=0 S b , η ( f b , η ) = 0 D 2 S b , η = 0 D^2S_{b,\eta}=0 D 2 S b , η = 0
0 = S b , η ( f ^ b , η ) = S b , η ( f b , η ) + D S b , η ( f b , η ) Δ f = S b , η ( f b , η ) + [ D S b , η ( f b , η ) − D S η ( f b , η ) ] Δ f + D S η ( f b , η ) Δ f = S b , η ( f b , η ) + Γ X ( Δ f ) + Δ f . \begin{aligned}
0&=S_{b,\eta}(\widehat f_{b,\eta})=S_{b,\eta}(f_{b,\eta})+DS_{b,\eta}(f_{b,\eta})\Delta f\\
&=S_{b,\eta}(f_{b,\eta})+[DS_{b,\eta}(f_{b,\eta})-DS_\eta(f_{b,\eta})]\Delta f+DS_\eta(f_{b,\eta})\Delta f\\
&=S_{b,\eta}(f_{b,\eta})+\Gamma_X(\Delta f)+\Delta f.
\end{aligned} 0 = S b , η ( f b , η ) = S b , η ( f b , η ) + D S b , η ( f b , η ) Δ f = S b , η ( f b , η ) + [ D S b , η ( f b , η ) − D S η ( f b , η )] Δ f + D S η ( f b , η ) Δ f = S b , η ( f b , η ) + Γ X ( Δ f ) + Δ f . On the event E \mathcal{E} E
∥ Γ X ( Δ f ) ∥ X = ∥ Γ X ( C φ ⟨ X ⟩ − 1 g 1 / 2 Δ f / ∥ Δ f ∥ X ) ∥ X × C φ ⟨ X ⟩ g − 1 / 2 ∥ Δ f ∥ X ≤ sup f ∈ H g ∥ Γ X ( f ) ∥ X C φ ⟨ X ⟩ g − 1 / 2 ∥ Δ f ∥ X ≤ 1 2 ∥ Δ f ∥ X , \begin{aligned}
\|\Gamma_X(\Delta f)\|_X
&=\left\|\Gamma_X\left(C_{\varphi\langle X\rangle}^{-1}g^{1/2}\Delta f/\|\Delta f\|_X\right)\right\|_X\times C_{\varphi\langle X\rangle}g^{-1/2}\|\Delta f\|_X\\
&\leq\sup_{f\in\mathcal{H}_g}\|\Gamma_X(f)\|_XC_{\varphi\langle X\rangle}g^{-1/2}\|\Delta f\|_X\leq\frac{1}{2}\|\Delta f\|_X,
\end{aligned} ∥ Γ X ( Δ f ) ∥ X = Γ X ( C φ ⟨ X ⟩ − 1 g 1/2 Δ f /∥Δ f ∥ X ) X × C φ ⟨ X ⟩ g − 1/2 ∥Δ f ∥ X ≤ f ∈ H g sup ∥ Γ X ( f ) ∥ X C φ ⟨ X ⟩ g − 1/2 ∥Δ f ∥ X ≤ 2 1 ∥Δ f ∥ X , which leads to
∥ f ^ b , η − f b , η ∥ X = ∥ Δ f ∥ X ≤ 2 ∥ S b , η ( f b , η ) ∥ X . (42) \|\widehat f_{b,\eta}-f_{b,\eta}\|_X=\|\Delta f\|_X\leq2\|S_{b,\eta}(f_{b,\eta})\|_X. \tag*{(42)} ∥ f b , η − f b , η ∥ X = ∥Δ f ∥ X ≤ 2∥ S b , η ( f b , η ) ∥ X . (42) Since S η ( f b , η ) = 0 S_\eta(f_{b,\eta})=0 S η ( f b , η ) = 0 f b , η − f b = − P η ⟨ X ⟩ f b f_{b,\eta}-f_b=-P_\eta^{\langle X\rangle}f_b f b , η − f b = − P η ⟨ X ⟩ f b f b f_b f b Z 1 b , … , Z n b Z_1^b,\ldots,Z_{n}^b Z 1 b , … , Z n b f b f_b f b
E [ ∥ f ^ b , η − f b , η ∥ X 2 ∣ E ∣ f b ] ≤ 4 E [ ∥ S b , η ( f b , η ) − S η ( f b , η ) ∥ X 2 ∣ f b ] = 8 E [ ∥ n b − 1 ∑ i = 1 n b ε i b K X i b ⟨ X ⟩ ∥ X 2 ∣ f b ] + 8 E [ ∥ Γ X ( P η ⟨ X ⟩ f b ) ∥ X 2 ∣ f b ] ; \begin{aligned}
\mathbb{E}\left[\left\|\widehat{f}_{b,\eta}-f_{b,\eta}\right\|_{X}^{2}\mathrel{\big|}\mathcal{E}\mathrel{\big|}f_b\right]
\le4\mathbb{E}\left[\left\|S_{b,\eta}(f_{b,\eta})-S_{\eta}(f_{b,\eta})\right\|_{X}^{2}\mathrel{\big|}f_b\right] \\
&=8\mathbb{E}\left[\left\|n_b^{-1}\sum_{i=1}^{n_b}\varepsilon_i^bK_{X_i^b}^{\langle X\rangle}\right\|_{X}^{2}\mathrel{\big|}f_b\right]
+8\mathbb{E}\left[\left\|\Gamma_X(P_\eta^{\langle X\rangle}f_b)\right\|_{X}^{2}\mathrel{\big|}f_b\right];
\end{aligned} E [ f b , η − f b , η X 2 E f b ] ≤ 4 E [ ∥ S b , η ( f b , η ) − S η ( f b , η ) ∥ X 2 f b ] = 8 E n b − 1 i = 1 ∑ n b ε i b K X i b ⟨ X ⟩ X 2 f b + 8 E [ Γ X ( P η ⟨ X ⟩ f b ) X 2 f b ] ; recalling ε i b = Y i b − f b ( X i b ) \varepsilon_i^b=Y_i^b-f_b(X_i^b) ε i b = Y i b − f b ( X i b ) (41) and independence of ( ε i b , X i b ) (\varepsilon_i^b,X_i^b) ( ε i b , X i b ) f b f_b f b
E [ ∥ n b − 1 ∑ i = 1 n b ε i b K X i b ⟨ X ⟩ ∥ X 2 ∣ f b ] = 1 n b 2 ∑ i = 1 n b E [ ∥ ε i b K X i b ⟨ X ⟩ ∥ X 2 ] ≤ C φ ⟨ X ⟩ σ ε 2 n b g , \mathbb{E}\left[\left\|n_b^{-1}\sum_{i=1}^{n_b}\varepsilon_i^bK_{X_i^b}^{\langle X\rangle}\right\|_{X}^{2}\mathrel{\big|}f_b\right]
=\frac{1}{n_b^2}\sum_{i=1}^{n_b}\mathbb{E}\left[\left\|\varepsilon_i^bK_{X_i^b}^{\langle X\rangle}\right\|_{X}^{2}\right]
\le\frac{C_{\varphi\langle X\rangle}\sigma_\varepsilon^2}{n_b\mathfrak{g}}, E n b − 1 i = 1 ∑ n b ε i b K X i b ⟨ X ⟩ X 2 f b = n b 2 1 i = 1 ∑ n b E [ ε i b K X i b ⟨ X ⟩ X 2 ] ≤ n b g C φ ⟨ X ⟩ σ ε 2 , and by (41) again,
E [ ∥ Γ X ( P η ⟨ X ⟩ f b ) ∥ X 2 ∣ f b ] = 1 n b 2 ∑ i = 1 n b ( E [ ∥ ( P η ⟨ X ⟩ f b ) ( X i b ) K X i b ⟨ X ⟩ ∥ X 2 ∣ f b ] − ∥ E X ∼ μ [ ( P η ⟨ X ⟩ f b ) ( X ) K X ⟨ X ⟩ ∣ f b ] ∥ X 2 ) ≤ n b − 1 E X ∼ μ [ ∥ ( P η ⟨ X ⟩ f b ) ( X ) K X ⟨ X ⟩ ∥ X 2 ∣ f b ] ≤ C φ ⟨ X ⟩ 2 n b g V X ( P η ⟨ X ⟩ f b , P η ⟨ X ⟩ f b ) ≤ C φ ⟨ X ⟩ 2 n b g ∥ P η ⟨ X ⟩ f b ∥ X 2 . \begin{aligned}
\mathbb{E}\left[\left\|\Gamma_X(P_\eta^{\langle X\rangle}f_b)\right\|_{X}^{2}\mathrel{\big|}f_b\right] \\
&=\frac{1}{n_b^2}\sum_{i=1}^{n_b}\left(\mathbb{E}\left[\left\|(P_\eta^{\langle X\rangle}f_b)(X_i^b)K_{X_i^b}^{\langle X\rangle}\right\|_{X}^{2}\mathrel{\big|}f_b\right]
-\left\|\mathbb{E}_{X\sim\mu}\left[(P_\eta^{\langle X\rangle}f_b)(X)K_X^{\langle X\rangle}\mathrel{\big|}f_b\right]\right\|_{X}^{2}\right) \\
&\le n_b^{-1}\mathbb{E}_{X\sim\mu}\left[\left\|(P_\eta^{\langle X\rangle}f_b)(X)K_X^{\langle X\rangle}\right\|_{X}^{2}\mathrel{\big|}f_b\right]
\le\frac{C_{\varphi\langle X\rangle}^{2}}{n_b\mathfrak{g}}V_X(P_\eta^{\langle X\rangle}f_b,P_\eta^{\langle X\rangle}f_b)
\le\frac{C_{\varphi\langle X\rangle}^{2}}{n_b\mathfrak{g}}\left\|P_\eta^{\langle X\rangle}f_b\right\|_{X}^{2}.
\end{aligned} E [ Γ X ( P η ⟨ X ⟩ f b ) X 2 f b ] = n b 2 1 i = 1 ∑ n b ( E [ ( P η ⟨ X ⟩ f b ) ( X i b ) K X i b ⟨ X ⟩ X 2 f b ] − E X ∼ μ [ ( P η ⟨ X ⟩ f b ) ( X ) K X ⟨ X ⟩ f b ] X 2 ) ≤ n b − 1 E X ∼ μ [ ( P η ⟨ X ⟩ f b ) ( X ) K X ⟨ X ⟩ X 2 f b ] ≤ n b g C φ ⟨ X ⟩ 2 V X ( P η ⟨ X ⟩ f b , P η ⟨ X ⟩ f b ) ≤ n b g C φ ⟨ X ⟩ 2 P η ⟨ X ⟩ f b X 2 . Obviously,
∥ P η ⟨ X ⟩ f b ∥ X = sup ∥ f ∥ X = 1 ⟨ P η ⟨ X ⟩ f b , f ⟩ X ≤ η 1 / 2 ∥ f b ∥ H X , \left\|P_\eta^{\langle X\rangle}f_b\right\|_{X}
=\sup_{\left\|f\right\|_{X}=1}\left\langle P_\eta^{\langle X\rangle}f_b,f\right\rangle_{X}
\le\eta^{1/2}\left\|f_b\right\|_{H_X}, P η ⟨ X ⟩ f b X = ∥ f ∥ X = 1 sup ⟨ P η ⟨ X ⟩ f b , f ⟩ X ≤ η 1/2 ∥ f b ∥ H X , so we have
E [ ∥ Γ X ( P η ⟨ X ⟩ f b ) ∥ X 2 ∣ f b ] ≤ C φ ⟨ X ⟩ 2 ∥ f b ∥ H X 2 η n b g . \mathbb{E}\left[\left\|\Gamma_X(P_\eta^{\langle X\rangle}f_b)\right\|_{X}^{2}\mathrel{\big|}f_b\right]
\le C_{\varphi\langle X\rangle}^{2}\left\|f_b\right\|_{H_X}^{2}\frac{\eta}{n_b\mathfrak{g}}. E [ Γ X ( P η ⟨ X ⟩ f b ) X 2 f b ] ≤ C φ ⟨ X ⟩ 2 ∥ f b ∥ H X 2 n b g η . Therefore,
E [ ∥ f ^ b , η − f b , η ∥ X 2 ∣ E ∣ f b ] ≤ 8 C φ ⟨ X ⟩ σ ε 2 n b g + 8 C φ ⟨ X ⟩ 2 ∥ f b ∥ H X 2 η n b g . \mathbb{E}\left[\left\|\widehat{f}_{b,\eta}-f_{b,\eta}\right\|_{X}^{2}\mathrel{\big|}\mathcal{E}\mathrel{\big|}f_b\right]
\le\frac{8C_{\varphi\langle X\rangle}\sigma_\varepsilon^2}{n_b\mathfrak{g}}
+\frac{8C_{\varphi\langle X\rangle}^{2}\left\|f_b\right\|_{H_X}^{2}\eta}{n_b\mathfrak{g}}. E [ f b , η − f b , η X 2 E f b ] ≤ n b g 8 C φ ⟨ X ⟩ σ ε 2 + n b g 8 C φ ⟨ X ⟩ 2 ∥ f b ∥ H X 2 η . Together with
∥ f b , η − f b ∥ X 2 = ∥ P η ⟨ X ⟩ f b ∥ X 2 ≤ ∥ f b ∥ H X 2 η , \left\|f_{b,\eta}-f_b\right\|_{X}^{2}
=\left\|P_\eta^{\langle X\rangle}f_b\right\|_{X}^{2}
\le\left\|f_b\right\|_{H_X}^{2}\eta, ∥ f b , η − f b ∥ X 2 = P η ⟨ X ⟩ f b X 2 ≤ ∥ f b ∥ H X 2 η , we have
E [ ∥ f ^ b , η − f b ∥ X 2 ∣ E ∣ f b ] ≤ 2 E [ ∥ f ^ b , η − f b , η ∥ X 2 ∣ E ∣ f b ] + 2 ∥ f b , η − f b ∥ X 2 ≤ 16 C φ ⟨ X ⟩ σ ε 2 n b g + 16 C φ ⟨ X ⟩ 2 ∥ f b ∥ H X 2 η n b g + 2 ∥ f b ∥ H X 2 η \begin{aligned}
\mathbb{E}\left[\left\|\widehat{f}_{b,\eta}-f_b\right\|_{X}^{2}\mathrel{\big|}\mathcal{E}\mathrel{\big|}f_b\right]
\le2\mathbb{E}\left[\left\|\widehat{f}_{b,\eta}-f_{b,\eta}\right\|_{X}^{2}\mathrel{\big|}\mathcal{E}\mathrel{\big|}f_b\right]
+2\left\|f_{b,\eta}-f_b\right\|_{X}^{2} \\
&\le\frac{16C_{\varphi\langle X\rangle}\sigma_\varepsilon^2}{n_b\mathfrak{g}}
+\frac{16C_{\varphi\langle X\rangle}^{2}\left\|f_b\right\|_{H_X}^{2}\eta}{n_b\mathfrak{g}}
+2\left\|f_b\right\|_{H_X}^{2}\eta
\end{aligned} E [ f b , η − f b X 2 E f b ] ≤ 2 E [ f b , η − f b , η X 2 E f b ] + 2 ∥ f b , η − f b ∥ X 2 ≤ n b g 16 C φ ⟨ X ⟩ σ ε 2 + n b g 16 C φ ⟨ X ⟩ 2 ∥ f b ∥ H X 2 η + 2 ∥ f b ∥ H X 2 η ≲ ( 1 + ∥ f b ∥ H X 2 ) n b − 2 β 2 β + 1 , (43) \lesssim\left(1+\left\|f_b\right\|_{H_X}^{2}\right)n_b^{-\frac{2\beta}{2\beta+1}}, \tag*{(43)} ≲ ( 1 + ∥ f b ∥ H X 2 ) n b − 2 β + 1 2 β , (43) where “≲ \lesssim ≲ f b f_b f b E [ ∥ f ^ b , η − f b ∥ X 2 ∣ E c ∣ f b ] \mathbb{E}\left[\left\|\widehat{f}_{b,\eta}-f_b\right\|_{X}^{2}\mathrel{\big|}\mathcal{E}^c\mathrel{\big|}f_b\right] E [ f b , η − f b X 2 E c f b ]
E [ ∥ f ^ b , η − f b ∥ X 2 ∣ E c ∣ f b ] ≤ ( E [ ∥ f ^ b , η − f b ∥ X 4 ∣ f b ] ) 1 / 2 P ( E c ) 1 / 2 , \mathbb{E}\left[\left\|\widehat{f}_{b,\eta}-f_b\right\|_{X}^{2}\mathrel{\big|}\mathcal{E}^c\mathrel{\big|}f_b\right]
\le\left(\mathbb{E}\left[\left\|\widehat{f}_{b,\eta}-f_b\right\|_{X}^{4}\mathrel{\big|}f_b\right]\right)^{1/2}\mathbb{P}(\mathcal{E}^c)^{1/2}, E [ f b , η − f b X 2 E c f b ] ≤ ( E [ f b , η − f b X 4 f b ] ) 1/2 P ( E c ) 1/2 , it is sufficient to control ∥ f ^ b , η − f b ∥ X 4 \|\widehat{f}_{b,\eta}-f_b\|_X^4 ∥ f b , η − f b ∥ X 4 f = ∑ k f k φ k ⟨ X ⟩ ∈ H X f=\sum_k f_k\varphi_k^{\langle X\rangle}\in H_X f = ∑ k f k φ k ⟨ X ⟩ ∈ H X
∥ f ∥ X 2 = ∑ k ( 1 + η ρ k ⟨ X ⟩ ) f k 2 ≤ ( 1 / ρ 1 ⟨ X ⟩ + η ) ∑ k f k 2 ρ k ⟨ X ⟩ = ( 1 / ρ 1 ⟨ X ⟩ + η ) ∥ f ∥ H X 2 ≡ C 4 ∥ f ∥ H X 2 . \|f\|_X^2=\sum_k(1+\eta\rho_k^{\langle X\rangle})f_k^2\leq(1/\rho_1^{\langle X\rangle}+\eta)\sum_k f_k^2\rho_k^{\langle X\rangle}=(1/\rho_1^{\langle X\rangle}+\eta)\|f\|_{H_X}^2\equiv C_4\|f\|_{H_X}^2. ∥ f ∥ X 2 = k ∑ ( 1 + η ρ k ⟨ X ⟩ ) f k 2 ≤ ( 1/ ρ 1 ⟨ X ⟩ + η ) k ∑ f k 2 ρ k ⟨ X ⟩ = ( 1/ ρ 1 ⟨ X ⟩ + η ) ∥ f ∥ H X 2 ≡ C 4 ∥ f ∥ H X 2 . Since
∥ f ^ b , η − f b ∥ X 4 ≤ C 4 2 ∥ f ^ b , η − f b ∥ H X 4 ≤ C 4 2 ( ∥ f ^ b , η ∥ H X + ∥ f b ∥ H X ) 4 \|\widehat{f}_{b,\eta}-f_b\|_X^4\leq C_4^2\|\widehat{f}_{b,\eta}-f_b\|_{H_X}^4\leq C_4^2\left(\|\widehat{f}_{b,\eta}\|_{H_X}+\|f_b\|_{H_X}\right)^4 ∥ f b , η − f b ∥ X 4 ≤ C 4 2 ∥ f b , η − f b ∥ H X 4 ≤ C 4 2 ( ∥ f b , η ∥ H X + ∥ f b ∥ H X ) 4 ≤ 16 C 4 2 ( ∥ f ^ b , η ∥ H X 4 + ∥ f b ∥ H X 4 ) . \leq16C_4^2\left(\|\widehat{f}_{b,\eta}\|_{H_X}^4+\|f_b\|_{H_X}^4\right). ≤ 16 C 4 2 ( ∥ f b , η ∥ H X 4 + ∥ f b ∥ H X 4 ) . Since ℓ b , η ( f ^ b , η ) ≤ ℓ b , η ( f b ) \ell_{b,\eta}(\widehat{f}_{b,\eta})\leq\ell_{b,\eta}(f_b) ℓ b , η ( f b , η ) ≤ ℓ b , η ( f b )
η 2 ∥ f ^ b , η ∥ H X 2 ≤ ℓ b , η ( f ^ b , η ) ≤ 1 2 n b ∑ i = 1 n b ∣ ε i b ∣ 2 + η 2 ∥ f b ∥ H X 2 , \frac{\eta}{2}\|\widehat{f}_{b,\eta}\|_{H_X}^2\leq\ell_{b,\eta}(\widehat{f}_{b,\eta})\leq\frac{1}{2n_b}\sum_{i=1}^{n_b}|\varepsilon_i^b|^2+\frac{\eta}{2}\|f_b\|_{H_X}^2, 2 η ∥ f b , η ∥ H X 2 ≤ ℓ b , η ( f b , η ) ≤ 2 n b 1 i = 1 ∑ n b ∣ ε i b ∣ 2 + 2 η ∥ f b ∥ H X 2 , together with Cauchy-Schwartz inequality,
∥ f ^ b , η ∥ H X 4 ≤ ( 1 n b η ∑ i = 1 n b ∣ ε i b ∣ 2 + ∥ f b ∥ H X 2 ) 2 ≤ 2 n b 2 η 2 ( ∑ i = 1 n b ∣ ε i b ∣ 2 ) 2 + 2 ∥ f b ∥ H X 4 ≤ 2 n b η 2 ∑ i = 1 n b ∣ ε i b ∣ 4 + 2 ∥ f b ∥ H X 4 . \begin{aligned}
\|\widehat{f}_{b,\eta}\|_{H_X}^4\leq\left(\frac{1}{n_b\eta}\sum_{i=1}^{n_b}|\varepsilon_i^b|^2+\|f_b\|_{H_X}^2\right)^2 \\
&\leq\frac{2}{n_b^2\eta^2}\left(\sum_{i=1}^{n_b}|\varepsilon_i^b|^2\right)^2+2\|f_b\|_{H_X}^4\leq\frac{2}{n_b\eta^2}\sum_{i=1}^{n_b}|\varepsilon_i^b|^4+2\|f_b\|_{H_X}^4.
\end{aligned} ∥ f b , η ∥ H X 4 ≤ ( n b η 1 i = 1 ∑ n b ∣ ε i b ∣ 2 + ∥ f b ∥ H X 2 ) 2 ≤ n b 2 η 2 2 ( i = 1 ∑ n b ∣ ε i b ∣ 2 ) 2 + 2∥ f b ∥ H X 4 ≤ n b η 2 2 i = 1 ∑ n b ∣ ε i b ∣ 4 + 2∥ f b ∥ H X 4 . So
∥ f ^ b , η − f b ∥ X 4 ≤ 16 C 4 2 ( 2 n b η 2 ∑ i = 1 n b ∣ ε i b ∣ 4 + 3 ∥ f b ∥ H X 4 ) , \|\widehat{f}_{b,\eta}-f_b\|_X^4\leq16C_4^2\left(\frac{2}{n_b\eta^2}\sum_{i=1}^{n_b}|\varepsilon_i^b|^4+3\|f_b\|_{H_X}^4\right), ∥ f b , η − f b ∥ X 4 ≤ 16 C 4 2 ( n b η 2 2 i = 1 ∑ n b ∣ ε i b ∣ 4 + 3∥ f b ∥ H X 4 ) , which together with finite fourth moment of ε i b \varepsilon_i^b ε i b
E [ ∥ f ^ b , η − f b ∥ X 4 ∣ f b ] ≤ 32 C 4 2 n b η 2 ∑ i = 1 n b E [ ∣ ε i b ∣ 4 ∣ f b ] + 48 C 4 2 ∥ f b ∥ H X 4 \mathbb{E}\left[\|\widehat{f}_{b,\eta}-f_b\|_X^4\mid f_b\right]\leq\frac{32C_4^2}{n_b\eta^2}\sum_{i=1}^{n_b}\mathbb{E}[|\varepsilon_i^b|^4\mid f_b]+48C_4^2\|f_b\|_{H_X}^4 E [ ∥ f b , η − f b ∥ X 4 ∣ f b ] ≤ n b η 2 32 C 4 2 i = 1 ∑ n b E [ ∣ ε i b ∣ 4 ∣ f b ] + 48 C 4 2 ∥ f b ∥ H X 4 ≤ 32 C 4 2 σ ε 4 η − 2 + 48 C 4 2 ∥ f b ∥ H X 4 . \leq32C_4^2\sigma_\varepsilon^4\eta^{-2}+48C_4^2\|f_b\|_{H_X}^4. ≤ 32 C 4 2 σ ε 4 η − 2 + 48 C 4 2 ∥ f b ∥ H X 4 . Combining above, we get that, as n → ∞ n\to\infty n → ∞
E [ ∥ f ^ b , η − f b ∥ X 2 1 ε ∣ f b ] ≤ ( 8 C 4 σ ε 2 η − 1 + 8 C 4 ∥ f b ∥ H X 2 ) 1 / 2 e − 2 − 1 C − 2 C γ − 2 n b ( 2 β − 1 ) 2 2 β ( 2 β + 1 ) ≤ ( 1 + ∥ f b ∥ H X 2 ) n b − 2 β 2 β + 1 , (44) \begin{aligned}
\mathbb{E}\left[\|\widehat{f}_{b,\eta}-f_b\|_X^2\mathbb{1}_{\varepsilon}\mid f_b\right]\leq\left(8C_4\sigma_\varepsilon^2\eta^{-1}+8C_4\|f_b\|_{H_X}^2\right)^{1/2}e^{-2^{-1}C^{-2}C_\gamma^{-2}n_b^{\frac{(2\beta-1)^2}{2\beta(2\beta+1)}}} \\
&\leq(1+\|f_b\|_{H_X}^2)n_b^{-\frac{2\beta}{2\beta+1}}, \tag*{(44)}
\end{aligned} E [ ∥ f b , η − f b ∥ X 2 1 ε ∣ f b ] ≤ ( 8 C 4 σ ε 2 η − 1 + 8 C 4 ∥ f b ∥ H X 2 ) 1/2 e − 2 − 1 C − 2 C γ − 2 n b 2 β ( 2 β + 1 ) ( 2 β − 1 ) 2 ≤ ( 1 + ∥ f b ∥ H X 2 ) n b − 2 β + 1 2 β , (44) where the last inequality follows from β > 1 / 2 \beta>1/2 β > 1/2 o ( n b − 2 β / ( 2 β + 1 ) ) o(n_b^{-2\beta/(2\beta+1)}) o ( n b − 2 β / ( 2 β + 1 ) ) (43) and (44) , we get that
E [ ∥ f ^ b , η − f b ∥ X 2 ∣ f b ] = E [ ∥ f ^ b , η − f b ∥ X 2 1 E ∣ f b ] + E [ ∥ f ^ b , η − f b ∥ X 2 1 E c ∣ f b ] \mathbb{E}\left[\|\widehat{f}_{b,\eta}-f_b\|_X^2\mid f_b\right]=\mathbb{E}\left[\|\widehat{f}_{b,\eta}-f_b\|_X^2\mathbb{1}_{\mathcal{E}}\mid f_b\right]+\mathbb{E}\left[\|\widehat{f}_{b,\eta}-f_b\|_X^2\mathbb{1}_{\mathcal{E}^c}\mid f_b\right] E [ ∥ f b , η − f b ∥ X 2 ∣ f b ] = E [ ∥ f b , η − f b ∥ X 2 1 E ∣ f b ] + E [ ∥ f b , η − f b ∥ X 2 1 E c ∣ f b ] ≲ ( 1 + ∥ f b ∥ H X 2 ) n b − 2 β 2 β + 1 , \lesssim(1+\|f_b\|_{H_X}^2)n_b^{-\frac{2\beta}{2\beta+1}}, ≲ ( 1 + ∥ f b ∥ H X 2 ) n b − 2 β + 1 2 β , where “≲ \lesssim ≲ f b f_b f b
Next we prove Lemma 2.
Proof of Lemma 2. Due to independence between X n b + 1 b X^{b}_{n_b+1} X n b + 1 b ( D n b b , f b ) (D^{b}_{n_b}, f_b) ( D n b b , f b )
E [ f b ( X n b + 1 b ) ∣ D n b b , X n b + 1 b ] = E [ f b ∣ D n b b , X n b + 1 b ] ( X n b + 1 b ) = E [ f b ∣ D n b b ] ( X n b + 1 b ) . \mathbb{E}[f_b(X^{b}_{n_b+1})\mid D^{b}_{n_b},X^{b}_{n_b+1}]
=\mathbb{E}[f_b\mid D^{b}_{n_b},X^{b}_{n_b+1}](X^{b}_{n_b+1})
=\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1}). E [ f b ( X n b + 1 b ) ∣ D n b b , X n b + 1 b ] = E [ f b ∣ D n b b , X n b + 1 b ] ( X n b + 1 b ) = E [ f b ∣ D n b b ] ( X n b + 1 b ) . Therefore,
E [ ∣ f ^ b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] = E [ ∣ f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] + E [ ∣ f ^ b , η ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] , \begin{aligned}
\mathbb{E}[|\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-f_b(X^{b}_{n_b+1})|^2\mid D^{b}_{n_b},X^{b}_{n_b+1}]
={}&\mathbb{E}[|f_b(X^{b}_{n_b+1})-\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1})|^2\mid D^{b}_{n_b},X^{b}_{n_b+1}]\\
&+\mathbb{E}[|\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1})|^2\mid D^{b}_{n_b},X^{b}_{n_b+1}],
\end{aligned} E [ ∣ f b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] = E [ ∣ f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] + E [ ∣ f b , η ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] , where the equality is due to the vanishing cross term:
E [ ( f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ) ( f ^ b , η ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ) ∣ D n b b , X n b + 1 b ] = E [ f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ D n b b , X n b + 1 b ] ( f ^ b , η ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ) = 0. \begin{aligned}
&\mathbb{E}[(f_b(X^{b}_{n_b+1})-\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1}))(\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1}))\mid D^{b}_{n_b},X^{b}_{n_b+1}]\\
={}&\mathbb{E}[f_b(X^{b}_{n_b+1})-\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1})\mid D^{b}_{n_b},X^{b}_{n_b+1}](\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1}))=0.
\end{aligned} = E [( f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b )) ( f b , η ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b )) ∣ D n b b , X n b + 1 b ] E [ f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ D n b b , X n b + 1 b ] ( f b , η ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b )) = 0. So we have
E [ ∣ f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] ≤ E [ ∣ f ^ b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] . \mathbb{E}[|f_b(X^{b}_{n_b+1})-\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1})|^2\mid D^{b}_{n_b},X^{b}_{n_b+1}]
\leq\mathbb{E}[|\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-f_b(X^{b}_{n_b+1})|^2\mid D^{b}_{n_b},X^{b}_{n_b+1}]. E [ ∣ f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] ≤ E [ ∣ f b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ∣ D n b b , X n b + 1 b ] . Taking expectations w.r.t. D n b b , X n b + 1 b D^{b}_{n_b},X^{b}_{n_b+1} D n b b , X n b + 1 b
V b ( n b ) 2 = E [ ∣ f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ 2 ] ≤ E [ ∣ f ^ b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ] . V_b(n_b)^2=\mathbb{E}[|f_b(X^{b}_{n_b+1})-\mathbb{E}[f_b\mid D^{b}_{n_b}](X^{b}_{n_b+1})|^2]\leq\mathbb{E}[|\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-f_b(X^{b}_{n_b+1})|^2]. V b ( n b ) 2 = E [ ∣ f b ( X n b + 1 b ) − E [ f b ∣ D n b b ] ( X n b + 1 b ) ∣ 2 ] ≤ E [ ∣ f b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ] . By Lemma 5, we have
E [ ∣ f ^ b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ∣ D n b b , f b ] = V X ( f ^ b , η − f b ) ≤ ∥ f ^ b , η − f b ∥ X 2 . \mathbb{E}[|\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-f_b(X^{b}_{n_b+1})|^2\mid D^{b}_{n_b},f_b]=V_X(\widehat{f}_{b,\eta}-f_b)\leq\|\widehat{f}_{b,\eta}-f_b\|_X^2. E [ ∣ f b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ∣ D n b b , f b ] = V X ( f b , η − f b ) ≤ ∥ f b , η − f b ∥ X 2 . Taking expectations w.r.t. D n b b ∣ f b D^{b}_{n_b}\mid f_b D n b b ∣ f b
E [ ∣ f ^ b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ∣ f b ] ≤ E [ ∥ f ^ b , η − f b ∥ X 2 ∣ f b ] ≤ C 1 ( 1 + ∥ f b ∥ X 2 ) n b − 2 β 2 β + 1 . \mathbb{E}[|\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-f_b(X^{b}_{n_b+1})|^2\mid f_b]\leq\mathbb{E}[\|\widehat{f}_{b,\eta}-f_b\|_X^2\mid f_b]\leq C_1(1+\|f_b\|_X^2)n_b^{-\frac{2\beta}{2\beta+1}}. E [ ∣ f b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ∣ f b ] ≤ E [ ∥ f b , η − f b ∥ X 2 ∣ f b ] ≤ C 1 ( 1 + ∥ f b ∥ X 2 ) n b − 2 β + 1 2 β . Taking expectations w.r.t. f b f_b f b
V b ( n b ) 2 ≤ E [ ∣ f ^ b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ] ≤ C 1 ( 1 + E [ ∥ f b ∥ X 2 ] ) n b − 2 β 2 β + 1 . V_b(n_b)^2\leq\mathbb{E}[|\widehat{f}_{b,\eta}(X^{b}_{n_b+1})-f_b(X^{b}_{n_b+1})|^2]\leq C_1(1+\mathbb{E}[\|f_b\|_X^2])n_b^{-\frac{2\beta}{2\beta+1}}. V b ( n b ) 2 ≤ E [ ∣ f b , η ( X n b + 1 b ) − f b ( X n b + 1 b ) ∣ 2 ] ≤ C 1 ( 1 + E [ ∥ f b ∥ X 2 ]) n b − 2 β + 1 2 β . Proof is completed.
Proof of Theorem 3. Let Δ M = M ^ n , N , B , λ − M ∗ \Delta M=\widehat{M}_{n,N,B,\lambda}-M_* Δ M = M n , N , B , λ − M ∗
S n , N , B , λ ( M ∗ + Δ M ) − S n , N , B , λ ( M ∗ ) − ( S ˉ n , B , λ ( M ∗ + Δ M ) − S ˉ n , B , λ ( M ∗ ) ) = D S n , N , B , λ ( M ∗ ) Δ M − D S ˉ n , B , λ ( M ∗ ) Δ M = 1 N ∑ b = 1 B ∑ i = 1 N b [ Δ M ( π ^ n b b , X n b + i b ) K ( π ^ n b b , X n b + i b ) − E [ Δ M ( π ^ n b b , X n b + i b ) K ( π ^ n b b , X n b + i b ) ] ] = Γ ( Δ M ) ≡ Γ 1 ( Δ M ) + Γ 2 ( Δ M ) . \begin{aligned}
&S_{n,N,B,\lambda}(M_*+\Delta M)-S_{n,N,B,\lambda}(M_*)-(\bar{S}_{n,B,\lambda}(M_*+\Delta M)-\bar{S}_{n,B,\lambda}(M_*))\\
&=DS_{n,N,B,\lambda}(M_*)\Delta M-D\bar{S}_{n,B,\lambda}(M_*)\Delta M\\
&=\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}\left[\Delta M(\widehat{\pi}^{b}_{n_b},X^{b}_{n_b+i})K(\widehat{\pi}^{b}_{n_b},X^{b}_{n_b+i})-\mathbb{E}\left[\Delta M(\widehat{\pi}^{b}_{n_b},X^{b}_{n_b+i})K(\widehat{\pi}^{b}_{n_b},X^{b}_{n_b+i})\right]\right]\\
&=\Gamma(\Delta M)\equiv\Gamma_1(\Delta M)+\Gamma_2(\Delta M).
\end{aligned} S n , N , B , λ ( M ∗ + Δ M ) − S n , N , B , λ ( M ∗ ) − ( S ˉ n , B , λ ( M ∗ + Δ M ) − S ˉ n , B , λ ( M ∗ )) = D S n , N , B , λ ( M ∗ ) Δ M − D S ˉ n , B , λ ( M ∗ ) Δ M = N 1 b = 1 ∑ B i = 1 ∑ N b [ Δ M ( π n b b , X n b + i b ) K ( π n b b , X n b + i b ) − E [ Δ M ( π n b b , X n b + i b ) K ( π n b b , X n b + i b ) ] ] = Γ ( Δ M ) ≡ Γ 1 ( Δ M ) + Γ 2 ( Δ M ) . By (31) , (33) , Theorem 2 and (17) , we have
∥ Γ ( Δ M ) ∥ ≤ ∥ Γ 1 ( Δ M ) ∥ + ∥ Γ 2 ( Δ M ) ∥ = O P ( ( a h ∑ b = 1 B N b 2 N h + a h N h 2 ) ( λ 1 / 2 + ( N h ) − 1 / 2 ) ) = O P ( a h ∑ b = 1 B N b 2 N h ( λ 1 / 2 + ( N h ) − 1 / 2 ) ) , \begin{aligned}
\lVert\Gamma(\Delta M)\rVert&\le\lVert\Gamma_1(\Delta M)\rVert+\lVert\Gamma_2(\Delta M)\rVert\\
&=O_{\mathbb{P}}\left(\left(\frac{a_h\sqrt{\sum_{b=1}^{B}N_b^2}}{Nh}+\frac{a_h}{\sqrt{Nh^2}}\right)\left(\lambda^{1/2}+(Nh)^{-1/2}\right)\right)\\
&=O_{\mathbb{P}}\left(\frac{a_h\sqrt{\sum_{b=1}^{B}N_b^2}}{Nh}\left(\lambda^{1/2}+(Nh)^{-1/2}\right)\right),
\end{aligned} ∥ Γ ( Δ M )∥ ≤ ∥ Γ 1 ( Δ M )∥ + ∥ Γ 2 ( Δ M )∥ = O P N h a h ∑ b = 1 B N b 2 + N h 2 a h ( λ 1/2 + ( N h ) − 1/2 ) = O P N h a h ∑ b = 1 B N b 2 ( λ 1/2 + ( N h ) − 1/2 ) , in which the last equality holds due to the trivial fact
a h ∑ b = 1 B N b 2 N h ≥ a h N h 2 . \frac{a_h\sqrt{\sum_{b=1}^{B}N_b^2}}{Nh}\geq\frac{a_h}{\sqrt{Nh^2}}. N h a h ∑ b = 1 B N b 2 ≥ N h 2 a h . Since S n , N , B , λ ( M ⋆ + Δ M ) = 0 S_{n,N,B,\lambda}(M_\star+\Delta M)=0 S n , N , B , λ ( M ⋆ + Δ M ) = 0 D S ˉ n , B , λ ( M ⋆ ) = id D\bar{S}_{n,B,\lambda}(M_\star)=\operatorname{id} D S ˉ n , B , λ ( M ⋆ ) = id
∥ M ^ n , N , B , λ − M ⋆ + S n , N , B , λ ( M ⋆ ) ∥ = O P ( a h ∑ b = 1 B N b 2 N h ( λ 1 / 2 + ( N h ) − 1 / 2 ) ) . \lVert\widehat{M}_{n,N,B,\lambda}-M_\star+S_{n,N,B,\lambda}(M_\star)\rVert=O_{\mathbb{P}}\left(\frac{a_h\sqrt{\sum_{b=1}^{B}N_b^2}}{Nh}\left(\lambda^{1/2}+(Nh)^{-1/2}\right)\right). ∥ M n , N , B , λ − M ⋆ + S n , N , B , λ ( M ⋆ )∥ = O P N h a h ∑ b = 1 B N b 2 ( λ 1/2 + ( N h ) − 1/2 ) . By Lemma 4 and reproducing property,
M ^ n , N , B , λ ( π ^ , x ) − M ⋆ ( π ^ , x ) + S n , N , B , λ ( M ⋆ ) ( π ^ , x ) = O P ( a h ∑ b = 1 B N b 2 N h 3 / 2 ( λ 1 / 2 + ( N h ) − 1 / 2 ) ) . \widehat{M}_{n,N,B,\lambda}(\hat{\pi},x)-M_\star(\hat{\pi},x)+S_{n,N,B,\lambda}(M_\star)(\hat{\pi},x)=O_{\mathbb{P}}\left(\frac{a_h\sqrt{\sum_{b=1}^{B}N_b^2}}{Nh^{3/2}}\left(\lambda^{1/2}+(Nh)^{-1/2}\right)\right). M n , N , B , λ ( π ^ , x ) − M ⋆ ( π ^ , x ) + S n , N , B , λ ( M ⋆ ) ( π ^ , x ) = O P N h 3/2 a h ∑ b = 1 B N b 2 ( λ 1/2 + ( N h ) − 1/2 ) . Since
S n , N , B , λ ( M ⋆ ) ( π ^ , x ) = − 1 N ∑ b = 1 B ∑ i = 1 N b e n b , i b K ( π ^ n b b , X n b + i b ) ( π ^ , x ) + ( P λ M ⋆ ) ( π ^ , x ) = − 1 N ∑ b = 1 B W b + ( P λ M ⋆ ) ( π ^ , x ) , \begin{aligned}
S_{n,N,B,\lambda}(M_\star)(\hat{\pi},x)
&=-\frac{1}{N}\sum_{b=1}^{B}\sum_{i=1}^{N_b}e_{n_b,i}^{b}K(\hat{\pi}_{n_b}^{b},X_{n_b+i}^{b})(\hat{\pi},x)+(P_\lambda M_\star)(\hat{\pi},x)\\
&=-\frac{1}{N}\sum_{b=1}^{B}W_b+(P_\lambda M_\star)(\hat{\pi},x),
\end{aligned} S n , N , B , λ ( M ⋆ ) ( π ^ , x ) = − N 1 b = 1 ∑ B i = 1 ∑ N b e n b , i b K ( π ^ n b b , X n b + i b ) ( π ^ , x ) + ( P λ M ⋆ ) ( π ^ , x ) = − N 1 b = 1 ∑ B W b + ( P λ M ⋆ ) ( π ^ , x ) , where W b = ∑ i = 1 N b e n b , i b K ( π ^ n b b , X n b + i b ) ( π ^ , x ) W_b=\sum_{i=1}^{N_b}e_{n_b,i}^{b}K(\hat{\pi}_{n_b}^{b},X_{n_b+i}^{b})(\hat{\pi},x) W b = ∑ i = 1 N b e n b , i b K ( π ^ n b b , X n b + i b ) ( π ^ , x ) W 1 , … , W B W_1,\ldots,W_B W 1 , … , W B E [ e n b , i b ∣ π ^ n b b , X n b + i b ] = 0 \mathbb{E}[e_{n_b,i}^{b}\mid\hat{\pi}_{n_b}^{b},X_{n_b+i}^{b}]=0 E [ e n b , i b ∣ π ^ n b b , X n b + i b ] = 0 E [ W b ] = 0 \mathbb{E}[W_b]=0 E [ W b ] = 0 s − 1 ∑ b = 1 B W b → d N ( 0 , 1 ) s^{-1}\sum_{b=1}^{B}W_b\xrightarrow{d}N(0,1) s − 1 ∑ b = 1 B W b d N ( 0 , 1 )
s 2 = E [ ∣ ∑ b = 1 B W b ∣ 2 ] = ∑ b = 1 B E [ W b 2 ] . s^2=\mathbb{E}\left[\left|\sum_{b=1}^{B}W_b\right|^2\right]=\sum_{b=1}^{B}\mathbb{E}[W_b^2]. s 2 = E b = 1 ∑ B W b 2 = b = 1 ∑ B E [ W b 2 ] . Next we identify the limit of s 2 s^2 s 2 b b b
E [ W b 2 ] = ∑ i = 1 N b E [ ∣ e n b , i b ∣ 2 K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) 2 ] + ∑ i ≠ j E [ e n b , i b e n b , j b K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) K ( ( π ^ n b b , X n b + j b ) , ( π ^ , x ) ) ] ≡ I b + I I b . \begin{aligned}
\mathbb{E}[W_b^2] &= \sum_{i=1}^{N_b}\mathbb{E}[|e_{n_b,i}^b|^2K((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x))^2] \\
&\quad+\sum_{i\ne j}\mathbb{E}[e_{n_b,i}^b e_{n_b,j}^bK((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x))K((\widehat{\pi}_{n_b}^b,X_{n_b+j}^b),(\widehat{\pi},x))]\equiv I_b+\mathrm{II}_b.
\end{aligned} E [ W b 2 ] = i = 1 ∑ N b E [ ∣ e n b , i b ∣ 2 K (( π n b b , X n b + i b ) , ( π , x ) ) 2 ] + i = j ∑ E [ e n b , i b e n b , j b K (( π n b b , X n b + i b ) , ( π , x )) K (( π n b b , X n b + j b ) , ( π , x ))] ≡ I b + II b . By (8) and
e n b , i b = Y n b + i b − M ∗ ( π ^ n b b , X n b + i b ) = ε i b + f b ( X n b + i b ) − E [ f b ∣ π ^ n b ] ( X n b + i b ) = ε i b + ( f b − H ∗ ) ( X n b + i b ) , e_{n_b,i}^b=Y_{n_b+i}^b-M_*(\widehat{\pi}_{n_b}^b,X_{n_b+i}^b)=\varepsilon_i^b+f_b(X_{n_b+i}^b)-\mathbb{E}[f_b|\widehat{\pi}_{n_b}](X_{n_b+i}^b)=\varepsilon_i^b+(f_b-H_*)(X_{n_b+i}^b), e n b , i b = Y n b + i b − M ∗ ( π n b b , X n b + i b ) = ε i b + f b ( X n b + i b ) − E [ f b ∣ π n b ] ( X n b + i b ) = ε i b + ( f b − H ∗ ) ( X n b + i b ) , we have
E [ ∣ e n b , i b ∣ 2 ∣ π ^ n b b , X n b + i b ] = E [ ∣ ε i b + ( f b − H ∗ ) ( X n b + i b ) ∣ 2 ∣ π ^ n b b , X n b + i b ] = σ ε 2 + E [ ∣ ( f b − H ∗ ) ( X n b + i b ) ∣ 2 ∣ π ^ n b b , X n b + i b ] , \begin{aligned}
\mathbb{E}[|e_{n_b,i}^b|^2|\widehat{\pi}_{n_b}^b,X_{n_b+i}^b] &= \mathbb{E}[|\varepsilon_i^b+(f_b-H_*)(X_{n_b+i}^b)|^2|\widehat{\pi}_{n_b}^b,X_{n_b+i}^b] \\
&=\sigma_\varepsilon^2+\mathbb{E}[|(f_b-H_*)(X_{n_b+i}^b)|^2|\widehat{\pi}_{n_b}^b,X_{n_b+i}^b],
\end{aligned} E [ ∣ e n b , i b ∣ 2 ∣ π n b b , X n b + i b ] = E [ ∣ ε i b + ( f b − H ∗ ) ( X n b + i b ) ∣ 2 ∣ π n b b , X n b + i b ] = σ ε 2 + E [ ∣ ( f b − H ∗ ) ( X n b + i b ) ∣ 2 ∣ π n b b , X n b + i b ] , where the last equality is because E [ ε i b ( f b − H ∗ ) ( X n b + i b ) ∣ π ^ n b b , X n b + i b , f b ] = 0 \mathbb{E}[\varepsilon_i^b(f_b-H_*)(X_{n_b+i}^b)|\widehat{\pi}_{n_b}^b,X_{n_b+i}^b,f_b]=0 E [ ε i b ( f b − H ∗ ) ( X n b + i b ) ∣ π n b b , X n b + i b , f b ] = 0 E [ ε i b ( f b − H ∗ ) ( X n b + i b ) ∣ π ^ n b b , X n b + i b ] = 0 \mathbb{E}[\varepsilon_i^b(f_b-H_*)(X_{n_b+i}^b)|\widehat{\pi}_{n_b}^b,X_{n_b+i}^b]=0 E [ ε i b ( f b − H ∗ ) ( X n b + i b ) ∣ π n b b , X n b + i b ] = 0 ∣ K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) ∣ ≤ C φ 2 h − 1 |K((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x))|\le C_\varphi^2h^{-1} ∣ K (( π n b b , X n b + i b ) , ( π , x )) ∣ ≤ C φ 2 h − 1
∑ b = 1 B I b = ∑ b = 1 B ∑ i = 1 N b E [ E [ ∣ e n b , i b ∣ 2 ∣ π ^ n b b , X n b + i b ] K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) 2 ] = ∑ b = 1 B ∑ i = 1 N b E [ ( σ ε 2 + E [ ∣ ( f b − H ∗ ) ( X n b + i b ) ∣ 2 ∣ π ^ n b b , X n b + i b ] ) K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) 2 ] = σ ε 2 ∑ b = 1 B N b E [ K ( ( π ^ n b b , X n b + 1 b ) , ( π ^ , x ) ) 2 ] + ∑ b = 1 B N b E [ ∣ ( f b − H ∗ ) ( X n b + 1 b ) ∣ 2 K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) 2 ] . \begin{aligned}
\sum_{b=1}^{B}I_b
&=\sum_{b=1}^{B}\sum_{i=1}^{N_b}\mathbb{E}[\mathbb{E}[|e_{n_b,i}^b|^2|\widehat{\pi}_{n_b}^b,X_{n_b+i}^b]K((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x))^2] \\
&=\sum_{b=1}^{B}\sum_{i=1}^{N_b}\mathbb{E}[(\sigma_\varepsilon^2+\mathbb{E}[|(f_b-H_*)(X_{n_b+i}^b)|^2|\widehat{\pi}_{n_b}^b,X_{n_b+i}^b])K((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x))^2] \\
&=\sigma_\varepsilon^2\sum_{b=1}^{B}N_b\mathbb{E}[K((\widehat{\pi}_{n_b}^b,X_{n_b+1}^b),(\widehat{\pi},x))^2]+\sum_{b=1}^{B}N_b\mathbb{E}[|(f_b-H_*)(X_{n_b+1}^b)|^2K((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x))^2].
\end{aligned} b = 1 ∑ B I b = b = 1 ∑ B i = 1 ∑ N b E [ E [ ∣ e n b , i b ∣ 2 ∣ π n b b , X n b + i b ] K (( π n b b , X n b + i b ) , ( π , x ) ) 2 ] = b = 1 ∑ B i = 1 ∑ N b E [( σ ε 2 + E [ ∣ ( f b − H ∗ ) ( X n b + i b ) ∣ 2 ∣ π n b b , X n b + i b ]) K (( π n b b , X n b + i b ) , ( π , x ) ) 2 ] = σ ε 2 b = 1 ∑ B N b E [ K (( π n b b , X n b + 1 b ) , ( π , x ) ) 2 ] + b = 1 ∑ B N b E [ ∣ ( f b − H ∗ ) ( X n b + 1 b ) ∣ 2 K (( π n b b , X n b + i b ) , ( π , x ) ) 2 ] . By (29), we have
∑ b = 1 B N b E [ K ( ( π ^ n b b , X n b + 1 b ) , ( π ^ , x ) ) 2 ] = ∑ b = 1 B N b E [ ( ∑ k = 1 ∞ φ k ( π ^ n b b , X n b + 1 b ) φ k ( π ^ , x ) 1 + λ ρ k ) 2 ] = ∑ k , k ′ = 1 ∞ N ( 1 + λ ρ k ) ( 1 + λ ρ k ′ ) 1 N ∑ b = 1 B N b E [ φ k ( π ^ n b b , X n b + 1 b ) φ k ′ ( π ^ n b b , X n b + 1 b ) ] φ k ( π ^ , x ) φ k ′ ( π ^ , x ) = ∑ k , k ′ = 1 ∞ N ( 1 + λ ρ k ) ( 1 + λ ρ k ′ ) 1 N V ( φ k , φ k ′ ) φ k ( π ^ , x ) φ k ′ ( π ^ , x ) = N ∑ k = 1 ∞ ∣ φ k ( π ^ , x ) ∣ 2 ( 1 + λ ρ k ) 2 = N h − 1 ∑ k = 1 ∞ h ∣ φ k ( π ^ , x ) ∣ 2 ( 1 + λ ρ k ) 2 . \begin{aligned}
\sum_{b=1}^{B}N_b\mathbb{E}[K((\widehat{\pi}_{n_b}^b,X_{n_b+1}^b),(\widehat{\pi},x))^2]
&=\sum_{b=1}^{B}N_b\mathbb{E}\left[\left(\sum_{k=1}^{\infty}\frac{\varphi_k(\widehat{\pi}_{n_b}^b,X_{n_b+1}^b)\varphi_k(\widehat{\pi},x)}{1+\lambda\rho_k}\right)^2\right] \\
&=\sum_{k,k'=1}^{\infty}\frac{N}{(1+\lambda\rho_k)(1+\lambda\rho_{k'})}\frac{1}{N}\sum_{b=1}^{B}N_b\mathbb{E}[\varphi_k(\widehat{\pi}_{n_b}^b,X_{n_b+1}^b)\varphi_{k'}(\widehat{\pi}_{n_b}^b,X_{n_b+1}^b)]\varphi_k(\widehat{\pi},x)\varphi_{k'}(\widehat{\pi},x) \\
&=\sum_{k,k'=1}^{\infty}\frac{N}{(1+\lambda\rho_k)(1+\lambda\rho_{k'})}\frac{1}{N}V(\varphi_k,\varphi_{k'})\varphi_k(\widehat{\pi},x)\varphi_{k'}(\widehat{\pi},x) \\
&=N\sum_{k=1}^{\infty}\frac{|\varphi_k(\widehat{\pi},x)|^2}{(1+\lambda\rho_k)^2}=Nh^{-1}\sum_{k=1}^{\infty}\frac{h|\varphi_k(\widehat{\pi},x)|^2}{(1+\lambda\rho_k)^2}.
\end{aligned} b = 1 ∑ B N b E [ K (( π n b b , X n b + 1 b ) , ( π , x ) ) 2 ] = b = 1 ∑ B N b E ( k = 1 ∑ ∞ 1 + λ ρ k φ k ( π n b b , X n b + 1 b ) φ k ( π , x ) ) 2 = k , k ′ = 1 ∑ ∞ ( 1 + λ ρ k ) ( 1 + λ ρ k ′ ) N N 1 b = 1 ∑ B N b E [ φ k ( π n b b , X n b + 1 b ) φ k ′ ( π n b b , X n b + 1 b )] φ k ( π , x ) φ k ′ ( π , x ) = k , k ′ = 1 ∑ ∞ ( 1 + λ ρ k ) ( 1 + λ ρ k ′ ) N N 1 V ( φ k , φ k ′ ) φ k ( π , x ) φ k ′ ( π , x ) = N k = 1 ∑ ∞ ( 1 + λ ρ k ) 2 ∣ φ k ( π , x ) ∣ 2 = N h − 1 k = 1 ∑ ∞ ( 1 + λ ρ k ) 2 h ∣ φ k ( π , x ) ∣ 2 . Proofs in Section 3.3
Meanwhile,
∑ b = 1 B N b E [ ∣ ( f b − H ∗ ) ( X n b + 1 b ) ∣ 2 K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) 2 ] ≤ C φ 4 h − 2 ∑ b = 1 B N b V b ( n b ) 2 . \sum_{b=1}^{B} N_b \mathbb{E}\left[\left|(f_b-H_*)(X_{n_b+1}^b)\right|^2 K\left((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x)\right)^2\right] \le C_\varphi^4 h^{-2}\sum_{b=1}^{B}N_bV_b(n_b)^2. b = 1 ∑ B N b E [ ( f b − H ∗ ) ( X n b + 1 b ) 2 K ( ( π n b b , X n b + i b ) , ( π , x ) ) 2 ] ≤ C φ 4 h − 2 b = 1 ∑ B N b V b ( n b ) 2 . Therefore,
∑ b = 1 B I b = σ ε 2 N h − 1 ∑ k = 1 ∞ h ∣ φ k ( π ^ , x ) ∣ 2 ( 1 + λ ρ k ) 2 + O ( h − 2 ∑ b = 1 B N b V b ( n b ) 2 ) . (45) \sum_{b=1}^{B} I_b=\sigma_\varepsilon^2Nh^{-1}\sum_{k=1}^{\infty}\frac{h|\varphi_k(\widehat{\pi},x)|^2}{(1+\lambda\rho_k)^2}+O\left(h^{-2}\sum_{b=1}^{B}N_bV_b(n_b)^2\right). \tag*{(45)} b = 1 ∑ B I b = σ ε 2 N h − 1 k = 1 ∑ ∞ ( 1 + λ ρ k ) 2 h ∣ φ k ( π , x ) ∣ 2 + O ( h − 2 b = 1 ∑ B N b V b ( n b ) 2 ) . (45) To deal with I I b II_b I I b i ≠ j i\ne j i = j ε i \varepsilon_i ε i ε j \varepsilon_j ε j D n b b D_{n_b}^b D n b b X n b + i X_{n_b+i} X n b + i X n b + j X_{n_b+j} X n b + j f b f_b f b
E [ ε n b , i b ε n b , j b ∣ π ^ n b b , X n b + i b , X n b + j , f b ] = E [ ( ε i b + ( f b − H ∗ ) ( X n b + i b ) ) ( ε j b + ( f b − H ∗ ) ( X n b + j b ) ) ∣ π ^ n b b , X n b + i b , X n b + j , f b ] = ( f b − H ∗ ) ( X n b + i b ) ( f b − H ∗ ) ( X n b + j b ) . \begin{aligned}
\mathbb{E}\left[\varepsilon_{n_b,i}^b\varepsilon_{n_b,j}^b\mid\widehat{\pi}_{n_b}^b,X_{n_b+i}^b,X_{n_b+j},f_b\right]
&=\mathbb{E}\left[(\varepsilon_i^b+(f_b-H_*)(X_{n_b+i}^b))(\varepsilon_j^b+(f_b-H_*)(X_{n_b+j}^b))\mid\widehat{\pi}_{n_b}^b,X_{n_b+i}^b,X_{n_b+j},f_b\right]\\
&=(f_b-H_*)(X_{n_b+i}^b)(f_b-H_*)(X_{n_b+j}^b).
\end{aligned} E [ ε n b , i b ε n b , j b ∣ π n b b , X n b + i b , X n b + j , f b ] = E [ ( ε i b + ( f b − H ∗ ) ( X n b + i b )) ( ε j b + ( f b − H ∗ ) ( X n b + j b )) ∣ π n b b , X n b + i b , X n b + j , f b ] = ( f b − H ∗ ) ( X n b + i b ) ( f b − H ∗ ) ( X n b + j b ) . By Lemma 4 4 4
∣ E [ ε n b , i b ε n b , j b K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) K ( ( π ^ n b b , X n b + j b ) , ( π ^ , x ) ) ] ∣ = ∣ E [ E [ ε n b , i b ε n b , j b ∣ π ^ n b b , X n b + i b , X n b + j , f b ] K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) K ( ( π ^ n b b , X n b + j b ) , ( π ^ , x ) ) ] ∣ = ∣ E [ ( f b − H ∗ ) ( X n b + i b ) ( f b − H ∗ ) ( X n b + j b ) K ( ( π ^ n b b , X n b + i b ) , ( π ^ , x ) ) K ( ( π ^ n b b , X n b + j b ) , ( π ^ , x ) ) ] ∣ ≤ C φ 4 h − 2 E [ ∣ ( f b − H ∗ ) ( X n b + i b ) ( f b − H ∗ ) ( X n b + j b ) ∣ ] ≤ C φ 2 h − 2 V b ( n b ) 2 . \begin{aligned}
\left|\mathbb{E}\left[\varepsilon_{n_b,i}^b\varepsilon_{n_b,j}^bK\left((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x)\right)K\left((\widehat{\pi}_{n_b}^b,X_{n_b+j}^b),(\widehat{\pi},x)\right)\right]\right|\\
=\left|\mathbb{E}\left[\mathbb{E}\left[\varepsilon_{n_b,i}^b\varepsilon_{n_b,j}^b\mid\widehat{\pi}_{n_b}^b,X_{n_b+i}^b,X_{n_b+j},f_b\right]K\left((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x)\right)K\left((\widehat{\pi}_{n_b}^b,X_{n_b+j}^b),(\widehat{\pi},x)\right)\right]\right|\\
=\left|\mathbb{E}\left[(f_b-H_*)(X_{n_b+i}^b)(f_b-H_*)(X_{n_b+j}^b)K\left((\widehat{\pi}_{n_b}^b,X_{n_b+i}^b),(\widehat{\pi},x)\right)K\left((\widehat{\pi}_{n_b}^b,X_{n_b+j}^b),(\widehat{\pi},x)\right)\right]\right|\\
\le C_\varphi^4h^{-2}\mathbb{E}\left[|(f_b-H_*)(X_{n_b+i}^b)(f_b-H_*)(X_{n_b+j}^b)|\right]\le C_\varphi^2h^{-2}V_b(n_b)^2.
\end{aligned} E [ ε n b , i b ε n b , j b K ( ( π n b b , X n b + i b ) , ( π , x ) ) K ( ( π n b b , X n b + j b ) , ( π , x ) ) ] = E [ E [ ε n b , i b ε n b , j b ∣ π n b b , X n b + i b , X n b + j , f b ] K ( ( π n b b , X n b + i b ) , ( π , x ) ) K ( ( π n b b , X n b + j b ) , ( π , x ) ) ] = E [ ( f b − H ∗ ) ( X n b + i b ) ( f b − H ∗ ) ( X n b + j b ) K ( ( π n b b , X n b + i b ) , ( π , x ) ) K ( ( π n b b , X n b + j b ) , ( π , x ) ) ] ≤ C φ 4 h − 2 E [ ∣ ( f b − H ∗ ) ( X n b + i b ) ( f b − H ∗ ) ( X n b + j b ) ∣ ] ≤ C φ 2 h − 2 V b ( n b ) 2 . Therefore,
∑ b = 1 B I I b = O ( h − 2 ∑ b = 1 B N b 2 V b ( n b ) 2 ) . (46) \sum_{b=1}^{B}II_b=O\left(h^{-2}\sum_{b=1}^{B}N_b^2V_b(n_b)^2\right). \tag*{(46)} b = 1 ∑ B I I b = O ( h − 2 b = 1 ∑ B N b 2 V b ( n b ) 2 ) . (46) Combining (45) and (46) , and Rate Conditions (18) and (21) , we have
s 2 = ∑ b = 1 B I b + ∑ b = 1 B I I b = σ ε 2 N h − 1 ∑ k = 1 ∞ h ∣ φ k ( π ^ , x ) ∣ 2 ( 1 + λ ρ k ) 2 + O ( h − 2 ∑ b = 1 B N b 2 V b ( n b ) 2 ) = σ ε 2 σ ( π ^ , x ) 2 N h − 1 ( 1 + o ( 1 ) ) . \begin{aligned}
s^2&=\sum_{b=1}^{B}I_b+\sum_{b=1}^{B}II_b=\sigma_\varepsilon^2Nh^{-1}\sum_{k=1}^{\infty}\frac{h|\varphi_k(\widehat{\pi},x)|^2}{(1+\lambda\rho_k)^2}+O\left(h^{-2}\sum_{b=1}^{B}N_b^2V_b(n_b)^2\right)\\
&=\sigma_\varepsilon^2\sigma_{(\widehat{\pi},x)}^2Nh^{-1}(1+o(1)).
\end{aligned} s 2 = b = 1 ∑ B I b + b = 1 ∑ B I I b = σ ε 2 N h − 1 k = 1 ∑ ∞ ( 1 + λ ρ k ) 2 h ∣ φ k ( π , x ) ∣ 2 + O ( h − 2 b = 1 ∑ B N b 2 V b ( n b ) 2 ) = σ ε 2 σ ( π , x ) 2 N h − 1 ( 1 + o ( 1 )) . By Conditions (19) , (20) , we have
N h σ ε σ ( π ^ , x ) ( M n , N , B , λ ( π ^ , x ) − M ∗ ( π ^ , x ) ) = s − 1 ∑ b = 1 B W b + N s − 1 ( P λ M ∗ ) ( π ^ , x ) + O P ( a h ∑ b = 1 B N b 2 N h ( λ 1 / 2 + ( N h ) − 1 / 2 ) ) → d N ( 0 , 1 ) . \frac{\sqrt{Nh}}{\sigma_\varepsilon\sigma_{(\widehat{\pi},x)}}\left(M_{n,N,B,\lambda}(\widehat{\pi},x)-M_*(\widehat{\pi},x)\right)
=s^{-1}\sum_{b=1}^{B}W_b+Ns^{-1}(P_\lambda M_*)(\widehat{\pi},x)+O_P\left(\frac{a_h\sqrt{\sum_{b=1}^{B}N_b^2}}{\sqrt{Nh}}\left(\lambda^{1/2}+(Nh)^{-1/2}\right)\right)\xrightarrow{d}N(0,1). σ ε σ ( π , x ) N h ( M n , N , B , λ ( π , x ) − M ∗ ( π , x ) ) = s − 1 b = 1 ∑ B W b + N s − 1 ( P λ M ∗ ) ( π , x ) + O P N h a h ∑ b = 1 B N b 2 ( λ 1/2 + ( N h ) − 1/2 ) d N ( 0 , 1 ) . Proof is completed.
Proofs in Section 3.3
Proof of Lemma 3. Let us first prove H ∗ ∈ H Π H_* \in H_\Pi H ∗ ∈ H Π τ = C X ∥ Γ ∗ ∥ o p / 2 \tau= C_X\|\Gamma_*\|_{\mathrm{op}}/2 τ = C X ∥ Γ ∗ ∥ op /2 c = σ ε 2 η − 1 + τ c = \sigma_\varepsilon^2\eta^{-1}+\tau c = σ ε 2 η − 1 + τ 0 ⪯ J π ⪯ 2 τ id 0 \preceq J_\pi\preceq2\tau\operatorname{id} 0 ⪯ J π ⪯ 2 τ id ∥ J π − τ id ∥ o p ≤ τ \|J_\pi-\tau\operatorname{id}\|_{\mathrm{op}}\leq\tau ∥ J π − τ id ∥ op ≤ τ
Γ ∗ 1 / 2 T π G π = c − 1 Γ ∗ 1 / 2 ( id + J π − τ id c ) − 1 G π = c − 1 ∑ l = 0 ∞ ( − c − 1 ) l Γ ∗ 1 / 2 ( J π − τ id ) l G π = c − 1 ∑ l = 0 ∞ ( − c − 1 ) l ∑ j = 0 l ( l j ) ( − τ ) l − j J ∗ 1 / 2 ( J π ) j G π . (47) \begin{aligned}
\Gamma_*^{1/2}T_\pi G_\pi= c^{-1}\Gamma_*^{1/2}\left(\operatorname{id}+\frac{J_\pi-\tau\operatorname{id}}{c}\right)^{-1}G_\pi= c^{-1}\sum_{l=0}^{\infty}(-c^{-1})^l\Gamma_*^{1/2}(J_\pi-\tau\operatorname{id})^lG_\pi\\
&= c^{-1}\sum_{l=0}^{\infty}(-c^{-1})^l\sum_{j=0}^{l}\binom{l}{j}(-\tau)^{l-j}J_*^{1/2}(J_\pi)^jG_\pi. \tag*{(47)}
\end{aligned} Γ ∗ 1/2 T π G π = c − 1 Γ ∗ 1/2 ( id + c J π − τ id ) − 1 G π = c − 1 l = 0 ∑ ∞ ( − c − 1 ) l Γ ∗ 1/2 ( J π − τ id ) l G π = c − 1 l = 0 ∑ ∞ ( − c − 1 ) l j = 0 ∑ l ( j l ) ( − τ ) l − j J ∗ 1/2 ( J π ) j G π . (47) Let t x = Γ ∗ 1 / 2 R x ⟨ X ⟩ t_x=\Gamma_*^{1/2}R_x^{\langle X\rangle} t x = Γ ∗ 1/2 R x ⟨ X ⟩ j ≥ 1 j\geq1 j ≥ 1
Γ ∗ 1 / 2 ( J π ) j G π = ∫ Z ∫ X j ( y − f ∗ ( x ) ) Γ ∗ 1 / 2 ( t x 1 ⊗ t x 1 ) ⋯ ( t x j ⊗ t x j ) t x d π X ( x 1 ) ⋯ d π X ( x j ) d π ( z ) = ∫ Z ∫ X j ( y − f ∗ ( x ) ) C ∗ ( x 1 , x 2 ) ⋯ C ∗ ( x j − 1 , x j ) C ∗ ( x j , x ) Γ ∗ R x ⟨ X ⟩ d π X ( x 1 ) ⋯ d π X ( x j ) d π ( z ) , \begin{aligned}
\Gamma_*^{1/2}(J_\pi)^jG_\pi\\
&=\int_{\mathcal{Z}}\int_{\mathcal{X}^j}(y-f_*(x))\Gamma_*^{1/2}(t_{x_1}\otimes t_{x_1})\cdots(t_{x_j}\otimes t_{x_j})t_x\,\mathrm{d}\pi^X(x_1)\cdots\mathrm{d}\pi^X(x_j)\mathrm{d}\pi(z) \\
&=\int_{\mathcal{Z}}\int_{\mathcal{X}^j}(y-f_*(x))C_*(x_1,x_2)\cdots C_*(x_{j-1},x_j)C_*(x_j,x)\Gamma_*R_{x}^{\langle X\rangle}\,\mathrm{d}\pi^X(x_1)\cdots\mathrm{d}\pi^X(x_j)\mathrm{d}\pi(z),
\end{aligned} Γ ∗ 1/2 ( J π ) j G π = ∫ Z ∫ X j ( y − f ∗ ( x )) Γ ∗ 1/2 ( t x 1 ⊗ t x 1 ) ⋯ ( t x j ⊗ t x j ) t x d π X ( x 1 ) ⋯ d π X ( x j ) d π ( z ) = ∫ Z ∫ X j ( y − f ∗ ( x )) C ∗ ( x 1 , x 2 ) ⋯ C ∗ ( x j − 1 , x j ) C ∗ ( x j , x ) Γ ∗ R x ⟨ X ⟩ d π X ( x 1 ) ⋯ d π X ( x j ) d π ( z ) , and
Γ ∗ 1 / 2 ( J π ) 0 G π = ∫ Z ( y − f ∗ ( x ) ) Γ ∗ R x ⟨ X ⟩ d π ( z ) . \Gamma_*^{1/2}(J_\pi)^0G_\pi=\int_{\mathcal{Z}}(y-f_*(x))\Gamma_*R_x^{\langle X\rangle}\,\mathrm{d}\pi(z). Γ ∗ 1/2 ( J π ) 0 G π = ∫ Z ( y − f ∗ ( x )) Γ ∗ R x ⟨ X ⟩ d π ( z ) . Let W = ∑ k = 1 ∞ w k φ k ⟨ X ⟩ W=\sum_{k=1}^{\infty}w_k\varphi_k^{\langle X\rangle} W = ∑ k = 1 ∞ w k φ k ⟨ X ⟩ C ∗ ( x , x ′ ) = ∑ k , l = 1 ∞ a k , l φ k ⟨ X ⟩ ( x ) φ l ⟨ X ⟩ ( x ′ ) C_*(x,x')=\sum_{k,l=1}^{\infty}a_{k,l}\varphi_k^{\langle X\rangle}(x)\varphi_l^{\langle X\rangle}(x') C ∗ ( x , x ′ ) = ∑ k , l = 1 ∞ a k , l φ k ⟨ X ⟩ ( x ) φ l ⟨ X ⟩ ( x ′ ) a k , l = E [ w k w l ] a_{k,l}=\mathbb{E}[w_kw_l] a k , l = E [ w k w l ] j ≥ 0 j\geq0 j ≥ 0
Γ ∗ 1 / 2 ( J π ) j G π = ∑ k 0 , l 0 , … , k j , l j a k 0 , l 0 ⋯ a k j , l j Φ l 0 , k 1 X ( π ) ⋯ Φ l j − 1 , k j X ( π ) Φ l j Z ( π ) φ k 0 ⟨ X ⟩ , \Gamma_*^{1/2}(J_\pi)^jG_\pi=\sum_{k_0,l_0,\ldots,k_j,l_j}a_{k_0,l_0}\cdots a_{k_j,l_j}\Phi^X_{l_0,k_1}(\pi)\cdots\Phi^X_{l_{j-1},k_j}(\pi)\Phi^Z_{l_j}(\pi)\varphi_{k_0}^{\langle X\rangle}, Γ ∗ 1/2 ( J π ) j G π = k 0 , l 0 , … , k j , l j ∑ a k 0 , l 0 ⋯ a k j , l j Φ l 0 , k 1 X ( π ) ⋯ Φ l j − 1 , k j X ( π ) Φ l j Z ( π ) φ k 0 ⟨ X ⟩ , where ψ l j − 1 , k j ( x ) = φ l j − 1 ⟨ X ⟩ ( x ) φ k j ⟨ X ⟩ ( x ) \psi_{l_{j-1},k_j}(x)=\varphi_{l_{j-1}}^{\langle X\rangle}(x)\varphi_{k_j}^{\langle X\rangle}(x) ψ l j − 1 , k j ( x ) = φ l j − 1 ⟨ X ⟩ ( x ) φ k j ⟨ X ⟩ ( x )
Φ l j − 1 , k j X ( π ) = ∫ X ψ l j − 1 , k j ( x ) d π X ( x ) , Φ l j Z ( π ) = ∫ Z ( y − f ∗ ( x ) ) φ l j ⟨ X ⟩ ( x ) d π ( x , y ) . \Phi^X_{l_{j-1},k_j}(\pi)=\int_{\mathcal{X}}\psi_{l_{j-1},k_j}(x)\,\mathrm{d}\pi^X(x),\qquad\Phi^Z_{l_j}(\pi)=\int_{\mathcal{Z}}(y-f_*(x))\varphi_{l_j}^{\langle X\rangle}(x)\,\mathrm{d}\pi(x,y). Φ l j − 1 , k j X ( π ) = ∫ X ψ l j − 1 , k j ( x ) d π X ( x ) , Φ l j Z ( π ) = ∫ Z ( y − f ∗ ( x )) φ l j ⟨ X ⟩ ( x ) d π ( x , y ) . Let H Z H_Z H Z Z \mathcal{Z} Z R ⟨ Z ⟩ R^{\langle Z\rangle} R ⟨ Z ⟩ ⟨ ⋅ , ⋅ ⟩ H Z \langle\cdot,\cdot\rangle_{H_Z} ⟨ ⋅ , ⋅ ⟩ H Z
Q X = { Φ f X : f ∈ H X } , Q Z = { Φ g Z : g ∈ H Z } , Q^X=\{\Phi_f^X:f\in H_X\},\qquad Q^Z=\{\Phi_g^Z:g\in H_Z\}, Q X = { Φ f X : f ∈ H X } , Q Z = { Φ g Z : g ∈ H Z } , where Φ f X ( π ) = ∫ X f ( x ) d π X ( x ) \Phi_f^X(\pi)=\int_{\mathcal{X}}f(x)\,\mathrm{d}\pi^X(x) Φ f X ( π ) = ∫ X f ( x ) d π X ( x ) Φ g Z ( π ) = ∫ Z g ( z ) d π ( z ) \Phi_g^Z(\pi)=\int_{\mathcal{Z}}g(z)\,\mathrm{d}\pi(z) Φ g Z ( π ) = ∫ Z g ( z ) d π ( z ) π ∈ P ( Z ) \pi\in\mathcal{P}(\mathcal{Z}) π ∈ P ( Z ) Q X Q^X Q X Q Z Q^Z Q Z R ⟨ Π , X ⟩ R^{\langle\Pi,X\rangle} R ⟨ Π , X ⟩ R ⟨ Π , Z ⟩ R^{\langle\Pi,Z\rangle} R ⟨ Π , Z ⟩ (26) . The corresponding inner products are
⟨ Φ f X , Φ f ′ X ⟩ Q X = ⟨ f , f ′ ⟩ H X , ⟨ Φ g Z , Φ g ′ Z ⟩ Q Z = ⟨ g , g ′ ⟩ H Z . \langle\Phi_f^X,\Phi_{f'}^X\rangle_{Q^X}=\langle f,f'\rangle_{H_X},\qquad\langle\Phi_g^Z,\Phi_{g'}^Z\rangle_{Q^Z}=\langle g,g'\rangle_{H_Z}. ⟨ Φ f X , Φ f ′ X ⟩ Q X = ⟨ f , f ′ ⟩ H X , ⟨ Φ g Z , Φ g ′ Z ⟩ Q Z = ⟨ g , g ′ ⟩ H Z . Meanwhile, let H Π ( j ) H_\Pi^{(j)} H Π ( j ) P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) H X H_X H X
P ( Z ) × P ( Z ) ∋ ( π , π ′ ) ↦ [ c ε R ⟨ Π , X ⟩ ( π , π ′ ) ] j R ⟨ Π , Z ⟩ ( π , π ′ ) ⋅ id H X , \mathcal{P}(\mathcal{Z})\times\mathcal{P}(\mathcal{Z})\ni(\pi,\pi')\mapsto\left[c_\varepsilon R^{\langle\Pi,X\rangle}(\pi,\pi')\right]^jR^{\langle\Pi,Z\rangle}(\pi,\pi')\cdot\operatorname{id}_{H_X}, P ( Z ) × P ( Z ) ∋ ( π , π ′ ) ↦ [ c ε R ⟨ Π , X ⟩ ( π , π ′ ) ] j R ⟨ Π , Z ⟩ ( π , π ′ ) ⋅ id H X , which contains the linear span of the following mappings:
P ( Z ) ∋ π ↦ Φ f 1 X ( π ) ⋯ Φ f j X ( π ) Φ g Z ( π ) φ k ⟨ X ⟩ , f 1 , … , f j ∈ H X , g ∈ H Z , k ≥ 1. \mathcal{P}(\mathcal{Z}) \ni\pi\mapsto\Phi^{X}_{f_1}(\pi)\cdots\Phi^{X}_{f_j}(\pi)\Phi^{Z}_{g}(\pi)\varphi^{\langle X\rangle}_{k},\qquad f_1,\ldots,f_j\in\mathcal{H}_X,\ g\in\mathcal{H}_Z,\ k\geq1. P ( Z ) ∋ π ↦ Φ f 1 X ( π ) ⋯ Φ f j X ( π ) Φ g Z ( π ) φ k ⟨ X ⟩ , f 1 , … , f j ∈ H X , g ∈ H Z , k ≥ 1. The last assertion is due to the RKHS H ‾ Π ⟨ j ⟩ \overline{\mathcal{H}}_{\Pi}^{\langle j\rangle} H Π ⟨ j ⟩ P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z )
P ( Z ) × P ( Z ) ∋ ( π , π ′ ) ↦ [ c ε R ⟨ Π , X ⟩ ( π , π ′ ) ] j R ⟨ Π , Z ⟩ ( π , π ′ ) , \mathcal{P}(\mathcal{Z})\times\mathcal{P}(\mathcal{Z})\ni(\pi,\pi')\mapsto\left[c_{\varepsilon}R^{\langle\Pi,X\rangle}(\pi,\pi')\right]^jR^{\langle\Pi,Z\rangle}(\pi,\pi'), P ( Z ) × P ( Z ) ∋ ( π , π ′ ) ↦ [ c ε R ⟨ Π , X ⟩ ( π , π ′ ) ] j R ⟨ Π , Z ⟩ ( π , π ′ ) , which contains [ Q X ] j Q Z [Q^X]^jQ^Z [ Q X ] j Q Z π ↦ Φ l 0 , k 1 X ( π ) ⋯ Φ l j − 1 , k j X ( π ) Φ l j Z ( π ) \pi\mapsto\Phi^{X}_{l_0,k_1}(\pi)\cdots\Phi^{X}_{l_{j-1},k_j}(\pi)\Phi^{Z}_{l_j}(\pi) π ↦ Φ l 0 , k 1 X ( π ) ⋯ Φ l j − 1 , k j X ( π ) Φ l j Z ( π ) [ Q X ] j Q Z [Q^X]^jQ^Z [ Q X ] j Q Z π ↦ Φ k 0 , l 0 , … , k j , l j ( π ) ≜ Φ l 0 , k 1 X ( π ) ⋯ Φ l j − 1 , k j X ( π ) Φ l j Z ( π ) φ k 0 ⟨ X ⟩ \pi\mapsto\Phi_{k_0,l_0,\ldots,k_j,l_j}(\pi)\triangleq\Phi^{X}_{l_0,k_1}(\pi)\cdots\Phi^{X}_{l_{j-1},k_j}(\pi)\Phi^{Z}_{l_j}(\pi)\varphi^{\langle X\rangle}_{k_0} π ↦ Φ k 0 , l 0 , … , k j , l j ( π ) ≜ Φ l 0 , k 1 X ( π ) ⋯ Φ l j − 1 , k j X ( π ) Φ l j Z ( π ) φ k 0 ⟨ X ⟩ P ( Z ) \mathcal{P}(\mathcal{Z}) P ( Z ) H X \mathcal{H}_X H X H Π ⟨ j ⟩ \mathcal{H}_{\Pi}^{\langle j\rangle} H Π ⟨ j ⟩
∥ Γ ∗ 1 / 2 ( J π ) j G π ∥ H Π ⟨ j ⟩ ≤ ∑ k 0 , l 0 , … , k j , l j ∣ a k 0 , l 0 ⋯ a k j , l j ∣ × ∥ Φ k 0 , l 0 , … , k j , l j ∥ H Π ⟨ j ⟩ . \left\|\Gamma_*^{1/2}(J_\pi)^jG_\pi\right\|_{\mathcal{H}_{\Pi}^{\langle j\rangle}}
\leq
\sum_{k_0,l_0,\ldots,k_j,l_j}
\left|a_{k_0,l_0}\cdots a_{k_j,l_j}\right|
\times
\left\|\Phi_{k_0,l_0,\ldots,k_j,l_j}\right\|_{\mathcal{H}_{\Pi}^{\langle j\rangle}}. Γ ∗ 1/2 ( J π ) j G π H Π ⟨ j ⟩ ≤ k 0 , l 0 , … , k j , l j ∑ a k 0 , l 0 ⋯ a k j , l j × Φ k 0 , l 0 , … , k j , l j H Π ⟨ j ⟩ . Let ψ l j ( z ) = ( y − f ∗ ( x ) ) φ l j ⟨ X ⟩ ( x ) \psi_{l_j}(z)=(y-f_*(x))\varphi^{\langle X\rangle}_{l_j}(x) ψ l j ( z ) = ( y − f ∗ ( x )) φ l j ⟨ X ⟩ ( x ) H Z \mathcal{H}_Z H Z x ↦ f ∗ ( x ) φ l j ⟨ X ⟩ ( x ) x\mapsto f_*(x)\varphi^{\langle X\rangle}_{l_j}(x) x ↦ f ∗ ( x ) φ l j ⟨ X ⟩ ( x ) H X \mathcal{H}_X H X (22) ; ( x , y ) ↦ y f ∗ ( x ) (x,y)\mapsto yf_*(x) ( x , y ) ↦ y f ∗ ( x ) Z \mathcal{Z} Z y y ′ R ⟨ X ⟩ ( x , x ′ ) yy'R^{\langle X\rangle}(x,x') y y ′ R ⟨ X ⟩ ( x , x ′ )
∥ ψ l j ∥ H Z ≤ ∥ φ l j ⟨ X ⟩ ∥ H X + ∥ f ∗ φ l j ⟨ X ⟩ ∥ H X ≤ ( 1 + C d ∥ f ∗ ∥ H X ) ∥ φ l j ⟨ X ⟩ ∥ H X = ( 1 + C d ∥ f ∗ ∥ H X ) ( ρ l j ⟨ X ⟩ ) 1 / 2 . \|\psi_{l_j}\|_{\mathcal{H}_Z}\leq\|\varphi^{\langle X\rangle}_{l_j}\|_{\mathcal{H}_X}+\|f_*\varphi^{\langle X\rangle}_{l_j}\|_{\mathcal{H}_X}\leq(1+C_d\|f_*\|_{\mathcal{H}_X})\|\varphi^{\langle X\rangle}_{l_j}\|_{\mathcal{H}_X}=(1+C_d\|f_*\|_{\mathcal{H}_X})(\rho^{\langle X\rangle}_{l_j})^{1/2}. ∥ ψ l j ∥ H Z ≤ ∥ φ l j ⟨ X ⟩ ∥ H X + ∥ f ∗ φ l j ⟨ X ⟩ ∥ H X ≤ ( 1 + C d ∥ f ∗ ∥ H X ) ∥ φ l j ⟨ X ⟩ ∥ H X = ( 1 + C d ∥ f ∗ ∥ H X ) ( ρ l j ⟨ X ⟩ ) 1/2 . By (22) and definition of the H Π ⟨ j ⟩ \mathcal{H}_{\Pi}^{\langle j\rangle} H Π ⟨ j ⟩
∥ Φ k 0 , l 0 , … , k j , l j ∥ H Π ⟨ j ⟩ 2 = c ε j ∥ Φ l 0 , k 1 X ∥ Q X 2 ⋯ ∥ Φ l j − 1 , k j X ∥ Q X 2 ∥ Φ l j Z ∥ Q Z 2 ∥ φ k 0 ⟨ X ⟩ ∥ H X 2 = c ε j ∥ ψ l 0 , k 1 ∥ H X 2 ⋯ ∥ ψ l j − 1 , k j ∥ H X 2 ∥ ψ l j ∥ H Z 2 ∥ φ k 0 ⟨ X ⟩ ∥ H X 2 ≤ c ε j C d 2 ∥ φ l 0 ⟨ X ⟩ ∥ H X 2 ∥ φ k 1 ⟨ X ⟩ ∥ H X 2 ⋯ C d 2 ∥ φ l j − 1 ⟨ X ⟩ ∥ H X 2 ∥ φ k j ⟨ X ⟩ ∥ H X 2 ( 1 + C d ∥ f ∗ ∥ H X ) 2 ∥ φ l j ⟨ X ⟩ ∥ H X 2 ∥ φ k 0 ⟨ X ⟩ ∥ H X 2 = c ε j C d 2 j ( 1 + C d ∥ f ∗ ∥ H X ) 2 ρ k 0 ⟨ X ⟩ ρ l 0 ⟨ X ⟩ ⋯ ρ k j ⟨ X ⟩ ρ l j ⟨ X ⟩ . \begin{aligned}
\|\Phi_{k_0,l_0,\ldots,k_j,l_j}\|_{\mathcal{H}_{\Pi}^{\langle j\rangle}}^2
&=c_{\varepsilon}^j\|\Phi^X_{l_0,k_1}\|_{Q^X}^2\cdots\|\Phi^X_{l_{j-1},k_j}\|_{Q^X}^2\|\Phi^Z_{l_j}\|_{Q^Z}^2\|\varphi^{\langle X\rangle}_{k_0}\|_{\mathcal{H}_X}^2\\
&=c_{\varepsilon}^j\|\psi_{l_0,k_1}\|_{\mathcal{H}_X}^2\cdots\|\psi_{l_{j-1},k_j}\|_{\mathcal{H}_X}^2\|\psi_{l_j}\|_{\mathcal{H}_Z}^2\|\varphi^{\langle X\rangle}_{k_0}\|_{\mathcal{H}_X}^2\\
&\leq c_{\varepsilon}^jC_d^2\|\varphi^{\langle X\rangle}_{l_0}\|_{\mathcal{H}_X}^2\|\varphi^{\langle X\rangle}_{k_1}\|_{\mathcal{H}_X}^2\cdots C_d^2\|\varphi^{\langle X\rangle}_{l_{j-1}}\|_{\mathcal{H}_X}^2\|\varphi^{\langle X\rangle}_{k_j}\|_{\mathcal{H}_X}^2(1+C_d\|f_*\|_{\mathcal{H}_X})^2\|\varphi^{\langle X\rangle}_{l_j}\|_{\mathcal{H}_X}^2\|\varphi^{\langle X\rangle}_{k_0}\|_{\mathcal{H}_X}^2\\
&=c_{\varepsilon}^jC_d^{2j}(1+C_d\|f_*\|_{\mathcal{H}_X})^2\rho^{\langle X\rangle}_{k_0}\rho^{\langle X\rangle}_{l_0}\cdots\rho^{\langle X\rangle}_{k_j}\rho^{\langle X\rangle}_{l_j}.
\end{aligned} ∥ Φ k 0 , l 0 , … , k j , l j ∥ H Π ⟨ j ⟩ 2 = c ε j ∥ Φ l 0 , k 1 X ∥ Q X 2 ⋯ ∥ Φ l j − 1 , k j X ∥ Q X 2 ∥ Φ l j Z ∥ Q Z 2 ∥ φ k 0 ⟨ X ⟩ ∥ H X 2 = c ε j ∥ ψ l 0 , k 1 ∥ H X 2 ⋯ ∥ ψ l j − 1 , k j ∥ H X 2 ∥ ψ l j ∥ H Z 2 ∥ φ k 0 ⟨ X ⟩ ∥ H X 2 ≤ c ε j C d 2 ∥ φ l 0 ⟨ X ⟩ ∥ H X 2 ∥ φ k 1 ⟨ X ⟩ ∥ H X 2 ⋯ C d 2 ∥ φ l j − 1 ⟨ X ⟩ ∥ H X 2 ∥ φ k j ⟨ X ⟩ ∥ H X 2 ( 1 + C d ∥ f ∗ ∥ H X ) 2 ∥ φ l j ⟨ X ⟩ ∥ H X 2 ∥ φ k 0 ⟨ X ⟩ ∥ H X 2 = c ε j C d 2 j ( 1 + C d ∥ f ∗ ∥ H X ) 2 ρ k 0 ⟨ X ⟩ ρ l 0 ⟨ X ⟩ ⋯ ρ k j ⟨ X ⟩ ρ l j ⟨ X ⟩ . Consequently,
∥ Γ ∗ 1 / 2 ( J π ) j G π ∥ H Π ⟨ j ⟩ ≤ c ε j / 2 C d j ( 1 + C d ∥ f ∗ ∥ H X ) ∑ k 0 , l 0 , … , k j , l j ∣ a k 0 , l 0 ⋯ a k j , l j ∣ ( ρ k 0 ⟨ X ⟩ ρ l 0 ⟨ X ⟩ ⋯ ρ k j ⟨ X ⟩ ρ l j ⟨ X ⟩ ) 1 / 2 = c ε j / 2 C d j ( 1 + C d ∥ f ∗ ∥ H X ) ( ∑ k , l ≥ 1 ∣ a k , l ∣ ( ρ k ⟨ X ⟩ ρ l ⟨ X ⟩ ) 1 / 2 ) j + 1 . \begin{aligned}
\left\|\Gamma_*^{1/2}(J_\pi)^jG_\pi\right\|_{\mathcal{H}_{\Pi}^{\langle j\rangle}}
&\leq c_{\varepsilon}^{j/2}C_d^j(1+C_d\|f_*\|_{\mathcal{H}_X})
\sum_{k_0,l_0,\ldots,k_j,l_j}|a_{k_0,l_0}\cdots a_{k_j,l_j}|
\left(\rho^{\langle X\rangle}_{k_0}\rho^{\langle X\rangle}_{l_0}\cdots\rho^{\langle X\rangle}_{k_j}\rho^{\langle X\rangle}_{l_j}\right)^{1/2}\\
&=c_{\varepsilon}^{j/2}C_d^j(1+C_d\|f_*\|_{\mathcal{H}_X})
\left(\sum_{k,l\geq1}|a_{k,l}|(\rho^{\langle X\rangle}_{k}\rho^{\langle X\rangle}_{l})^{1/2}\right)^{j+1}.
\end{aligned} Γ ∗ 1/2 ( J π ) j G π H Π ⟨ j ⟩ ≤ c ε j /2 C d j ( 1 + C d ∥ f ∗ ∥ H X ) k 0 , l 0 , … , k j , l j ∑ ∣ a k 0 , l 0 ⋯ a k j , l j ∣ ( ρ k 0 ⟨ X ⟩ ρ l 0 ⟨ X ⟩ ⋯ ρ k j ⟨ X ⟩ ρ l j ⟨ X ⟩ ) 1/2 = c ε j /2 C d j ( 1 + C d ∥ f ∗ ∥ H X ) k , l ≥ 1 ∑ ∣ a k , l ∣ ( ρ k ⟨ X ⟩ ρ l ⟨ X ⟩ ) 1/2 j + 1 . By (28) , we have
∑ k , l ≥ 1 ∣ a k , l ∣ ( ρ k ⟨ X ⟩ ρ l ⟨ X ⟩ ) 1 / 2 ≤ ∑ k , l ≥ 1 ( a k , k a l , l ) 1 / 2 ( ρ k ⟨ X ⟩ ρ l ⟨ X ⟩ ) 1 / 2 = ( ∑ k ≥ 1 ( a k , k ρ k ⟨ X ⟩ ) 1 / 2 ) 2 < ∞ . \sum_{k,l\geq1}|a_{k,l}|(\rho^{\langle X\rangle}_{k}\rho^{\langle X\rangle}_{l})^{1/2}
\leq
\sum_{k,l\geq1}(a_{k,k}a_{l,l})^{1/2}(\rho^{\langle X\rangle}_{k}\rho^{\langle X\rangle}_{l})^{1/2}
=
\left(\sum_{k\geq1}(a_{k,k}\rho^{\langle X\rangle}_{k})^{1/2}\right)^2
<\infty. k , l ≥ 1 ∑ ∣ a k , l ∣ ( ρ k ⟨ X ⟩ ρ l ⟨ X ⟩ ) 1/2 ≤ k , l ≥ 1 ∑ ( a k , k a l , l ) 1/2 ( ρ k ⟨ X ⟩ ρ l ⟨ X ⟩ ) 1/2 = ( k ≥ 1 ∑ ( a k , k ρ k ⟨ X ⟩ ) 1/2 ) 2 < ∞. So π ↦ Γ ∗ 1 / 2 ( J π ) j G π \pi\mapsto\Gamma_*^{1/2}(J_\pi)^jG_\pi π ↦ Γ ∗ 1/2 ( J π ) j G π H Π ( j ) \mathcal{H}_{\Pi}^{(j)} H Π ( j ) (47) , this implies that the mapping π ↦ Γ ∗ 1 / 2 ( J π − τ id ) l G π \pi\mapsto\Gamma_*^{1/2}(J_\pi-\tau\operatorname{id})^lG_\pi π ↦ Γ ∗ 1/2 ( J π − τ id ) l G π ⨁ j = 0 ∞ H Π ( j ) \bigoplus_{j=0}^{\infty}\mathcal{H}_{\Pi}^{(j)} ⨁ j = 0 ∞ H Π ( j ) (47) again, the mapping π ↦ f ∗ + Γ ∗ 1 / 2 T π G π \pi\mapsto f_*+\Gamma_*^{1/2}T_\pi G_\pi π ↦ f ∗ + Γ ∗ 1/2 T π G π H Π = H X ⊕ ( ⨁ j = 0 ∞ H Π ( j ) ) \mathcal{H}_{\Pi}=\mathcal{H}_X\oplus\left(\bigoplus_{j=0}^{\infty}\mathcal{H}_{\Pi}^{(j)}\right) H Π = H X ⊕ ( ⨁ j = 0 ∞ H Π ( j ) ) H Π \mathcal{H}_{\Pi} H Π
R ( Π ) ( π , π ′ ) = id H X + ∑ j = 0 ∞ [ c ϵ R ( Π , X ) ( π , π ′ ) ] j R ( Π , Z ) ( π , π ′ ) ⋅ id H X = ( 1 + R ( Π , Z ) ( π , π ′ ) 1 − c ϵ R ( Π , X ) ( π , π ′ ) ) ⋅ id H X . \begin{aligned}
R^{(\Pi)}(\pi,\pi')=\operatorname{id}_{\mathcal{H}_X}+\sum_{j=0}^{\infty}\left[c_{\epsilon}R^{(\Pi,X)}(\pi,\pi')\right]^jR^{(\Pi,Z)}(\pi,\pi')\cdot\operatorname{id}_{\mathcal{H}_X} \\
&=\left(1+\frac{R^{(\Pi,Z)}(\pi,\pi')}{1-c_{\epsilon}R^{(\Pi,X)}(\pi,\pi')}\right)\cdot\operatorname{id}_{\mathcal{H}_X}.
\end{aligned} R ( Π ) ( π , π ′ ) = id H X + j = 0 ∑ ∞ [ c ϵ R ( Π , X ) ( π , π ′ ) ] j R ( Π , Z ) ( π , π ′ ) ⋅ id H X = ( 1 + 1 − c ϵ R ( Π , X ) ( π , π ′ ) R ( Π , Z ) ( π , π ′ ) ) ⋅ id H X . Next, let us show that M ∗ M_* M ∗ H ∗ \mathcal{H}_* H ∗
f ∗ = ( f ∗ ( X 1 ) , … , f ∗ ( X n ) ) ⊤ , Y = ( Y 1 , … , Y n ) ⊤ , f_*=(f_*(X_1),\ldots,f_*(X_n))^\top,\quad\mathbf{Y}=(Y_1,\ldots,Y_n)^\top, f ∗ = ( f ∗ ( X 1 ) , … , f ∗ ( X n ) ) ⊤ , Y = ( Y 1 , … , Y n ) ⊤ , C x = ( C ∗ ( X 1 , x ) , … , C ∗ ( X n , x ) ) ⊤ , C n × n = [ C ∗ ( X i , X j ) ] i , j = 1 n . \mathbf{C}_x=(C_*(X_1,x),\ldots,C_*(X_n,x))^\top,\quad C_{n\times n}=[C_*(X_i,X_j)]_{i,j=1}^{n}. C x = ( C ∗ ( X 1 , x ) , … , C ∗ ( X n , x ) ) ⊤ , C n × n = [ C ∗ ( X i , X j ) ] i , j = 1 n . By (1) , we have
( f ( x ) Y ) ∣ X 1 , … , X n ∼ N ( ( f ∗ ( x ) f ∗ ) , ( C ∗ ( x , x ) C x ⊤ C x C n × n + σ ϵ 2 I n × n ) ) . \left.\begin{pmatrix}f(x)\\\mathbf{Y}\end{pmatrix}\right|X_1,\ldots,X_n\sim\mathcal{N}\left(\begin{pmatrix}f_*(x)\\\mathbf{f}_*\end{pmatrix},\begin{pmatrix}C_*(x,x)&\mathbf{C}_x^\top\\\mathbf{C}_x&C_{n\times n}+\sigma_\epsilon^2I_{n\times n}\end{pmatrix}\right). ( f ( x ) Y ) X 1 , … , X n ∼ N ( ( f ∗ ( x ) f ∗ ) , ( C ∗ ( x , x ) C x C x ⊤ C n × n + σ ϵ 2 I n × n ) ) . By conditional normal formula,
f ( x ) ∣ D n ∼ N ( f ∗ ( x ) + C x ⊤ ( C n × n + σ ϵ 2 I n × n ) − 1 ( Y − f ∗ ) , C ∗ ( x , x ) − C x ⊤ ( C n × n + σ ϵ 2 I n × n ) − 1 C x ) . f(x)\mid D_n\sim\mathcal{N}\left(f_*(x)+\mathbf{C}_x^\top(C_{n\times n}+\sigma_\epsilon^2I_{n\times n})^{-1}(\mathbf{Y}-\mathbf{f}_*),\ C_*(x,x)-\mathbf{C}_x^\top(C_{n\times n}+\sigma_\epsilon^2I_{n\times n})^{-1}\mathbf{C}_x\right). f ( x ) ∣ D n ∼ N ( f ∗ ( x ) + C x ⊤ ( C n × n + σ ϵ 2 I n × n ) − 1 ( Y − f ∗ ) , C ∗ ( x , x ) − C x ⊤ ( C n × n + σ ϵ 2 I n × n ) − 1 C x ) . consequently,
M ∗ ( π ^ n , x ) = E [ f ∣ π ^ n ] ( x ) = f ∗ ( x ) + C x ⊤ ( C n × n + σ ϵ 2 I n × n ) − 1 ( Y − f ∗ ) . (48) M_*(\widehat{\pi}_n,x)=\mathbb{E}[f\mid\widehat{\pi}_n](x)=f_*(x)+\mathbf{C}_x^\top(C_{n\times n}+\sigma_\epsilon^2I_{n\times n})^{-1}(\mathbf{Y}-\mathbf{f}_*). \tag*{(48)} M ∗ ( π n , x ) = E [ f ∣ π n ] ( x ) = f ∗ ( x ) + C x ⊤ ( C n × n + σ ϵ 2 I n × n ) − 1 ( Y − f ∗ ) . (48) We will derive an explicit expression of the RHS of (48) in terms of π ^ n , x \widehat{\pi}_n,x π n , x
A n v = ∑ i = 1 n v i Γ ∗ 1 / 2 R X i ⟨ X ⟩ , v = ( v 1 , … , v n ) ⊤ ∈ R n , A_n\mathbf{v}=\sum_{i=1}^{n}v_i\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle},\quad\mathbf{v}=(v_1,\ldots,v_n)^\top\in\mathbb{R}^n, A n v = i = 1 ∑ n v i Γ ∗ 1/2 R X i ⟨ X ⟩ , v = ( v 1 , … , v n ) ⊤ ∈ R n , which is a linear operator from R n \mathbb{R}^n R n H X \mathcal{H}_X H X A n A_n A n
A n † g = ( ( Γ ∗ 1 / 2 g ) ( X 1 ) , … , ( Γ ∗ 1 / 2 g ) ( X n ) ) ⊤ , g ∈ H X , A_n^\dagger g=((\Gamma_*^{1/2}g)(X_1),\ldots,(\Gamma_*^{1/2}g)(X_n))^\top,\quad g\in\mathcal{H}_X, A n † g = (( Γ ∗ 1/2 g ) ( X 1 ) , … , ( Γ ∗ 1/2 g ) ( X n ) ) ⊤ , g ∈ H X , which is a linear operator from H X \mathcal{H}_X H X R n \mathbb{R}^n R n v = ( v 1 , … , v n ) ⊤ ∈ R n \mathbf{v}=(v_1,\ldots,v_n)^\top\in\mathbb{R}^n v = ( v 1 , … , v n ) ⊤ ∈ R n g ∈ H X g\in\mathcal{H}_X g ∈ H X
⟨ v , A n † g ⟩ R n = ⟨ A n v , g ⟩ H X = ∑ i = 1 n v i ⟨ Γ ∗ 1 / 2 R X i ⟨ X ⟩ , g ⟩ H X = ∑ i = 1 n v i ⟨ R X i ⟨ X ⟩ , Γ ∗ 1 / 2 g ⟩ H X = ∑ i = 1 n v i ( Γ ∗ 1 / 2 g ) ( X i ) = v ⊤ ( ( Γ ∗ 1 / 2 g ) ( X 1 ) , … , ( Γ ∗ 1 / 2 g ) ( X n ) ) ⊤ . \begin{aligned}
\langle\mathbf{v},A_n^\dagger g\rangle_{\mathbb{R}^n}=\langle A_n\mathbf{v},g\rangle_{\mathcal{H}_X}=\sum_{i=1}^{n}v_i\langle\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle},g\rangle_{\mathcal{H}_X} \\
&=\sum_{i=1}^{n}v_i\langle R_{X_i}^{\langle X\rangle},\Gamma_*^{1/2}g\rangle_{\mathcal{H}_X}=\sum_{i=1}^{n}v_i(\Gamma_*^{1/2}g)(X_i)=\mathbf{v}^{\top}((\Gamma_*^{1/2}g)(X_1),\ldots,(\Gamma_*^{1/2}g)(X_n))^\top.
\end{aligned} ⟨ v , A n † g ⟩ R n = ⟨ A n v , g ⟩ H X = i = 1 ∑ n v i ⟨ Γ ∗ 1/2 R X i ⟨ X ⟩ , g ⟩ H X = i = 1 ∑ n v i ⟨ R X i ⟨ X ⟩ , Γ ∗ 1/2 g ⟩ H X = i = 1 ∑ n v i ( Γ ∗ 1/2 g ) ( X i ) = v ⊤ (( Γ ∗ 1/2 g ) ( X 1 ) , … , ( Γ ∗ 1/2 g ) ( X n ) ) ⊤ . It can be checked that
A n † A n = C n × n and n − 1 A n A n † = J π ^ n . (49) A_n^\dagger A_n=C_{n\times n}\ \text{and}\ n^{-1}A_nA_n^\dagger=J_{\widehat{\pi}_n}. \tag*{(49)} A n † A n = C n × n and n − 1 A n A n † = J π n . (49) To see this, for any v ∈ R n \mathbf{v} \in\mathbb{R}^{n} v ∈ R n (24) and direct calculations,
A n † A n v = A n † ∑ i = 1 n v i Γ ∗ 1 / 2 R X i ⟨ X ⟩ = ∑ i = 1 n v i A n † ( Γ ∗ 1 / 2 R X i ⟨ X ⟩ ) = ∑ i = 1 n v i ( ( Γ ∗ R X i ⟨ X ⟩ ) ( X 1 ) , … , ( Γ ∗ R X i ⟨ X ⟩ ) ( X n ) ) T = ( ∑ i = 1 n v i C ∗ ( X 1 , X i ) , … , ∑ i = 1 n v i C ∗ ( X n , X i ) ) T = C n × n v , \begin{aligned}
A_n^{\dagger} A_n \mathbf{v} = A_n^{\dagger} \sum_{i=1}^{n} v_i \Gamma_*^{1/2} R_{X_i}^{\langle X\rangle} = \sum_{i=1}^{n} v_i A_n^{\dagger}(\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle}) \\
&= \sum_{i=1}^{n} v_i \left((\Gamma_*R_{X_i}^{\langle X\rangle})(X_1),\ldots,(\Gamma_*R_{X_i}^{\langle X\rangle})(X_n)\right)^{\mathsf{T}} \\
&= \left(\sum_{i=1}^{n}v_iC_*(X_1,X_i),\ldots,\sum_{i=1}^{n}v_iC_*(X_n,X_i)\right)^{\mathsf{T}} = C_{n\times n}\mathbf{v},
\end{aligned} A n † A n v = A n † i = 1 ∑ n v i Γ ∗ 1/2 R X i ⟨ X ⟩ = i = 1 ∑ n v i A n † ( Γ ∗ 1/2 R X i ⟨ X ⟩ ) = i = 1 ∑ n v i ( ( Γ ∗ R X i ⟨ X ⟩ ) ( X 1 ) , … , ( Γ ∗ R X i ⟨ X ⟩ ) ( X n ) ) T = ( i = 1 ∑ n v i C ∗ ( X 1 , X i ) , … , i = 1 ∑ n v i C ∗ ( X n , X i ) ) T = C n × n v , and for any g ∈ H X g \in\mathcal{H}_X g ∈ H X
n − 1 A n A n † g = n − 1 A n ( ( Γ ∗ 1 / 2 g ) ( X 1 ) , … , Γ ∗ 1 / 2 g ) ( X n ) ) T = n − 1 ∑ i = 1 n ( Γ ∗ 1 / 2 g ) ( X i ) Γ ∗ 1 / 2 R X i ⟨ X ⟩ = n − 1 ∑ i = 1 n ⟨ Γ ∗ 1 / 2 g , R X i ⟨ X ⟩ ⟩ H X Γ ∗ 1 / 2 R X i ⟨ X ⟩ = n − 1 ∑ i = 1 n ⟨ g , Γ ∗ 1 / 2 R X i ⟨ X ⟩ ⟩ H X Γ ∗ 1 / 2 R X i ⟨ X ⟩ = n − 1 ∑ i = 1 n ( Γ ∗ 1 / 2 R X i ⟨ X ⟩ ) ⊗ ( Γ ∗ 1 / 2 R X i ⟨ X ⟩ ) g = J π ^ n g . \begin{aligned}
n^{-1}A_nA_n^{\dagger}g = n^{-1}A_n\left((\Gamma_*^{1/2}g)(X_1),\ldots,\Gamma_*^{1/2}g)(X_n)\right)^{\mathsf{T}} = n^{-1}\sum_{i=1}^{n}(\Gamma_*^{1/2}g)(X_i)\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle} \\
&= n^{-1}\sum_{i=1}^{n}\langle\Gamma_*^{1/2}g,R_{X_i}^{\langle X\rangle}\rangle_{\mathcal{H}_X}\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle}
= n^{-1}\sum_{i=1}^{n}\langle g,\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle}\rangle_{\mathcal{H}_X}\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle} \\
&= n^{-1}\sum_{i=1}^{n}\left(\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle}\right)\otimes\left(\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle}\right)g = J_{\widehat{\pi}_n}g.
\end{aligned} n − 1 A n A n † g = n − 1 A n ( ( Γ ∗ 1/2 g ) ( X 1 ) , … , Γ ∗ 1/2 g ) ( X n ) ) T = n − 1 i = 1 ∑ n ( Γ ∗ 1/2 g ) ( X i ) Γ ∗ 1/2 R X i ⟨ X ⟩ = n − 1 i = 1 ∑ n ⟨ Γ ∗ 1/2 g , R X i ⟨ X ⟩ ⟩ H X Γ ∗ 1/2 R X i ⟨ X ⟩ = n − 1 i = 1 ∑ n ⟨ g , Γ ∗ 1/2 R X i ⟨ X ⟩ ⟩ H X Γ ∗ 1/2 R X i ⟨ X ⟩ = n − 1 i = 1 ∑ n ( Γ ∗ 1/2 R X i ⟨ X ⟩ ) ⊗ ( Γ ∗ 1/2 R X i ⟨ X ⟩ ) g = J π n g . We also notice by (24) that
A n † ( Γ ∗ 1 / 2 R x ⟨ X ⟩ ) = ( ( Γ ∗ R x ⟨ X ⟩ ) ( X 1 ) , … , ( Γ ∗ R x ⟨ X ⟩ ) ( X n ) ) T = ( C ∗ ( X 1 , x ) , … , C ∗ ( X n , x ) ) T = C x , A_n^{\dagger}(\Gamma_*^{1/2}R_x^{\langle X\rangle}) = ((\Gamma_*R_x^{\langle X\rangle})(X_1),\ldots,(\Gamma_*R_x^{\langle X\rangle})(X_n))^{\mathsf{T}} = (C_*(X_1,x),\ldots,C_*(X_n,x))^{\mathsf{T}} = C_x, A n † ( Γ ∗ 1/2 R x ⟨ X ⟩ ) = (( Γ ∗ R x ⟨ X ⟩ ) ( X 1 ) , … , ( Γ ∗ R x ⟨ X ⟩ ) ( X n ) ) T = ( C ∗ ( X 1 , x ) , … , C ∗ ( X n , x ) ) T = C x , and
A n ( Y − f ∗ ) = ∑ i = 1 n ( Y i − f ∗ ( X i ) ) Γ ∗ 1 / 2 R X i ⟨ X ⟩ = n G π ^ n . A_n(\mathbf{Y}-f_*) = \sum_{i=1}^{n}(Y_i-f_*(X_i))\Gamma_*^{1/2}R_{X_i}^{\langle X\rangle} = nG_{\widehat{\pi}_n}. A n ( Y − f ∗ ) = i = 1 ∑ n ( Y i − f ∗ ( X i )) Γ ∗ 1/2 R X i ⟨ X ⟩ = n G π n . Let T π ^ n = ( J π ^ n + σ ε 2 n − 1 id ) − 1 T_{\widehat{\pi}_n}=(J_{\widehat{\pi}_n}+\sigma_{\varepsilon}^{2}n^{-1}\operatorname{id})^{-1} T π n = ( J π n + σ ε 2 n − 1 id ) − 1 (48) and (49) , we have
M ∗ ( π ^ n , x ) = f ∗ ( x ) + ⟨ C x , ( C n × n + σ ε 2 I n × n ) − 1 ( Y − f ∗ ) ⟩ R n = f ∗ ( x ) + ⟨ A n † ( Γ ∗ 1 / 2 R x ⟨ X ⟩ ) , ( A n † A n + σ ε 2 I n × n ) − 1 ( Y − f ∗ ) ⟩ R n = f ∗ ( x ) + ⟨ Γ ∗ 1 / 2 R x ⟨ X ⟩ , A n ( A n † A n + σ ε 2 I n × n ) − 1 ( Y − f ∗ ) ⟩ H X = f ∗ ( x ) + ⟨ Γ ∗ 1 / 2 R x ⟨ X ⟩ , ( A n A n † + σ ε 2 id ) − 1 A n ( Y − f ∗ ) ⟩ H X = f ∗ ( x ) + ⟨ Γ ∗ 1 / 2 R x ⟨ X ⟩ , ( J π ^ n + σ ε 2 n − 1 id ) − 1 G π ^ n ⟩ H X = f ∗ ( x ) + ⟨ R x ⟨ X ⟩ , Γ ∗ 1 / 2 T π ^ n G π ^ n ⟩ H X = ⟨ f ∗ + Γ ∗ 1 / 2 T π ^ n G π ^ n , R x ⟨ X ⟩ ⟩ H X = ⟨ H ∗ ( π ^ n ) , R x ⟨ X ⟩ ⟩ H X . \begin{aligned}
M_*(\widehat{\pi}_n,x)
&= f_*(x)+\langle C_x,(C_{n\times n}+\sigma_{\varepsilon}^{2}I_{n\times n})^{-1}(\mathbf{Y}-f_*)\rangle_{\mathbb{R}^n}\\
&= f_*(x)+\langle A_n^{\dagger}(\Gamma_*^{1/2}R_x^{\langle X\rangle}),(A_n^{\dagger}A_n+\sigma_{\varepsilon}^{2}I_{n\times n})^{-1}(\mathbf{Y}-f_*)\rangle_{\mathbb{R}^n}\\
&= f_*(x)+\langle\Gamma_*^{1/2}R_x^{\langle X\rangle},A_n(A_n^{\dagger}A_n+\sigma_{\varepsilon}^{2}I_{n\times n})^{-1}(\mathbf{Y}-f_*)\rangle_{\mathcal{H}_X}\\
&= f_*(x)+\langle\Gamma_*^{1/2}R_x^{\langle X\rangle},(A_nA_n^{\dagger}+\sigma_{\varepsilon}^{2}\operatorname{id})^{-1}A_n(\mathbf{Y}-f_*)\rangle_{\mathcal{H}_X}\\
&= f_*(x)+\langle\Gamma_*^{1/2}R_x^{\langle X\rangle},(J_{\widehat{\pi}_n}+\sigma_{\varepsilon}^{2}n^{-1}\operatorname{id})^{-1}G_{\widehat{\pi}_n}\rangle_{\mathcal{H}_X}\\
&= f_*(x)+\langle R_x^{\langle X\rangle},\Gamma_*^{1/2}T_{\widehat{\pi}_n}G_{\widehat{\pi}_n}\rangle_{\mathcal{H}_X}
= \langle f_*+\Gamma_*^{1/2}T_{\widehat{\pi}_n}G_{\widehat{\pi}_n},R_x^{\langle X\rangle}\rangle_{\mathcal{H}_X}
= \langle H_*(\widehat{\pi}_n),R_x^{\langle X\rangle}\rangle_{\mathcal{H}_X}.
\end{aligned} M ∗ ( π n , x ) = f ∗ ( x ) + ⟨ C x , ( C n × n + σ ε 2 I n × n ) − 1 ( Y − f ∗ ) ⟩ R n = f ∗ ( x ) + ⟨ A n † ( Γ ∗ 1/2 R x ⟨ X ⟩ ) , ( A n † A n + σ ε 2 I n × n ) − 1 ( Y − f ∗ ) ⟩ R n = f ∗ ( x ) + ⟨ Γ ∗ 1/2 R x ⟨ X ⟩ , A n ( A n † A n + σ ε 2 I n × n ) − 1 ( Y − f ∗ ) ⟩ H X = f ∗ ( x ) + ⟨ Γ ∗ 1/2 R x ⟨ X ⟩ , ( A n A n † + σ ε 2 id ) − 1 A n ( Y − f ∗ ) ⟩ H X = f ∗ ( x ) + ⟨ Γ ∗ 1/2 R x ⟨ X ⟩ , ( J π n + σ ε 2 n − 1 id ) − 1 G π n ⟩ H X = f ∗ ( x ) + ⟨ R x ⟨ X ⟩ , Γ ∗ 1/2 T π n G π n ⟩ H X = ⟨ f ∗ + Γ ∗ 1/2 T π n G π n , R x ⟨ X ⟩ ⟩ H X = ⟨ H ∗ ( π n ) , R x ⟨ X ⟩ ⟩ H X . Therefore, M ∗ M_* M ∗ ( π ^ n , x ) (\widehat{\pi}_n,x) ( π n , x )
References
Adams, R. A. and Fournier, J. J. (2003). Sobolev Spaces , volume 140. Elsevier.
Agarwal, R., Singh, A., Zhang, L., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J., Chu, E., Behbahani, F., Faust, A., and Larochelle, H. (2024). Many-
