Introduction
The equivalence between optimal estimation under the Gaussian hypothesis and regularized least squares is one of the organizing principles of estimation theory. It is also the reason why the same algorithm circulates under different names in different communities: Tikhonov regularization [55 ] and ridge regression [18 ] in inverse problems and statistics, the Wiener filter in signal processing [24 , 67 ] , kriging in geostatistics [35 , 39 ] , and Gaussian-process regression in machine learning [5 , 44 ] . In finite dimensions, the equivalence is elementary. Let us consider the linear measurement model y = H x + ϵ \mathbf{y} = \mathbf{H}\mathbf{x} + \boldsymbol{\epsilon} y = Hx + ϵ , where the signal x ∈ R N \mathbf{x} \in\mathbb{R}^{N} x ∈ R N is a realization of a Gaussian random vector X ∼ N ( 0 , C X ) \mathbf{X} \sim\mathcal{N}(\mathbf{0}, \mathbf{C}_{X}) X ∼ N ( 0 , C X ) and where ϵ ∼ N ( 0 , σ 2 I ) \boldsymbol{\epsilon} \sim\mathcal{N}(\mathbf{0}, \sigma^{2}\mathbf{I}) ϵ ∼ N ( 0 , σ 2 I ) is an independent measurement noise. The posterior distribution of X \mathbf{X} X given y \mathbf{y} y is Gaussian, so that its mean and its mode coincide: the minimum-mean-square-error (MMSE) estimator and the maximum-a-posteriori (MAP) estimator are one and the same. The MAP estimator minimizes the negative log-posterior, which makes it the solution of a penalized least-squares problem. What is specific to the Gaussian hypothesis is that the penalty is quadratic and that the minimizer is also the MMSE estimator: the common estimator is the solution of the Tikhonov problem
x λ = arg min x ∈ R N ( ∥ y − H x ∥ 2 2 + λ ∥ L x ∥ 2 2 ) (1) \mathbf{x}_{\lambda} = \underset{\mathbf{x}\in\mathbb{R}^{N}}{\operatorname{arg\,min}}\left(\lVert\mathbf{y}-\mathbf{H}\mathbf{x}\rVert_{2}^{2}+\lambda\lVert\mathbf{L}\mathbf{x}\rVert_{2}^{2}\right) \tag*{(1)} x λ = x ∈ R N arg min ( ∥ y − Hx ∥ 2 2 + λ ∥ Lx ∥ 2 2 ) (1)
with λ = σ 2 \lambda= \sigma^{2} λ = σ 2 , provided that the regularization matrix L \mathbf{L} L whitens the prior, in the sense that L C X L T = I \mathbf{L}\mathbf{C}_{X}\mathbf{L}^{T} = \mathbf{I} L C X L T = I . This classic correspondence is the template that the present paper transposes to the continuum. Our deterministic reference is the formulation of the variational theory of splines in [59 , 60 ] , which is the most extensive that we are aware of: the regularization operator L \mathbf{L} L is any spline-admissible operator, whose null space may be nontrivial, the signals are generalized functions, and the measurements are linear functionals that are continuous on the native space. Our aim is to establish the correspondence at this level of generality. The generalized splines take the role of the linear estimators: they are the solutions of the continuous-domain counterparts of (1), and they are of the form f = ∑ m = 1 M a m A { ν m } + p f = \sum_{m=1}^{M} a_{m}A\{\nu_{m}\} + p f = ∑ m = 1 M a m A { ν m } + p , where the ν m \nu_{m} ν m are the measurement functionals, A A A is the Green’s operator (conditional inverse) of the normal operator L ∗ L \mathbf{L}^{*}\mathbf{L} L ∗ L , and p p p lies in the null space of L \mathbf{L} L . The generalized Gaussian processes (GGPs) take the role of the Gaussian random vectors; they are random elements of the dual S ′ \mathcal{S}^{\prime} S ′ of a nuclear Gelfand pair ( S , S ′ ) (\mathcal{S},\mathcal{S}^{\prime}) ( S , S ′ ) , in the sense of Gelfand and Vilenkin [15 ] ; the prototype of such a pair is the Schwartz space of rapidly decaying test functions together with its dual, the space of tempered distributions.
The continuous-domain version of this correspondence has a long history. It was uncovered by Kimeldorf and Wahba, first for the case where L \mathbf{L} L is shift-invariant with a trivial null space [26 ] and then for the Tchebycheffian splines on an interval [27 ] , and it was developed into a complete statistical methodology in [50 , 63 ] . In parallel, Parzen [41 ] recast time-series estimation in the language of reproducing-kernel Hilbert spaces (RKHS) [1 ] , while Kailath and his students connected RKHS norms, innovations representations, and Fredholm equations [22 , 23 , 64 , 65 ] . The modern kernel methods of machine learning inherit the same duality [19 , 25 , 40 , 49 ] , as does the stochastic-partial-differential-equation approach to spatial statistics [30 , 31 ] , in which a Gaussian field is specified through the operator that whitens it rather than through its covariance.
There is, however, a loose end in this otherwise satisfying picture, and it affects precisely the estimators that approximation theory favors. When the regularization operator L L L has a nontrivial null space N p = span { p 1 , … , p N 0 } \mathcal{N}_p = \operatorname{span}\{p_1,\ldots,p_{N_0}\} N p = span { p 1 , … , p N 0 } —which is the case of the derivative operators D m D^m D m that generate the polynomial splines, the workhorses of the field—the L L L -smoothing spline loses its MMSE optimality. What survives is the weaker statement that it is the best linear unbiased estimator in a so-called fixed-effect model, where the bias is measured relative to a fixed but unknown drift p ∈ N p p \in\mathcal{N}_p p ∈ N p [63 Sec. 1.5] . Geostatistics meets the same obstacle: the intrinsic random functions of order k k k of Matheron [35 ] are defined only modulo the polynomials of degree k k k , and universal kriging is, likewise, a best linear unbiased predictor. This is frustrating, for these are the estimators with the best deterministic approximation rates, and they owe those rates to the very null space that degrades their statistical status. The reason for the degradation is easy to identify: the process whose covariance operator is the inverse of L ∗ L L^*L L ∗ L does not exist when L L L is not invertible, so that there is no prior with respect to which a Bayesian estimator could be optimal.
The resolution that we propose is to construct the missing prior, in a setting that is broad enough to accommodate signals that are generalized functions. The conditioned parametrization of [60 ] provides the two ingredients: a right-inverse L ϕ − 1 L_\phi^{-1} L ϕ − 1 of L L L that is made unique by N 0 N_0 N 0 boundary conditions ϕ ( f ) = ( ⟨ ϕ 1 , f ⟩ , … , ⟨ ϕ N 0 , f ⟩ ) = 0 \phi(f) = (\langle\phi_1,f\rangle,\ldots,\langle\phi_{N_0},f\rangle) = \mathbf{0} ϕ ( f ) = (⟨ ϕ 1 , f ⟩ , … , ⟨ ϕ N 0 , f ⟩) = 0 , and the associated Gram operator A ϕ = L ϕ − 1 L ϕ − ∗ A_\phi= L_\phi^{-1}L_\phi^{-*} A ϕ = L ϕ − 1 L ϕ −∗ , which is symmetric and p p p -conditionally positive-definite: its quadratic form is positive on the test functions that annihilate N p \mathcal{N}_p N p , while it vanishes on the boundary functionals ϕ n \phi_n ϕ n . The object that we are after is the Gaussian process with covariance operator A ϕ : S → S ′ A_\phi: \mathcal{S} \to\mathcal{S}' A ϕ : S → S ′ . Since A ϕ A_\phi A ϕ induces the norm of H L ′ \mathcal{H}'_L H L ′ (Section 2.2), it is not of trace class, and Sazonov’s theorem [7 , 46 ] rules out a realization of this process in the native space H L \mathcal{H}_L H L itself. The correct home is the dual of a nuclear space, where the theorem of Minlos [15 , 37 ] converts the continuity of a characteristic functional into a bona fide probability measure. This is the same ( S , S ′ ) (\mathcal{S},\mathcal{S}') ( S , S ′ ) sandwich that hosts the native space [59 , 60 ] , which is what makes the two constructions superposable.
Our contributions are the following.
Generalized Gaussian processes on Hilbert–Gelfand spaces (§ 3.2 ). Our starting point is the classical result of Gelfand and Vilenkin (Theorem 3.2), which attaches a GGP G ∼ N ( μ , Q ) G \sim\mathcal{N}(\mu,Q) G ∼ N ( μ , Q ) to any symmetric positive-semi-definite covariance operator Q : S → S ′ Q:S \to S' Q : S → S ′ . In Theorem 3.3 , we extend the action of the process to test functions in the dual H L ′ ⊃ S \mathcal{H}'_L \supset S H L ′ ⊃ S of the native space, and we characterize the “degenerate” class that corresponds to a p p p -conditionally positive-definite covariance: such a process vanishes almost surely on the span N ϕ \mathcal{N}_{\phi} N ϕ of the boundary functionals, which is the probabilistic transcription of the boundary conditions.
Innovation models (§ 3.3 ). We show that the processes of this class are exactly those that admit the innovation representation G = L ϕ − 1 { W } G = L_{\phi}^{-1}\{W\} G = L ϕ − 1 { W } in terms of a Gaussian white noise W W W , or equivalently, that solve the stochastic differential equation L G = W LG = W L G = W subject to ϕ ( G ) = 0 \phi(G) = 0 ϕ ( G ) = 0 . The formalism subsumes the classical integral representations: the fractional Brownian motion of Mandelbrot and Van Ness [34 ] is recovered as the case L = D γ L = D^{\gamma} L = D γ with a single boundary condition, and its higher-order extensions are worked out in Appendix § 7 .
Congruence (Proposition 3.5 ). The Gaussian space of the process—the closed linear span of the random variables G ( φ ) G(\varphi) G ( φ ) —is isometric to the core H A = L ∗ ( L 2 ) \mathcal{H}_{A} = L^{*}(L_{2}) H A = L ∗ ( L 2 ) , which is the part of the dual H L ′ \mathcal{H}'_{L} H L ′ that does not depend on the boundary conditions. Under this isometry, the Riesz pairing of the native space is the covariance of the process, and the innovation provides an orthonormal coordinate system. This is the statement that welds the deterministic and the stochastic descriptions together; Table 1 lists the correspondence term by term.
Table 1. Dictionary between the deterministic objects of the variational theory [59 , 60 ] and their stochastic counterparts in the present paper.
Reproducing kernels and covariance functions (§ 4 ). For the classic setting where H L \mathcal{H}_{L} H L is an RKHS on R d \mathbb{R}^{d} R d , we identify in Theorem 4.2 a single bivariate function a ϕ a_{\phi} a ϕ as the Schwartz kernel of the Gram operator, the reproducing kernel of the primary native space H L , ϕ \mathcal{H}_{L,\phi} H L , ϕ (the functions of H L \mathcal{H}_{L} H L that satisfy the boundary conditions), and the covariance function of the process. The formula involves the Green’s function of the normal operator L ∗ L L^{*}L L ∗ L , corrected in each variable by the finite-rank term that carries the boundary conditions. (For a trivial null space, the correction vanishes and a ϕ a_{\phi} a ϕ reduces to the Green’s function itself, which is the classical kernel [60 Section 4.2] .) In Theorem 4.4 , we match the operator-level notion of (p p p -conditional) positive
definiteness with the kernel-level one of the radial-basis-function literature [8 , 66 ] . As an illustration, we give in § 4.3 a complete and explicit characterization of the isotropic fractional Brownian fields of arbitrary order on R d \mathbb{R}^{d} R d —the innovation kernel, the stochastic integral representation à la Mandelbrot, and the pointwise covariance—which appears to be new beyond the classical range H ∈ ( 0 , 1 ) H \in(0,1) H ∈ ( 0 , 1 ) of the Hurst exponent, the range of Lévy’s field and of its fractional extensions.
Splines as MMSE estimators (§ 5 ). For the measurement model y = ν ( g ) + ϵ \mathbf{y} = \nu(g) + \epsilon y = ν ( g ) + ϵ , we identify in Theorem 5.3 the MMSE estimator of G ( φ ) G(\varphi) G ( φ ) with the generalized smoothing spline that is constrained by the boundary conditions ϕ ( g ) = 0 \phi(g) = \mathbf{0} ϕ ( g ) = 0 [60 ] , evaluated at φ \varphi φ , with λ = σ 2 \lambda= \sigma^{2} λ = σ 2 and without any restriction to linear or to unbiased estimators. This closes the loose end: once the prior is the conditioned process, a generalized spline is the MMSE estimator outright, and not merely the best linear unbiased one. In Corollary 5.5 , we give the noise-free counterpart: every generalized spline interpolant is the MMSE interpolator of a corresponding GGP.
A single mathematical object underlies the whole construction, and it admits three readings. In the language of functional analysis, it is a conditional Riesz pair ( A , B ) (A, B) ( A , B ) over the nuclear Gelfand pair ( S , S ′ ) (\mathcal{S}, \mathcal{S}') ( S , S ′ ) [59 ] ; in the language of approximation theory, it is the native space H L \mathcal{H}_{L} H L of the p p p -admissible operator L L L , together with its representer theorem [60 ] ; and in the language of probability, which is the one of the present paper, it is the covariance structure of a generalized Gaussian process. What makes the third reading possible is the conditioned parametrization: the boundary conditions that single out the right-inverse L ϕ − 1 L_{\phi}^{-1} L ϕ − 1 are exactly the ones that make the Gaussian process well-defined.
The paper proceeds as follows. In § 2 , we recall the functional-analytic setting of [59 , 60 ] and the Minlos–Bochner theorem. In § 3 , we develop the theory of generalized Gaussian processes, from the abstract definition to the innovation models and the congruence. In § 4 , we make the link with the classical theory: we identify the reproducing kernel of the native space with the covariance function of the process, and we relate the operator-level and kernel-level notions of conditional positive definiteness. In § 5 , we establish the equivalence between generalized splines and MMSE estimators, first in the non-degenerate case, and then in full generality. The framework is illustrated in § 4.3 with the characterization of the fractional Brownian fields on R d \mathbb{R}^{d} R d , and in Appendix § 7 with the higher-order extensions of fractional Brownian motion (fBm).
Preliminaries
Nuclear Gelfand Pairs and Hilbert–Gelfand Spaces
We briefly recall the functional-analytic setting of [59 , 60 ] , to which we refer for the complete exposition.
Definition 2.1 (Nuclear Gelfand pair). The couple ( S , S ′ ) (\mathcal{S},\mathcal{S}') ( S , S ′ ) that is formed by a topological vector space S \mathcal{S} S and its topological dual S ′ \mathcal{S}' S ′ is called a nuclear Gelfand pair if
S \mathcal{S} S is nuclear and Montel [17 , 57 ] ;
there exists a continuous and dense embedding ι : S → S ′ \iota:\mathcal{S}\to\mathcal{S}' ι : S → S ′ that is canonical, in the sense that ⟨ ι φ , ψ ⟩ = ⟨ ι ψ , φ ⟩ \langle\iota\varphi,\psi\rangle=\langle\iota\psi,\varphi\rangle ⟨ ι φ , ψ ⟩ = ⟨ ι ψ , φ ⟩ for all φ , ψ ∈ S \varphi,\psi\in\mathcal{S} φ , ψ ∈ S .
Unless stated otherwise, S ′ \mathcal{S}' S ′ is endowed with its weak-∗ * ∗ topology.
The prototypical example is the pair ( S ( R d ) , S ′ ( R d ) ) (\mathcal{S}(\mathbb{R}^{d}),\mathcal{S}'(\mathbb{R}^{d})) ( S ( R d ) , S ′ ( R d )) formed by the Schwartz space of smooth and rapidly decaying test functions on R d \mathbb{R}^{d} R d and its dual, the space of tempered distributions.
A Hilbert space H \mathcal{H} H is said to be a Hilbert–Gelfand space if S ↪ d H ↪ d S ′ \mathcal{S}\xhookrightarrow{\mathrm{d}}\mathcal{H}\xhookrightarrow{\mathrm{d}}\mathcal{S}' S d H d S ′ , where the symbol “↪ d \xhookrightarrow{\mathrm{d}} d ” denotes a continuous and dense embedding; such a space can be constructed by completion as H = ( S , ∥ ⋅ ∥ H ) \mathcal{H}=(\mathcal{S},\|\cdot\|_{\mathcal{H}}) H = ( S , ∥ ⋅ ∥ H ) , and its dual H ′ \mathcal{H}' H ′ is a Hilbert–Gelfand space as well, with the duality pairing that is compatible with that of ( S , S ′ ) (\mathcal{S},\mathcal{S}') ( S , S ′ ) . The baseline example is the pivot space L 2 = ( S , ∥ ⋅ ∥ L 2 ) L_{2}=(\mathcal{S},\|\cdot\|_{L_{2}}) L 2 = ( S , ∥ ⋅ ∥ L 2 ) .
Native Spaces and Spline-Admissible Operators
We now recall, without proof, the operator-theoretic objects of [59 , 60 ] that the stochastic constructions of this paper rest upon. The regularization (or whitening) operator that guides the construction is L : S → L 2 L:\mathcal{S}\to L_{2} L : S → L 2 . By suitable completion, this operator induces the native space H L \mathcal{H}_{L} H L , which is such that L : H L → L 2 L:\mathcal{H}_{L}\to L_{2} L : H L → L 2 with finite dimensional null space N p = span ( p 1 , ⋯ , p N 0 ) = { f ∈ H L : L f = 0 } \mathcal{N}_{p}=\operatorname{span}(p_{1},\cdots,p_{N_{0}})=\{f\in\mathcal{H}_{L}:Lf=0\} N p = span ( p 1 , ⋯ , p N 0 ) = { f ∈ H L : L f = 0 } . The adjoint of L L L is L ∗ : L 2 → H A ⊂ H L ′ L^{*}:L_{2}\to\mathcal{H}_{A}\subset\mathcal{H}'_{L} L ∗ : L 2 → H A ⊂ H L ′ where H L ′ \mathcal{H}'_{L} H L ′ is the topological dual of H L \mathcal{H}_{L} H L and H A : = L ∗ ( L 2 ) \mathcal{H}_{A}:=L^{*}(L_{2}) H A := L ∗ ( L 2 ) .
Since the conditions under which this construction is valid are central to what follows, let us spell them out explicitly. Let p = ( p 1 , … , p N 0 ) ⊂ S ′ \boldsymbol{p}=(p_{1},\ldots,p_{N_{0}})\subset\mathcal{S}' p = ( p 1 , … , p N 0 ) ⊂ S ′ be a linearly independent family with N p = span { p n } n = 1 N 0 \mathcal{N}_{p}=\operatorname{span}\{p_{n}\}_{n=1}^{N_{0}} N p = span { p n } n = 1 N 0 , and let S p = { φ ∈ S : p ( φ ) = 0 } \mathcal{S}_{p}=\{\varphi\in\mathcal{S}:p(\varphi)=0\} S p = { φ ∈ S : p ( φ ) = 0 } be the space of test functions that annihilate it. A continuous symmetric operator T : S → S ′ T:\mathcal{S}\to\mathcal{S}' T : S → S ′ induces the energy seminorm ∣ φ ∣ T = ⟨ T φ , φ ⟩ 1 / 2 |\varphi|_{T}=\langle T\varphi,\varphi\rangle^{1/2} ∣ φ ∣ T = ⟨ T φ , φ ⟩ 1/2 , whose attainable null space
N T = { p ∈ S ′ : p = lim k → ∞ χ k in S ′ for some ( χ k ) ⊂ S with ∣ χ k ∣ T → 0 } (2) \mathcal{N}_{T}=\left\{p\in\mathcal{S}':\ p=\lim_{k\to\infty}\chi_k\text{ in }\mathcal{S}'\text{ for some }(\chi_k)\subset\mathcal{S}\text{ with }|\chi_k|_{T}\to0\right\} \tag*{(2)} N T = { p ∈ S ′ : p = k → ∞ lim χ k in S ′ for some ( χ k ) ⊂ S with ∣ χ k ∣ T → 0 } (2)
is the space of generalized functions that are reached as limits of test functions of vanishing energy [60 Definition 2.4] . Let us stress that N T \mathcal{N}_{T} N T is generally non-trivial even when ∣ ⋅ ∣ T |\cdot|_{T} ∣ ⋅ ∣ T is a genuine norm on S \mathcal{S} S , since the limit is taken in the much weaker topology of S ′ \mathcal{S}' S ′ .
Definition 2.2 (p p p -admissible operator). Let L : S → L 2 L:\mathcal{S}\to L_2 L : S → L 2 be a continuous linear operator that is injective and whose range is dense in L 2 L_2 L 2 , and let us assume that its normal operator B = L ∗ L : S → S ′ B=L^*L:\mathcal{S}\to\mathcal{S}' B = L ∗ L : S → S ′ admits a conditional inverse A : S → S ′ A:\mathcal{S}\to\mathcal{S}' A : S → S ′ , in the sense that
A B = Id on S and B A = Id on S p . (3) AB=\operatorname{Id}\text{ on }\mathcal{S}\qquad\text{and}\qquad BA=\operatorname{Id}\text{ on }\mathcal{S}_p. \tag*{(3)} A B = Id on S and B A = Id on S p . (3) Then, L L L is said to be p p p -admissible if the family p p p spans the attainable null space of the spline energy ∣ φ ∣ B = ∥ L φ ∥ L 2 |\varphi|_{B}=\|L\varphi\|_{L_2} ∣ φ ∣ B = ∥ L φ ∥ L 2 ; that is, if
N B = N p . (4) \mathcal{N}_{B}=\mathcal{N}_{p}. \tag*{(4)} N B = N p . (4) The single condition (4) says that p p p is neither too small nor too large, and it is what makes the null space of the continuous extension L : H L → L 2 L:\mathcal{H}_{L}\to L_2 L : H L → L 2 be exactly N p \mathcal{N}_{p} N p . Let us also point at the asymmetry between the two operators, which is the delicate point of the construction. The normal operator B B B is positive-definite on the whole of S \mathcal{S} S , since ∣ φ ∣ B = ∥ L φ ∥ L 2 |\varphi|_{B}=\|L\varphi\|_{L_2} ∣ φ ∣ B = ∥ L φ ∥ L 2 and L L L is injective, while its conditional inverse A A A is only p p p -conditionally positive-definite, in the sense that ⟨ A φ , φ ⟩ > 0 \langle A\varphi,\varphi\rangle>0 ⟨ A φ , φ ⟩ > 0 for all φ ∈ S p ∖ { 0 } \varphi\in\mathcal{S}_{p}\setminus\{0\} φ ∈ S p ∖ { 0 } ; this is also why the second identity in (3) is restricted to S p \mathcal{S}_{p} S p . Under the hypotheses of Definition 2.2, the condition (4) is equivalent to ( A , B ) (A,B) ( A , B ) being a p p p -conditional Riesz pair [59 Corollary 4.1] : the two completions H A = ( S p , ∣ ⋅ ∣ A ) \mathcal{H}_{A}=(\mathcal{S}_{p},|\cdot|_{A}) H A = ( S p , ∣ ⋅ ∣ A ) and H ˙ B = ( S , ∣ ⋅ ∣ B ) \dot{\mathcal{H}}_{B}=(\mathcal{S},|\cdot|_{B}) H ˙ B = ( S , ∣ ⋅ ∣ B ) are then put in duality by A A A and B B B , with H A = L ∗ ( L 2 ) \mathcal{H}_{A}=L^*(L_2) H A = L ∗ ( L 2 ) a genuine Hilbert subspace of S ′ \mathcal{S}' S ′ and H ˙ B ≃ L 2 \dot{\mathcal{H}}_{B}\simeq L_2 H ˙ B ≃ L 2 a space of equivalence classes modulo N p \mathcal{N}_{p} N p .
Let us now turn to the objects that the boundary conditions bring in. Given a p p p -admissible operator L L L , there always exists a biorthogonal family ϕ = ( ϕ 1 , … , ϕ N 0 ) ⊂ H L ′ \phi=(\phi_1,\ldots,\phi_{N_0})\subset\mathcal{H}'_{L} ϕ = ( ϕ 1 , … , ϕ N 0 ) ⊂ H L ′ such that ⟨ p m , ϕ n ⟩ = δ m , n \langle p_m,\phi_n\rangle=\delta_{m,n} ⟨ p m , ϕ n ⟩ = δ m , n [59 Lemma 3.5] . To any such family ν = ( ν 1 , … , ν N 0 ) \nu=(\nu_1,\ldots,\nu_{N_0}) ν = ( ν 1 , … , ν N 0 ) , we associate the rank-N 0 N_0 N 0 operator R ν : = ∑ n = 1 N 0 ν n ⟨ ν n , ⋅ ⟩ R_{\nu}:=\sum_{n=1}^{N_0}\nu_n\langle\nu_n,\cdot\rangle R ν := ∑ n = 1 N 0 ν n ⟨ ν n , ⋅ ⟩ . The relevant projectors are then
P ϕ = Id − R ϕ R p : H L ′ → H A (5) P_{\phi}=\operatorname{Id}-R_{\phi}R_{p}:\mathcal{H}'_{L}\to\mathcal{H}_{A} \tag*{(5)} P ϕ = Id − R ϕ R p : H L ′ → H A (5)
and its adjoint P ϕ ∗ = Id − R p R ϕ : H L → H L , ϕ P_{\phi}^{*}=\operatorname{Id}-R_{p}R_{\phi}:\mathcal{H}_{L}\to\mathcal{H}_{L,\phi} P ϕ ∗ = Id − R p R ϕ : H L → H L , ϕ . The three objects that we shall use repeatedly are the following.
The Gram operator , which is symmetric and p p p -conditionally positive-definite, is defined as
A ϕ = P ϕ ∗ A P ϕ : S → S ′ . (6) A_{\phi}=P_{\phi}^{*}AP_{\phi}:S\to S'. \tag*{(6)} A ϕ = P ϕ ∗ A P ϕ : S → S ′ . (6)
Its augmentation A + = A ϕ + R p A_{+}=A_{\phi}+R_{p} A + = A ϕ + R p is positive-definite.
The conditioned inverse is the unique right-inverse of L L L that imposes the boundary conditions ϕ ( f ) = 0 \phi(f)=0 ϕ ( f ) = 0 . It is given by
L ϕ − 1 = A ϕ L ∗ : L 2 → S ′ , (7) L_{\phi}^{-1}=A_{\phi}L^{*}:L_{2}\to S', \tag*{(7)} L ϕ − 1 = A ϕ L ∗ : L 2 → S ′ , (7)
and it factorizes the Gram operator as (see [59 Proposition 5.5] )
A ϕ = L ϕ − 1 L ϕ − 1 ∗ . (8) A_{\phi}=L_{\phi}^{-1}L_{\phi}^{-1*}. \tag*{(8)} A ϕ = L ϕ − 1 L ϕ − 1 ∗ . (8)
The native space H L \mathcal{H}_{L} H L [60 Theorem 3.1] is the Hilbert–Gelfand space equipped with the inner product
⟨ f , g ⟩ H L = ⟨ L f , L g ⟩ L 2 + ∑ n = 1 N 0 ⟨ ϕ n , f ⟩ ⟨ ϕ n , g ⟩ . (9) \langle f,g\rangle_{\mathcal{H}_{L}}=\langle Lf,Lg\rangle_{L_{2}}+\sum_{n=1}^{N_{0}}\langle\phi_{n},f\rangle\langle\phi_{n},g\rangle. \tag*{(9)} ⟨ f , g ⟩ H L = ⟨ L f , Lg ⟩ L 2 + n = 1 ∑ N 0 ⟨ ϕ n , f ⟩ ⟨ ϕ n , g ⟩ . (9)
Consequently, H L = ( S , ∥ ⋅ ∥ H L ) ‾ \mathcal{H}_{L}=\overline{(S,\|\cdot\|_{\mathcal{H}_{L}})} H L = ( S , ∥ ⋅ ∥ H L ) and it decomposes as H L = H L , ϕ ⊕ N p \mathcal{H}_{L}=\mathcal{H}_{L,\phi}\oplus N_{p} H L = H L , ϕ ⊕ N p with H L , ϕ = { f ∈ H L : ϕ ( f ) = 0 } = L ϕ − 1 ( L 2 ) \mathcal{H}_{L,\phi}=\{f\in\mathcal{H}_{L}:\phi(f)=0\}=L_{\phi}^{-1}(L_{2}) H L , ϕ = { f ∈ H L : ϕ ( f ) = 0 } = L ϕ − 1 ( L 2 ) , while its dual H L ′ = ( S , ∥ ⋅ ∥ H L ′ ) ‾ \mathcal{H}'_{L}=\overline{(S,\|\cdot\|_{\mathcal{H}'_{L}})} H L ′ = ( S , ∥ ⋅ ∥ H L ′ ) , with ∥ ϕ ∥ H L ′ = ⟨ A + ϕ , ϕ ⟩ \|\phi\|_{\mathcal{H}'_{L}}=\langle A_{+}\phi,\phi\rangle ∥ ϕ ∥ H L ′ = ⟨ A + ϕ , ϕ ⟩ , decomposes as H L ′ = H A ⊕ N ϕ \mathcal{H}'_{L}=\mathcal{H}_{A}\oplus N_{\phi} H L ′ = H A ⊕ N ϕ with H A = L ∗ ( L 2 ) = { ν ∈ H L ′ : p ( ν ) = 0 } \mathcal{H}_{A}=L^{*}(L_{2})=\{\nu\in\mathcal{H}'_{L}:p(\nu)=0\} H A = L ∗ ( L 2 ) = { ν ∈ H L ′ : p ( ν ) = 0 } . The Riesz map from H A \mathcal{H}_{A} H A onto H L , ϕ \mathcal{H}_{L,\phi} H L , ϕ is A ϕ A_{\phi} A ϕ , so that
⟨ ν , υ ⟩ H L ′ = ⟨ A ϕ ν , υ ⟩ for all ν , υ ∈ H A . (10) \langle\nu,\upsilon\rangle_{\mathcal{H}'_{L}}=\langle A_{\phi}\nu,\upsilon\rangle\qquad\text{for all }\nu,\upsilon\in\mathcal{H}_{A}. \tag*{(10)} ⟨ ν , υ ⟩ H L ′ = ⟨ A ϕ ν , υ ⟩ for all ν , υ ∈ H A . (10)
We refer to H A = L ∗ ( L 2 ) \mathcal{H}_{A}=L^{*}(L_{2}) H A = L ∗ ( L 2 ) as the core of the construction: it is the only member of the family that is independent of the boundary conditions ϕ \phi ϕ , from which all the others are generated once ϕ \phi ϕ is selected, and it is the space that carries the second-order structure of the process (see Proposition 3.5 , which shows that H A \mathcal{H}_{A} H A is congruent to the Gaussian space G G \mathcal{G}_{G} G G ).
The theme of the present paper is that the Gram operator A ϕ A_{\phi} A ϕ plays a double role. In the variational theory of [59 , 60 ] , it is the Riesz map of the core H A \mathcal{H}_{A} H A , which induces the inner product (10). Here, it is the covariance operator of a generalized Gaussian process G ∼ N ( 0 , A ϕ ) G\sim\mathcal{N}(0,A_{\phi}) G ∼ N ( 0 , A ϕ ) (Theorem 3.2 ), so that (10) becomes the identity ⟨ ν , υ ⟩ H L ′ = Cov G ( ν , υ ) \langle\nu,\upsilon\rangle_{\mathcal{H}'_{L}}=\operatorname{Cov}_{G}(\nu,\upsilon) ⟨ ν , υ ⟩ H L ′ = Cov G ( ν , υ ) and every term of the variational construction acquires a stochastic counterpart. Table 1 summarizes the correspondence, whose mathematical content is the congruence of Proposition 3.5 .
Probability Measures on S ′ S^{\prime} S ′ : The Minlos Theorem
The stochastic part of our story requires probability measures on infinite-dimensional topological vector spaces. The adequate device to specify them is the characteristic functional P G : S → C \mathcal{P}_{G}: S \to\mathbb{C} P G : S → C with P G ( φ ) = E { e i ⟨ G , φ ⟩ } \mathcal{P}_{G}(\varphi) = \mathbb{E}\{e^{i\langle G,\varphi\rangle}\} P G ( φ ) = E { e i ⟨ G , φ ⟩ } , which is the infinite-dimensional generalization of the characteristic function (14) of a random vector. In finite dimensions, Bochner’s theorem asserts that the characteristic functions on R N \mathbb{R}^{N} R N are exactly the continuous positive-definite functions that are normalized at the origin, and that they are in one-to-one correspondence with the probability measures on R N \mathbb{R}^{N} R N . The fundamental result of Minlos [37 ] extends this correspondence to any nuclear space, with the measure being supported by the dual [15 ] .
Theorem 2.3 (Minlos–Bochner). Let S S S be a real nuclear locally convex space and S ′ S^{\prime} S ′ its topological dual (with the cylindrical σ \sigma σ -algebra). Let P \mathcal{P} P :
S → C \mathcal{S} \to\mathbb{C} S → C be a map that satisfies the three defining properties of a characteristic functional:
Normalization: P ^ ( 0 ) = 1 \widehat{\mathcal{P}}(0)=1 P ( 0 ) = 1 ,
Positive-definiteness: For all N ∈ N N \in\mathbb{N} N ∈ N , φ 1 , … , φ N ∈ S \varphi_1,\ldots,\varphi_N \in\mathcal{S} φ 1 , … , φ N ∈ S and c 1 , … , c N ∈ C c_1,\ldots,c_N \in\mathbb{C} c 1 , … , c N ∈ C ,
∑ m = 1 N ∑ n = 1 N c m c n ‾ P ^ ( φ m − φ n ) ≥ 0 , (11) \sum_{m=1}^{N}\sum_{n=1}^{N}c_m\overline{c_n}\widehat{\mathcal{P}}(\varphi_m-\varphi_n)\geq0, \tag*{(11)} m = 1 ∑ N n = 1 ∑ N c m c n P ( φ m − φ n ) ≥ 0 , (11)
Continuity: P ^ \widehat{\mathcal{P}} P is continuous at 0 0 0 .
Then, there exists a unique probability measure P \mathcal{P} P on S ′ \mathcal{S}' S ′ such that
P ^ ( φ ) = ∫ S ′ e i ⟨ ω , φ ⟩ d P ( ω ) , φ ∈ S . (12) \widehat{\mathcal{P}}(\varphi)=\int_{\mathcal{S}'}e^{\mathrm{i}\langle\omega,\varphi\rangle}\,\mathrm{d}\mathcal{P}(\omega),\qquad\varphi\in\mathcal{S}. \tag*{(12)} P ( φ ) = ∫ S ′ e i ⟨ ω , φ ⟩ d P ( ω ) , φ ∈ S . (12)
Conversely, every probability measure on S ′ \mathcal{S}' S ′ has a characteristic functional that satisfies (1)–(3).
An important complement is that the three defining properties in Theorem 2.3 imply that P ^ \widehat{\mathcal{P}} P is continuous on S \mathcal{S} S , Hermitian symmetric (i.e., P ^ ( φ ) = P ^ ( − φ ) \widehat{\mathcal{P}}(\varphi)=\widehat{\mathcal{P}}(-\varphi) P ( φ ) = P ( − φ ) ), and such that 0 ≤ ∣ P ^ ( φ ) ∣ ≤ P ^ ( 0 ) = 1 0\leq\lvert\widehat{\mathcal{P}}(\varphi)\rvert\leq\widehat{\mathcal{P}}(0)=1 0 ≤ ∣ P ( φ )∣ ≤ P ( 0 ) = 1 for any φ ∈ S \varphi\in\mathcal{S} φ ∈ S . The first of these implications explains why the third defining hypothesis is often replaced by the more natural requirement that P ^ \widehat{\mathcal{P}} P be continuous on S \mathcal{S} S .
Let us explain why the dual of a nuclear space—rather than a Hilbert space—is the correct home for such measures. The naive transposition of Bochner’s theorem to an infinite-dimensional Hilbert space fails, and it fails on the most important example: the functional
P ^ W ( φ ) = e − 1 2 ∥ φ ∥ L 2 2 , (13) \widehat{\mathcal{P}}_{W}(\varphi)=e^{-\frac{1}{2}\lVert\varphi\rVert_{L^{2}}^{2}}, \tag*{(13)} P W ( φ ) = e − 2 1 ∥ φ ∥ L 2 2 , (13)
which is continuous and positive-definite on L 2 L_{2} L 2 , and which is the natural candidate for the characteristic functional of a white Gaussian noise with correlation E { ⟨ W , φ ⟩ ⟨ W , ψ ⟩ } = ⟨ φ , ψ ⟩ \mathbb{E}\{\langle W,\varphi\rangle\langle W,\psi\rangle\}=\langle\varphi,\psi\rangle E {⟨ W , φ ⟩ ⟨ W , ψ ⟩} = ⟨ φ , ψ ⟩ . Were W W W a random element of L 2 L_{2} L 2 , its coefficients ξ k = ⟨ W , e k ⟩ \xi_k=\langle W,e_k\rangle ξ k = ⟨ W , e k ⟩ in an orthonormal basis ( e k ) k ∈ N (e_k)_{k\in\mathbb{N}} ( e k ) k ∈ N would be i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) , so that ∥ W ∥ L 2 2 = ∑ k ∣ ξ k ∣ 2 = + ∞ \lVert W\rVert_{L^{2}}^{2}=\sum_k\lvert\xi_k\rvert^{2}=+\infty ∥ W ∥ L 2 2 = ∑ k ∣ ξ k ∣ 2 = + ∞ almost surely: the candidate measure assigns zero mass to the very space that is supposed to support it. Sazonov’s theorem [7 , 46 ] pinpoints the obstruction: on a Hilbert space, a Gaussian functional e − 1 2 ⟨ Q φ , φ ⟩ e^{-\frac{1}{2}\langle Q\varphi,\varphi\rangle} e − 2 1 ⟨ Qφ , φ ⟩ is the characteristic functional of a countably additive measure if and only if its covariance Q Q Q is trace-class, which the identity operator in (13) is not in infinite dimensions.
The theorem of Minlos resolves the difficulty by enlarging the target space. When S \mathcal{S} S is nuclear, the canonical maps between the Hilbertian seminorm completions that define its topology are themselves nuclear (trace-class), which means that the correction demanded by Sazonov is built into the topology of S \mathcal{S} S . Consequently, the mere continuity of P ^ \widehat{\mathcal{P}} P on S \mathcal{S} S suffices to produce a probability measure, albeit on the larger space S ′ \mathcal{S}' S ′ . In particular, white Gaussian noise exists as a generalized process with realizations in S ′ \mathcal{S}' S ′ , but P W ( L 2 ) = 0 \mathcal{P}_{W}(L_{2})=0 P W ( L 2 ) = 0 . This is the division of labor within the sandwich S → d H L ↪ S ′ \mathcal{S}\xrightarrow{d}\mathcal{H}_{L}\hookrightarrow\mathcal{S}' S d H L ↪ S ′ : the native Hilbert space hosts the deterministic reconstruction, while S ′ \mathcal{S}' S ′ hosts the stochastic model, whose realizations almost surely fall outside the native space H L \mathcal{H}_{L} H L (and, for white noise, outside L 2 L_{2} L 2 ).
Generalized Gaussian Processes
Abstract Generalized Gaussian Processes
The definition of abstract Gaussian processes is motivated by two important observations: the first is that any linear functional of a Gaussian random vector is a scalar Gaussian random variable. The second is that the characteristic function of the Gaussian random vector X ∼ N ( μ X , C X ) \boldsymbol{X}\sim\mathcal{N}(\boldsymbol{\mu}_{X},C_{X}) X ∼ N ( μ X , C X ) with mean vector μ X ∈ R N \boldsymbol{\mu}_{X}\in\mathbb{R}^{N} μ X ∈ R N and covariance matrix C X ∈ R N × N C_{X}\in\mathbb{R}^{N\times N} C X ∈ R N × N is given by
p ^ X ( ξ ) = E { e i ⟨ ξ , X ⟩ } = exp ( i ⟨ μ X , ξ ⟩ − 1 2 ⟨ C X ξ , ξ ⟩ ) . (14) \widehat{p}_{X}(\boldsymbol{\xi})=\mathbb{E}\{e^{i\langle\boldsymbol{\xi},\boldsymbol{X}\rangle}\}=\exp\left(i\langle\boldsymbol{\mu}_{X},\boldsymbol{\xi}\rangle-\frac{1}{2}\langle C_{X}\boldsymbol{\xi},\boldsymbol{\xi}\rangle\right). \tag*{(14)} p X ( ξ ) = E { e i ⟨ ξ , X ⟩ } = exp ( i ⟨ μ X , ξ ⟩ − 2 1 ⟨ C X ξ , ξ ⟩ ) . (14)
This function is the conjugate Fourier transform of the corresponding multivariate Gaussian probability density function (pdf).
Generalized Gaussian processes take one more step of abstraction than the classical random functions: they are defined either as random elements of the dual S ′ \mathcal{S}' S ′ of a nuclear space S \mathcal{S} S , or as random linear functionals acting on S \mathcal{S} S . The choice S = S ( R d ) \mathcal{S}=\mathcal{S}(\mathbb{R}^{d}) S = S ( R d ) is the one of interest here, but the framework covers every Gaussian object of interest, down to random vectors (S = R N \mathcal{S}=\mathbb{R}^{N} S = R N ) and discrete signals (S = S ( Z ) \mathcal{S}=\mathcal{S}(\mathbb{Z}) S = S ( Z ) ).
To explain the concept, we recall that a generalized function g ∈ S ′ g\in\mathcal{S}' g ∈ S ′ is nothing but a rule that associates the real number g ( φ ) = ⟨ g , φ ⟩ g(\varphi)=\langle g,\varphi\rangle g ( φ ) = ⟨ g , φ ⟩ with every test function φ ∈ S \varphi\in\mathcal{S} φ ∈ S , linearly and continuously. A generalized stochastic process randomizes this description [4 , 9 , 15 ] , which can be done from two complementary points of view: one either randomizes the object g g g , or the rule itself.
Definition 3.1 (Generalized stochastic process). A generalized stochastic process in S ′ \mathcal{S}' S ′ admits the two following equivalent descriptions.
Random element of S ′ \mathcal{S}' S ′ (randomization of the object): G G G is an S ′ \mathcal{S}' S ′ -valued random variable G : Ω → S ′ G:\Omega\to\mathcal{S}' G : Ω → S ′ that is defined on some probability space ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) ; equivalently, it is specified by a probability measure P G \mathbb{P}_{G} P G on S ′ \mathcal{S}' S ′ , according to which the process produces its realizations g = G ( ω ) ∈ S ′ g=G(\omega)\in\mathcal{S}' g = G ( ω ) ∈ S ′ , ω ∈ Ω \omega\in\Omega ω ∈ Ω .
Random linear functional on S \mathcal{S} S (randomization of the rule): G = { G ( φ ) } φ ∈ S G=\{G(\varphi)\}_{\varphi\in\mathcal{S}} G = { G ( φ ) } φ ∈ S is a collection of real-valued random variables indexed by S \mathcal{S} S such that
(i) (linearity) G ( a 1 φ 1 + a 2 φ 2 ) = a 1 G ( φ 1 ) + a 2 G ( φ 2 ) G(a_{1}\varphi_{1}+a_{2}\varphi_{2})=a_{1}G(\varphi_{1})+a_{2}G(\varphi_{2}) G ( a 1 φ 1 + a 2 φ 2 ) = a 1 G ( φ 1 ) + a 2 G ( φ 2 ) almost surely, for any φ 1 , φ 2 ∈ S \varphi_{1},\varphi_{2}\in\mathcal{S} φ 1 , φ 2 ∈ S and a 1 , a 2 ∈ R a_{1},a_{2}\in\mathbb{R} a 1 , a 2 ∈ R ;
(ii) (consistency) the joint laws of the vectors ( G ( φ 1 ) , … , G ( φ N ) ) (G(\varphi_{1}),\ldots,G(\varphi_{N})) ( G ( φ 1 ) , … , G ( φ N )) , with φ 1 , … , φ N ∈ S \varphi_{1},\ldots,\varphi_{N}\in\mathcal{S} φ 1 , … , φ N ∈ S and N ∈ N N\in\mathbb{N} N ∈ N , fit together coherently, in the sense that they agree with each other under reordering and under restriction to subsets;
(iii) (continuity) φ k → φ \varphi_{k}\to\varphi φ k → φ in S \mathcal{S} S implies that G ( φ k ) → G ( φ ) G(\varphi_{k})\to G(\varphi) G ( φ k ) → G ( φ ) in probability.
The process is Gaussian —a generalized Gaussian process (GGP)—if the random vector Y = ( G ( φ 1 ) , … , G ( φ N ) ) Y=(G(\varphi_{1}),\ldots,G(\varphi_{N})) Y = ( G ( φ 1 ) , … , G ( φ N )) has a multivariate Gaussian distribution for every finite collection φ 1 , … , φ N ∈ S \varphi_{1},\ldots,\varphi_{N}\in\mathcal{S} φ 1 , … , φ N ∈ S .
The direction 1 ⇒ 2 1\Rightarrow2 1 ⇒ 2 of the equivalence is elementary: the variables G ( φ ) = ⟨ G , φ ⟩ G(\varphi)=\langle G,\varphi\rangle G ( φ ) = ⟨ G , φ ⟩ inherit the linearity and the continuity from the realizations pointwise (hence almost surely and in probability), and their joint laws are automatically coherent, because they are all marginals of the common measure P G \mathbb{P}_{G} P G . The converse is the fundamental benefit of the nuclear setting. By the Kolmogorov extension theorem, the coherent family of Description 2 specifies a cylindrical measure, whose characteristic functional φ ↦ E { e i G ( φ ) } \varphi\mapsto\mathbb{E}\{e^{iG(\varphi)}\} φ ↦ E { e i G ( φ ) } is normalized, positive-definite (by Item (i)), as in (18) , and continuous on S \mathcal{S} S (by Item (iii) and dominated convergence); the Minlos–Bochner theorem (Theorem 2.3) then upgrades that cylindrical measure to a genuine probability measure P G \mathbb{P}_{G} P G on S ′ \mathcal{S}' S ′ with the same finite-dimensional laws [15 ] . In that case, one may always take the canonical probability space ( Ω , F , P ) = ( S ′ , B , P G ) (\Omega,\mathcal{F},\mathbb{P})=(\mathcal{S}',\mathcal{B},\mathbb{P}_{G}) ( Ω , F , P ) = ( S ′ , B , P G ) , where B \mathcal{B} B is the cylindrical σ \sigma σ -field, with the tautological action G ( φ ) : ω ↦ ⟨ ω , φ ⟩ G(\varphi):\omega\mapsto\langle\omega,\varphi\rangle G ( φ ) : ω ↦ ⟨ ω , φ ⟩ . Description 2 is the operative one in this paper: it is the form in which we construct the processes in Theorem 3.2, and the one whose domain we extend beyond S S S in § 3.2 . Description 1 is what makes statements about the realizations meaningful, including the characterization of their MMSE estimators in § 5 .
Since Gaussian random variables have a finite variance, all the observations G ( φ ) G(\varphi) G ( φ ) of a GGP live in the Hilbert space L 2 ( Ω , P ) L_{2}(\Omega,\mathcal{P}) L 2 ( Ω , P ) of (equivalence classes of) random variables with finite second moments over the probability space ( Ω , F , P ) (\Omega,\mathcal{F},\mathcal{P}) ( Ω , F , P ) of Definition 3.1, which is equipped with the inner product ⟨ X , Y ⟩ L 2 ( Ω , P ) = E { X Y } \langle X,Y\rangle_{L_{2}(\Omega,\mathcal{P})}=\mathbb{E}\{XY\} ⟨ X , Y ⟩ L 2 ( Ω , P ) = E { X Y } and in which convergence is convergence in the quadratic mean. This space—not to be confused with L 2 = L 2 ( R d ) L_{2}=L_{2}(\mathbb{R}^{d}) L 2 = L 2 ( R d ) —is the ambient structure for the second-order theory of the paper: the extension of § 3.2 is obtained by L 2 ( Ω , P ) L_{2}(\Omega,\mathcal{P}) L 2 ( Ω , P ) -continuity, and the Gaussian space of Proposition 3.5 is one of its closed subspaces.
As in the finite-dimensional setting, a GGP is characterized by its first and second-order moments, with the difference that the mean μ = E { G } ∈ S ′ \mu=\mathbb{E}\{G\}\in S' μ = E { G } ∈ S ′ is now a continuous linear functional on S S S , while the “covariance matrix” takes the form of the symmetric positive-semi-definite operator Q : S → S ′ \mathbf{Q}:S\to S' Q : S → S ′ . [15 ] These enable the determination of the means and covariances of G G G according to the rules
μ ( φ ) = E { G ( φ ) } = ⟨ μ , φ ⟩ \mu(\varphi)=\mathbb{E}\{G(\varphi)\}=\langle\mu,\varphi\rangle μ ( φ ) = E { G ( φ )} = ⟨ μ , φ ⟩
Cov G ( φ 1 , φ 2 ) = E { ( G ( φ 1 ) − μ ( φ 1 ) ) ( G ( φ 2 ) − μ ( φ 2 ) ) } = ⟨ Q φ 1 , φ 2 ⟩ . \operatorname{Cov}_{G}(\varphi_{1},\varphi_{2})=\mathbb{E}\left\{(G(\varphi_{1})-\mu(\varphi_{1}))(G(\varphi_{2})-\mu(\varphi_{2}))\right\}=\langle\mathbf{Q}\varphi_{1},\varphi_{2}\rangle. Cov G ( φ 1 , φ 2 ) = E { ( G ( φ 1 ) − μ ( φ 1 )) ( G ( φ 2 ) − μ ( φ 2 )) } = ⟨ Q φ 1 , φ 2 ⟩ .
The corresponding generalized Gaussian process is denoted G ∼ N ( μ , Q ) G\sim\mathcal{N}(\mu,\mathbf{Q}) G ∼ N ( μ , Q ) , in direct analogy with the finite-dimensional convention.
In the next theorem, which is due to Gelfand and Vilenkin [15 ] , we recall that such a generalized Gaussian process exists for any μ ∈ S ′ \mu\in S' μ ∈ S ′ and symmetric, positive semi-definite operator Q \mathbf{Q} Q , and that it is characterized explicitly by its characteristic functional on S S S —the infinite-dimensional counterpart of (14) . We state the result in the form that the rest of the paper relies upon, and we include its short proof, on which Theorem 3.3 builds. In particular, the generalized Gaussian process W ∼ N ( 0 , Id ) W\sim\mathcal{N}(0,\operatorname{Id}) W ∼ N ( 0 , Id ) is called a white noise or Gaussian innovation .
Theorem 3.2 (Generalized Gaussian process in S ′ S' S ′ [15 ] ). For any given μ ∈ S ′ \mu\in S' μ ∈ S ′ (mean) and symmetric, positive semi-definite operator Q : S → S ′ \mathbf{Q}:S\to S' Q : S → S ′ (covariance), there exists a corresponding generalized Gaussian process G ∼ N ( μ , Q ) G\sim\mathcal{N}(\mu,\mathbf{Q}) G ∼ N ( μ , Q ) on S S S whose characteristic functional is
P ^ G ( φ ) = E { e i G ( φ ) } = exp ( i ⟨ μ , φ ⟩ − 1 2 ⟨ Q φ , φ ⟩ ) . (15) \widehat{\mathcal{P}}_{G}(\varphi)=\mathbb{E}\left\{\mathrm{e}^{\mathrm{i}G(\varphi)}\right\}=\exp\left(\mathrm{i}\langle\mu,\varphi\rangle-\frac{1}{2}\langle\mathbf{Q}\varphi,\varphi\rangle\right). \tag*{(15)} P G ( φ ) = E { e i G ( φ ) } = exp ( i ⟨ μ , φ ⟩ − 2 1 ⟨ Q φ , φ ⟩ ) . (15)
The latter specifies a Gaussian probability measure P G \mathcal{P}_{G} P G on S ′ \mathcal{S}' S ′ that induces compatible families of multivariate Gaussian distributions: For each finite set { φ 1 , … , φ N } ⊂ S \{\varphi_{1}, \ldots, \varphi_{N}\} \subset\mathcal{S} { φ 1 , … , φ N } ⊂ S , we have that Y = ( G ( φ 1 ) , … , G ( φ N ) ) ∼ N ( μ Y , C Y ) Y = (G(\varphi_{1}), \ldots, G(\varphi_{N})) \sim\mathcal{N}(\boldsymbol{\mu}_{Y}, \mathbf{C}_{Y}) Y = ( G ( φ 1 ) , … , G ( φ N )) ∼ N ( μ Y , C Y ) , whose mean vector μ Y ∈ R N \boldsymbol{\mu}_{Y} \in\mathbb{R}^{N} μ Y ∈ R N and covariance matrix C Y ∈ R N × N \mathbf{C}_{Y} \in\mathbb{R}^{N \times N} C Y ∈ R N × N are given by
[ μ Y ] n = ⟨ μ , φ n ⟩ (16) [\boldsymbol{\mu}_{Y}]_{n} = \langle\mu, \varphi_{n} \rangle \tag*{(16)} [ μ Y ] n = ⟨ μ , φ n ⟩ (16)
[ C Y ] m , n = Cov G ( φ m , φ n ) = ⟨ Q φ m , φ n ⟩ . (17) [\mathbf{C}_{Y}]_{m,n} = \operatorname{Cov}_{G}(\varphi_{m}, \varphi_{n}) = \langle Q\varphi_{m}, \varphi_{n} \rangle. \tag*{(17)} [ C Y ] m , n = Cov G ( φ m , φ n ) = ⟨ Q φ m , φ n ⟩ . (17)
Proof. By the Minlos–Bochner theorem (Theorem 2.3), it suffices to verify that the right-hand side of (15) is normalized, positive-definite (in the sense of (11)), and continuous on S \mathcal{S} S . The normalization holds by inspection. The continuity holds because φ k → φ \varphi_{k} \to\varphi φ k → φ in S \mathcal{S} S implies that ⟨ μ , φ k ⟩ → ⟨ μ , φ ⟩ \langle\mu, \varphi_{k} \rangle\to\langle\mu, \varphi\rangle ⟨ μ , φ k ⟩ → ⟨ μ , φ ⟩ and that Q φ k → Q φ Q\varphi_{k} \to Q\varphi Q φ k → Qφ in S ′ \mathcal{S}' S ′ , while the duality pairing is hypocontinuous, so that ⟨ Q φ k , φ k ⟩ → ⟨ Q φ , φ ⟩ \langle Q\varphi_{k}, \varphi_{k} \rangle\to\langle Q\varphi, \varphi\rangle ⟨ Q φ k , φ k ⟩ → ⟨ Qφ , φ ⟩ . As for the positive-definiteness, it only involves finitely many test functions at a time. Given φ 1 , … , φ N ∈ S \varphi_{1}, \ldots, \varphi_{N} \in\mathcal{S} φ 1 , … , φ N ∈ S , the vector μ Y \boldsymbol{\mu}_{Y} μ Y and the matrix C Y \mathbf{C}_{Y} C Y of (16)–(17) specify a Gaussian vector Y ∼ N ( μ Y , C Y ) Y \sim\mathcal{N}(\boldsymbol{\mu}_{Y}, \mathbf{C}_{Y}) Y ∼ N ( μ Y , C Y ) , which exists because C Y \mathbf{C}_{Y} C Y is symmetric and positive semi-definite (degenerate if it is singular). By the bilinearity of the duality product, ⟨ μ , φ m − φ n ⟩ = [ μ Y ] m − [ μ Y ] n \langle\mu, \varphi_{m} - \varphi_{n} \rangle= [\boldsymbol{\mu}_{Y}]_{m} - [\boldsymbol{\mu}_{Y}]_{n} ⟨ μ , φ m − φ n ⟩ = [ μ Y ] m − [ μ Y ] n and ⟨ Q ( φ m − φ n ) , φ m − φ n ⟩ = ( e m − e n ) T C Y ( e m − e n ) \langle Q(\varphi_{m} - \varphi_{n}), \varphi_{m} - \varphi_{n} \rangle= (\mathbf{e}_{m} - \mathbf{e}_{n})^{\mathrm{T}}\mathbf{C}_{Y}(\mathbf{e}_{m} - \mathbf{e}_{n}) ⟨ Q ( φ m − φ n ) , φ m − φ n ⟩ = ( e m − e n ) T C Y ( e m − e n ) , so that P G ( φ m − φ n ) = E { e i ( Y m − Y n ) } \mathcal{P}_{G}(\varphi_{m} - \varphi_{n}) = \mathbb{E}\{e^{i(Y_{m}-Y_{n})}\} P G ( φ m − φ n ) = E { e i ( Y m − Y n ) } by (14). Consequently, for all c 1 , … , c N ∈ C c_{1}, \ldots, c_{N} \in\mathbb{C} c 1 , … , c N ∈ C ,
∑ m = 1 N ∑ n = 1 N c m c n ‾ P ^ G ( φ m − φ n ) = E { ∣ ∑ m = 1 N c m e i Y m ∣ 2 } ≥ 0 , (18) \sum_{m=1}^{N}\sum_{n=1}^{N} c_{m}\overline{c_{n}}\,\widehat{\mathcal{P}}_{G}(\varphi_{m}-\varphi_{n}) = \mathbb{E}\left\{\left|\sum_{m=1}^{N} c_{m}e^{iY_{m}}\right|^{2}\right\} \ge0, \tag*{(18)} m = 1 ∑ N n = 1 ∑ N c m c n P G ( φ m − φ n ) = E ⎩ ⎨ ⎧ m = 1 ∑ N c m e i Y m 2 ⎭ ⎬ ⎫ ≥ 0 , (18)
which proves the existence of the measure P G \mathcal{P}_{G} P G . Finally, the characteristic function of ( G ( φ 1 ) , … , G ( φ N ) ) (G(\varphi_{1}), \ldots, G(\varphi_{N})) ( G ( φ 1 ) , … , G ( φ N )) at ξ ∈ R N \xi\in\mathbb{R}^{N} ξ ∈ R N is, by the linearity of G G G ,
E { e i G ( ∑ n ξ n φ n ) } = P ^ G ( ∑ n ξ n φ n ) = exp ( i μ Y T ξ − 1 2 ξ T C Y ξ ) , \mathbb{E}\{e^{iG(\sum_{n}\xi_{n}\varphi_{n})}\} = \widehat{\mathcal{P}}_{G}\left(\sum_{n}\xi_{n}\varphi_{n}\right) = \exp\left(i\boldsymbol{\mu}_{Y}^{\mathrm{T}}\xi- \frac{1}{2}\xi^{\mathrm{T}}\mathbf{C}_{Y}\xi\right), E { e i G ( ∑ n ξ n φ n ) } = P G ( n ∑ ξ n φ n ) = exp ( i μ Y T ξ − 2 1 ξ T C Y ξ ) ,
which is (14); that is, Y ∼ N ( μ Y , C Y ) Y \sim\mathcal{N}(\boldsymbol{\mu}_{Y}, \mathbf{C}_{Y}) Y ∼ N ( μ Y , C Y ) .
Theorem 3.2 only requires the covariance operator to be positive semi-definite. We call the GGP proper when Q Q Q is (strictly) positive-definite, and degenerate (or improper) otherwise, in which case the measure P G \mathcal{P}_{G} P G is supported on a proper subspace of S ′ \mathcal{S}' S ′ .
Gaussian Processes Acting on Hilbert–Gelfand Spaces
We now investigate the possibility of extending the action of a generalized Gaussian process in S ′ \mathcal{S}' S ′ to test functions φ \varphi φ that may lie outside S \mathcal{S} S . This will be achieved by establishing the connection with the Hilbert–Gelfand spaces of [59 , 60 ] , as recalled in § 2.2 . The relevant class of covariance operators is that of the Gram operators A ϕ A_{\phi} A ϕ of (6), whose augmentation A + = A ϕ + R p A_{+}=A_{\phi}+R_{p} A + = A ϕ + R p underlies the dual pair ( H L ′ , H L ) (\mathcal{H}^{\prime}_{L},\mathcal{H}_{L}) ( H L ′ , H L ) , with H L ′ = H A ⊕ N ϕ \mathcal{H}^{\prime}_{L}=\mathcal{H}_{A}\oplus\mathcal{N}_{\phi} H L ′ = H A ⊕ N ϕ . Note that this setting also covers the simpler non-degenerate case with p = { 0 } \mathfrak{p}=\{0\} p = { 0 } , which corresponds to the dual pair of Hilbert–Gelfand spaces ( H L ′ , H L ) (\mathcal{H}^{\prime}_{L},\mathcal{H}_{L}) ( H L ′ , H L ) defined in [59 Theorem 2.5] .
Theorem 3.3 (GGP linked to a Hilbert–Gelfand space). Let A : S → S ′ A:\mathcal{S}\to\mathcal{S}^{\prime} A : S → S ′ be a continuous linear operator that is p \mathfrak{p} p -conditionally positive-definite with corresponding A ϕ : H L ′ → H L A_{\phi}:\mathcal{H}^{\prime}_{L}\to\mathcal{H}_{L} A ϕ : H L ′ → H L as recalled in § 2.2 . Then, the characteristic functional
φ ↦ P ^ G ( φ ) = exp ( − 1 2 ⟨ A ϕ φ , φ ⟩ ) , (19) \varphi\mapsto\widehat{P}_{G}(\varphi)=\exp\left(-\frac{1}{2}\langle A_{\phi}\varphi,\varphi\rangle\right), \tag*{(19)} φ ↦ P G ( φ ) = exp ( − 2 1 ⟨ A ϕ φ , φ ⟩ ) , (19) specifies a generalized Gaussian process G ∼ N ( 0 , A ϕ ) G\sim\mathcal{N}(0,A_{\phi}) G ∼ N ( 0 , A ϕ ) in S ′ \mathcal{S}^{\prime} S ′ whose action is extendable to test functions φ ∈ H L ′ ⊃ S \varphi\in\mathcal{H}^{\prime}_{L}\supset\mathcal{S} φ ∈ H L ′ ⊃ S where H L ′ = H A ⊕ N ϕ \mathcal{H}^{\prime}_{L}=\mathcal{H}_{A}\oplus\mathcal{N}_{\phi} H L ′ = H A ⊕ N ϕ is the Hilbert space induced by the augmented operator A + = A ϕ + R p A_{+}=A_{\phi}+R_{p} A + = A ϕ + R p . Specifically, for each finite set { φ 1 , … , φ N } ⊂ H L ′ \{\varphi_{1},\ldots,\varphi_{N}\}\subset\mathcal{H}^{\prime}_{L} { φ 1 , … , φ N } ⊂ H L ′ , this process induces compatible families of random Gaussian vectors Y = ( G ( φ 1 ) , … , G ( φ N ) ) ∼ N ( 0 , C Y ) \mathbf{Y}=(G(\varphi_{1}),\ldots,G(\varphi_{N}))\sim\mathcal{N}(0,C_{\mathbf{Y}}) Y = ( G ( φ 1 ) , … , G ( φ N )) ∼ N ( 0 , C Y ) whose covariance matrices C Y ∈ R N × N C_{\mathbf{Y}}\in\mathbb{R}^{N\times N} C Y ∈ R N × N are given by
[ C Y ] m , n = ⟨ A ϕ φ m , φ n ⟩ = ⟨ A ϕ φ ~ m , φ ~ n ⟩ = ⟨ A φ ~ m , φ ~ n ⟩ (20) [C_{\mathbf{Y}}]_{m,n}=\langle A_{\phi}\varphi_{m},\varphi_{n}\rangle=\langle A_{\phi}\widetilde{\varphi}_{m},\widetilde{\varphi}_{n}\rangle=\langle A\widetilde{\varphi}_{m},\widetilde{\varphi}_{n}\rangle \tag*{(20)} [ C Y ] m , n = ⟨ A ϕ φ m , φ n ⟩ = ⟨ A ϕ φ m , φ n ⟩ = ⟨ A φ m , φ n ⟩ (20) where φ ~ n = P ϕ { φ n } ∈ H A \widetilde{\varphi}_{n}=P_{\phi}\{\varphi_{n}\}\in\mathcal{H}_{A} φ n = P ϕ { φ n } ∈ H A with P ϕ = Id − R ϕ R p P_{\phi}=\operatorname{Id}-R_{\phi}R_{p} P ϕ = Id − R ϕ R p . Moreover, Y = Y ~ = ( G ( φ ~ 1 ) , … , G ( φ ~ N ) ) \mathbf{Y}=\widetilde{\mathbf{Y}}=(G(\widetilde{\varphi}_{1}),\ldots,G(\widetilde{\varphi}_{N})) Y = Y = ( G ( φ 1 ) , … , G ( φ N )) almost surely, which also indicates that G G G vanishes on N ϕ \mathcal{N}_{\phi} N ϕ (degeneracy).
Finally, the matrices C Y = C Y ~ C_{\mathbf{Y}}=C_{\widetilde{\mathbf{Y}}} C Y = C Y are positive-definite if and only if the corresponding φ ~ 1 , … , φ ~ N ∈ H A \widetilde{\varphi}_{1},\ldots,\widetilde{\varphi}_{N}\in\mathcal{H}_{A} φ 1 , … , φ N ∈ H A are linearly independent. For the non-degenerate scenario p = { 0 } \mathfrak{p}=\{0\} p = { 0 } , the latter property holds for any combination of linearly independent φ 1 , … , φ N ∈ H L ′ ⊃ S \varphi_{1},\ldots,\varphi_{N}\in\mathcal{H}^{\prime}_{L}\supset\mathcal{S} φ 1 , … , φ N ∈ H L ′ ⊃ S .
Proof. Step 1: existence: The Gram operator A ϕ : S → S ′ A_{\phi}:\mathcal{S}\to\mathcal{S}^{\prime} A ϕ : S → S ′ is continuous and symmetric, and it is positive semi-definite because ⟨ A ϕ φ , φ ⟩ = ∥ L ϕ − 1 ∗ φ ∥ L 2 2 ≥ 0 \langle A_{\phi}\varphi,\varphi\rangle=\lVert L_{\phi}^{-1*}\varphi\rVert_{L_{2}}^{2}\ge0 ⟨ A ϕ φ , φ ⟩ = ∥ L ϕ − 1 ∗ φ ∥ L 2 2 ≥ 0 by (8). Theorem 3.2 with μ = 0 \mu=0 μ = 0 and Q = A ϕ Q=A_{\phi} Q = A ϕ therefore provides the GGP G ∼ N ( 0 , A ϕ ) G\sim\mathcal{N}(0,A_{\phi}) G ∼ N ( 0 , A ϕ ) in S ′ \mathcal{S}^{\prime} S ′ with characteristic functional (19), whose finite-dimensional distributions over S \mathcal{S} S are the zero-mean Gaussians with covariance [ C Y ] m , n = ⟨ A ϕ φ m , φ n ⟩ [C_{\mathbf{Y}}]_{m,n}=\langle A_{\phi}\varphi_{m},\varphi_{n}\rangle [ C Y ] m , n = ⟨ A ϕ φ m , φ n ⟩ , which is the first expression in (20).
Step 2: the algebra of the conditioned covariance: The key property here is that ( φ , ψ ) ↦ ⟨ A ϕ φ , ψ ⟩ (\varphi,\psi)\mapsto\langle A_{\phi}\varphi,\psi\rangle ( φ , ψ ) ↦ ⟨ A ϕ φ , ψ ⟩ specifies a semi-inner product on H L ′ \mathcal{H}^{\prime}_{L} H L ′ (resp., an inner product on H A \mathcal{H}_A H A ). Moreover, for all φ ~ , ψ ~ ∈ H A = { ν ∈ H L ′ : p ( ν ) = 0 } \tilde{\varphi},\tilde{\psi}\in\mathcal{H}_A=\{\nu\in\mathcal{H}'_L:p(\nu)=0\} φ ~ , ψ ~ ∈ H A = { ν ∈ H L ′ : p ( ν ) = 0 } , we have that
⟨ A ϕ φ ~ , ψ ~ ⟩ = ⟨ A φ ~ , ψ ~ ⟩ for any ϕ ⊂ H L ′ s.t. ⟨ p m , ϕ n ⟩ = δ m , n , (21) \langle A_\phi\tilde{\varphi},\tilde{\psi}\rangle=\langle A\tilde{\varphi},\tilde{\psi}\rangle\quad\text{for any }\phi\subset\mathcal{H}'_L\text{ s.t. }\langle p_m,\phi_n\rangle=\delta_{m,n}, \tag*{(21)} ⟨ A ϕ φ ~ , ψ ~ ⟩ = ⟨ A φ ~ , ψ ~ ⟩ for any ϕ ⊂ H L ′ s.t. ⟨ p m , ϕ n ⟩ = δ m , n , (21)
which is a direct consequence of the definition (6) of the Gram operator, together with the property that P ϕ P_\phi P ϕ acts as the identity on H A = { ν ∈ H L ′ : p ( ν ) = 0 } \mathcal{H}_A=\{\nu\in\mathcal{H}'_L:p(\nu)=0\} H A = { ν ∈ H L ′ : p ( ν ) = 0 } (see (5) ).
Step 3: extension to H L ′ \mathcal{H}'_L H L ′ : The extension is obtained by continuity in the quadratic mean. For φ ∈ S \varphi\in\mathcal{S} φ ∈ S , the variance of the Gaussian random variable G ( φ ) G(\varphi) G ( φ ) is E { ∣ G ( φ ) ∣ 2 } = ⟨ A ϕ φ , φ ⟩ ≤ ⟨ A + φ , φ ⟩ = ∥ φ ∥ H L ′ 2 \mathbb{E}\{|G(\varphi)|^2\}=\langle A_\phi\varphi,\varphi\rangle\leq\langle A_+\varphi,\varphi\rangle=\|\varphi\|^2_{\mathcal{H}'_L} E { ∣ G ( φ ) ∣ 2 } = ⟨ A ϕ φ , φ ⟩ ≤ ⟨ A + φ , φ ⟩ = ∥ φ ∥ H L ′ 2 , where the inequality holds because ⟨ R p φ , φ ⟩ = ∣ p ( φ ) ∣ 2 ≥ 0 \langle R_p\varphi,\varphi\rangle=|p(\varphi)|^2\geq0 ⟨ R p φ , φ ⟩ = ∣ p ( φ ) ∣ 2 ≥ 0 . The map φ ↦ G ( φ ) \varphi\mapsto G(\varphi) φ ↦ G ( φ ) is therefore a linear contraction from ( S , ∥ ⋅ ∥ H L ′ ) (\mathcal{S},\|\cdot\|_{\mathcal{H}'_L}) ( S , ∥ ⋅ ∥ H L ′ ) into L 2 ( Ω , F ) L_2(\Omega,\mathcal{F}) L 2 ( Ω , F ) ; that is, a linear map that does not increase the norm, with equality when p ( φ ) = 0 p(\varphi)=0 p ( φ ) = 0 . Since H L ′ \mathcal{H}'_L H L ′ is the completion of S \mathcal{S} S for that norm (Section 2.2), the map extends uniquely to a linear contraction H L ′ → L 2 ( Ω , F ) \mathcal{H}'_L\to L_2(\Omega,\mathcal{F}) H L ′ → L 2 ( Ω , F ) , and the covariance identity (20) passes to the limit, with its right-hand side given by (21) . The Gaussianity of the finite-dimensional distributions is preserved as well, because a limit in the quadratic mean of jointly Gaussian vectors is jointly Gaussian.
Step 4: degeneracy and non-degeneracy: The direct-sum structure of H L ′ \mathcal{H}'_L H L ′ implies that every φ ∈ H L ′ \varphi\in\mathcal{H}'_L φ ∈ H L ′ has a unique decomposition as φ = φ ~ + ϕ \varphi=\tilde{\varphi}+\phi φ = φ ~ + ϕ with φ ~ = P ϕ { φ } ∈ H A \tilde{\varphi}=P_\phi\{\varphi\}\in\mathcal{H}_A φ ~ = P ϕ { φ } ∈ H A and ϕ = φ − φ ~ ∈ N ϕ \phi=\varphi-\tilde{\varphi}\in\mathcal{N}_\phi ϕ = φ − φ ~ ∈ N ϕ . By linearity of the GGP, this yields Y = G ( φ ) = G ( φ ~ ) + G ( ϕ ) Y=G(\varphi)=G(\tilde{\varphi})+G(\phi) Y = G ( φ ) = G ( φ ~ ) + G ( ϕ ) where G ( φ ~ ) = Y ~ ∼ N ( 0 , σ Y 2 ) G(\tilde{\varphi})=\tilde{Y}\sim\mathcal{N}(0,\sigma_Y^2) G ( φ ~ ) = Y ~ ∼ N ( 0 , σ Y 2 ) with σ Y 2 = ⟨ A ϕ φ , φ ⟩ \sigma_Y^2=\langle A_\phi\varphi,\varphi\rangle σ Y 2 = ⟨ A ϕ φ , φ ⟩ (from (20) ) and G ( ϕ ) = N ( 0 , 0 ) G(\phi)=\mathcal{N}(0,0) G ( ϕ ) = N ( 0 , 0 ) since ⟨ A ϕ ϕ , ϕ ⟩ = 0 \langle A_\phi\phi,\phi\rangle=0 ⟨ A ϕ ϕ , ϕ ⟩ = 0 for all ϕ ∈ N ϕ \phi\in\mathcal{N}_\phi ϕ ∈ N ϕ . The probabilistic interpretation of these properties is Y = Y ~ Y=\tilde{Y} Y = Y ~ and G ( ϕ ) = 0 G(\phi)=0 G ( ϕ ) = 0 (almost surely) for all ( φ , ϕ ) ∈ H L ′ × N ϕ (\varphi,\phi)\in\mathcal{H}'_L\times\mathcal{N}_\phi ( φ , ϕ ) ∈ H L ′ × N ϕ . The final statement about the positive-definiteness of C Y C_Y C Y follows from the positive-definiteness of A ϕ A_\phi A ϕ on H A \mathcal{H}_A H A , since z T C Y z = ⟨ A ϕ { ∑ n z n φ ~ n } , ∑ n z n φ ~ n ⟩ z^{\mathsf{T}}C_Yz=\langle A_\phi\{\sum_n z_n\tilde{\varphi}_n\},\sum_n z_n\tilde{\varphi}_n\rangle z T C Y z = ⟨ A ϕ { ∑ n z n φ ~ n } , ∑ n z n φ ~ n ⟩ vanishes exactly when ∑ n z n φ ~ n = 0 \sum_n z_n\tilde{\varphi}_n=0 ∑ n z n φ ~ n = 0 .
The functional connection with Hilbert spaces in Theorem 3.3 should not be over-interpreted: the fact that G ( φ ) G(\varphi) G ( φ ) is a well-defined random variable for every φ ∈ H L ′ \varphi\in\mathcal{H}'_L φ ∈ H L ′ does not make G G G a random process in the predual H L \mathcal{H}_L H L , nor does it turn P G \mathcal{P}_G P G into a measure on H L \mathcal{H}_L H L —the white noise of Section 2.3 is again the witness. The rigorous home of these processes remains the nuclear dual pair ( S , S ′ ) (\mathcal{S},\mathcal{S}') ( S , S ′ ) , even when Hilbert spaces sit in the background.
An attractive aspect of the extended framework of Theorem 3.3 is the possibility of applying a broader class of linear transformations. This calls for a slight adaptation of the basic definition (for regular operators) from the theory of generalized stochastic processes, which is founded on the Minlos-Bochner theorem.
Definition 3.4 (Linear transformation of a generalized stochastic process). Consider the following setting:
a Hilbert–Gelfand space H \mathcal{H} H with S → d H → d S ′ \mathcal{S} \xrightarrow{d} \mathcal{H} \xrightarrow{d} \mathcal{S}' S d H d S ′ ;
a continuous linear operator T : H → S ′ \mathrm{T} : \mathcal{H} \to\mathcal{S}' T : H → S ′ and its adjoint T ∗ : S → H ′ \mathrm{T}^{*} : \mathcal{S} \to\mathcal{H}' T ∗ : S → H ′ ;
a generalized stochastic process Z \mathcal{Z} Z in S ′ \mathcal{S}' S ′ whose characteristic form P ^ Z : S → C \widehat{\mathcal{P}}_{\mathcal{Z}} : \mathcal{S} \to\mathbb{C} P Z : S → C has a continuous and positive-definite extension on H ′ \mathcal{H}' H ′ .
Then, the transformed process T { Z } \mathrm{T}\{\mathcal{Z}\} T { Z } is the generalized stochastic process in S ′ \mathcal{S}' S ′ whose characteristic functional is P ^ T { Z } ( φ ) = P ^ Z ( T ∗ φ ) \widehat{\mathcal{P}}_{\mathrm{T}\{\mathcal{Z}\}}(\varphi) = \widehat{\mathcal{P}}_{\mathcal{Z}}(\mathrm{T}^{*}\varphi) P T { Z } ( φ ) = P Z ( T ∗ φ ) . For the class of regular operators, the pair of spaces ( H , H ′ ) (\mathcal{H},\mathcal{H}') ( H , H ′ ) can be substituted by ( S ′ , S ) (\mathcal{S}',\mathcal{S}) ( S ′ , S ) with T ∗ : S → S \mathrm{T}^{*} : \mathcal{S} \to\mathcal{S} T ∗ : S → S .
In the same vein, the application of a regular operator L : S ′ → S ′ \mathrm{L} : \mathcal{S}' \to\mathcal{S}' L : S ′ → S ′ to a generic GGP G ∼ N ( μ , Q ) G \sim\mathcal{N}(\mu,Q) G ∼ N ( μ , Q ) yields a transformed process L { G } ∼ N ( L μ , L Q L ∗ ) \mathrm{L}\{G\} \sim\mathcal{N}(\mathrm{L}\mu,\mathrm{LQ}\mathrm{L}^{*}) L { G } ∼ N ( L μ , LQ L ∗ ) . The latter is a GGP in S ′ \mathcal{S}' S ′ with mean L μ ∈ S ′ \mathrm{L}\mu\in\mathcal{S}' L μ ∈ S ′ and covariance operator L Q L ∗ : S → S ′ \mathrm{LQ}\mathrm{L}^{*} : \mathcal{S} \to\mathcal{S}' LQ L ∗ : S → S ′ , with the transformation rules being the same as in finite dimension.
Innovation Models and Gaussian Solutions of SDEs
We shall now use the operator-theoretic tools of § 2.2 to identify the important subclass of GGP that admit an innovation representation. We first consider a proper GGP in S ′ \mathcal{S}' S ′ whose positive-definite covariance operator is factorizable as A = T T ∗ A = \mathrm{T}\mathrm{T}^{*} A = T T ∗ , with T : L 2 → H L ↪ S ′ \mathrm{T} : L_{2} \to\mathcal{H}_{L} \hookrightarrow\mathcal{S}' T : L 2 → H L ↪ S ′ . Let us assume that the factor T \mathrm{T} T is invertible, with an inverse L = T − 1 \mathrm{L} = \mathrm{T}^{-1} L = T − 1 that is admissible with p = { 0 } p = \{0\} p = { 0 } (§ 2.2 ); the positive-definiteness of ( T T ∗ ) : S → S ′ (\mathrm{T}\mathrm{T}^{*}) : \mathcal{S} \to\mathcal{S}' ( T T ∗ ) : S → S ′ is then automatic, since ⟨ T T ∗ φ , φ ⟩ = ∥ T ∗ φ ∥ L 2 2 \langle\mathrm{T}\mathrm{T}^{*}\varphi,\varphi\rangle= \lVert\mathrm{T}^{*}\varphi\rVert_{L_{2}}^{2} ⟨ T T ∗ φ , φ ⟩ = ∥ T ∗ φ ∥ L 2 2 vanishes only when φ = 0 \varphi= 0 φ = 0 .
We now show that such a process can be synthesized by applying the linear transformation T = L − 1 \mathrm{T} = \mathrm{L}^{-1} T = L − 1 to the Gaussian white noise (or innovation process) W ∼ N ( 0 , I d ) W \sim\mathcal{N}(0,\mathrm{Id}) W ∼ N ( 0 , Id ) . In our formalism, this generation mechanism is expressed as
G ( φ ) = T { W } ( φ ) = ⟨ T { W } , φ ⟩ = ⟨ W , T ∗ { φ } ⟩ = W ( T ∗ φ ) , (22) G(\varphi) = \mathrm{T}\{W\}(\varphi) = \langle\mathrm{T}\{W\},\varphi\rangle= \langle W,\mathrm{T}^{*}\{\varphi\}\rangle= W(\mathrm{T}^{*}\varphi), \tag*{(22)} G ( φ ) = T { W } ( φ ) = ⟨ T { W } , φ ⟩ = ⟨ W , T ∗ { φ }⟩ = W ( T ∗ φ ) , (22)
where T ∗ : ( S ′ ) ′ = S → ( L 2 ) ′ = L 2 \mathrm{T}^{*} : (\mathcal{S}')' = \mathcal{S} \to(L_{2})' = L_{2} T ∗ : ( S ′ ) ′ = S → ( L 2 ) ′ = L 2 is the adjoint of T : L 2 → S ′ \mathrm{T} : L_{2} \to\mathcal{S}' T : L 2 → S ′ . The continuity of T \mathrm{T} T together with the property that the domain of the white noise W W W is extendable to L 2 = ( L 2 ) ′ L_2 = (L_2)' L 2 = ( L 2 ) ′ (see § 3.2 ) ensure that the right-hand side of (22) is well-defined for any φ ∈ S \varphi\in\mathcal{S} φ ∈ S . This leads to the conclusion that
P ^ G ( φ ) = P ^ W ( T ∗ φ ) = exp ( − 1 2 ∥ T ∗ φ ∥ L 2 2 ) = exp ( − 1 2 ⟨ ( T T ∗ ) φ , φ ⟩ ) , \widehat{\mathcal{P}}_G(\varphi) = \widehat{\mathcal{P}}_W(T^*\varphi) = \exp\left(-\frac{1}{2}\lVert T^*\varphi\rVert_{L_2}^2\right) = \exp\left(-\frac{1}{2}\langle(TT^*)\varphi,\varphi\rangle\right), P G ( φ ) = P W ( T ∗ φ ) = exp ( − 2 1 ∥ T ∗ φ ∥ L 2 2 ) = exp ( − 2 1 ⟨( T T ∗ ) φ , φ ⟩ ) ,
which proves that G ∼ N ( 0 , A ) G \sim\mathcal{N}(0,A) G ∼ N ( 0 , A ) . We also observe that the factorization A = T T ∗ A = TT^* A = T T ∗ is not unique: there is a large equivalence class of possible transformations given by T ~ = T U \widetilde{T} = TU T = T U , where U U U is an arbitrary unitary operator L 2 → L 2 L_2 \to L_2 L 2 → L 2 with the property that U − 1 = U ∗ U^{-1} = U^* U − 1 = U ∗ .
Next, we move to the scenario where the “shaping” operator T = L φ − 1 T = L_\varphi^{-1} T = L φ − 1 is the distributional extension of the operator specified by (7) (right-inverse of a p p p -admissible L L L ). While this yields a generalized process G = L φ − 1 { W } G = L_\varphi^{-1}\{W\} G = L φ − 1 { W } whose covariance operator is only positive semi-definite, Theorem 3.3 guarantees that it is well-defined, with the p p p -conditionally positive-definite Gram operator A φ A_\varphi A φ of (6) as its covariance. To record that G G G is the unique GGP that is whitened by L L L and satisfies the boundary conditions φ ( G ) = 0 \varphi(G) = 0 φ ( G ) = 0 , we write its covariance operator as
C L , φ = L φ − 1 L φ − ∗ = A φ ∣ S : S → S ′ . (23) C_{L,\varphi} = L_\varphi^{-1}L_\varphi^{-*} = A_\varphi|_{\mathcal{S}} : \mathcal{S} \to\mathcal{S}'. \tag*{(23)} C L , φ = L φ − 1 L φ −∗ = A φ ∣ S : S → S ′ . (23)
and write G ∼ N ( 0 , C L , φ ) G \sim\mathcal{N}(0,C_{L,\varphi}) G ∼ N ( 0 , C L , φ ) .
The p p p -conditional-positiveness of C L , φ C_{L,\varphi} C L , φ translates into the GGP being degenerate in the sense that it vanishes almost surely on the non-trivial null space N φ \mathcal{N}_\varphi N φ of the extended operator A φ : H L ′ → H L A_\varphi: \mathcal{H}'_L \to\mathcal{H}_L A φ : H L ′ → H L (see Theorem 3.3 ). Yet, unlike degenerate scenarios in finite dimensions, G G G possesses an innovation W W W that can be recovered by applying the whitening operator L L L as in
L { G } : φ ↦ ⟨ L { G } , φ ⟩ = ⟨ L L φ − 1 { W } , φ ⟩ = W ( L φ − ∗ L ∗ φ ) = W ( φ ) , (24) L\{G\} : \varphi\mapsto\langle L\{G\},\varphi\rangle= \langle L L_\varphi^{-1}\{W\},\varphi\rangle= W(L_\varphi^{-*}L^*\varphi) = W(\varphi), \tag*{(24)} L { G } : φ ↦ ⟨ L { G } , φ ⟩ = ⟨ L L φ − 1 { W } , φ ⟩ = W ( L φ −∗ L ∗ φ ) = W ( φ ) , (24)
where the adjoint manipulation is legitimate because: (i) the adjoint operator L ∗ L^* L ∗ continuously maps S ↪ H L ′ → ( L 2 ) ′ = L 2 \mathcal{S} \hookrightarrow\mathcal{H}'_L \to(L_2)' = L_2 S ↪ H L ′ → ( L 2 ) ′ = L 2 (the extended domain of W W W ); and (ii) L φ − ∗ L ∗ = ( L L φ − 1 ) ∗ = Id L_\varphi^{-*}L^* = (LL_\varphi^{-1})^* = \operatorname{Id} L φ −∗ L ∗ = ( L L φ − 1 ) ∗ = Id on L 2 L_2 L 2 , because L φ − 1 L_\varphi^{-1} L φ − 1 is a right-inverse of $L ({xref}Section 2.2`). These properties imply that the process G G G , or rather its realizations g ∈ S ′ g \in\mathcal{S}' g ∈ S ′ , can also be specified as solutions of a linear stochastic differential equation (SDE) driven by a Gaussian white noise w ∈ S ′ w \in\mathcal{S}' w ∈ S ′ ; namely,
L { G } = W subject to φ ( G ) = 0 , (25) L\{G\} = W \qquad\text{subject to} \qquad\varphi(G) = 0, \tag*{(25)} L { G } = W subject to φ ( G ) = 0 , (25)
which is to be understood in the weak (or distributional) sense. The intuition there is that the ambiguity induced by the nontrivial null space of L L L gets resolved by imposing N 0 N_0 N 0 linear boundary conditions.
We are now in a position to state the property that ties the present paper to the variational theory of [59 , 60 ] : the second-order structure of G G G and the Hilbertian structure of the native space are one and the same, up to an isometry. We recall that the Gaussian space of a process is the closure, in quadratic mean, of the linear span of its observations [21 ] .
Proposition 3.5 (Congruence between the native space and the Gaussian space). *Let L L L be p p p -admissible and let G ∼ N ( 0 , C L , ϕ ) G \sim\mathcal{N}(0, C_{L,\phi}) G ∼ N ( 0 , C L , ϕ ) be the generalized Gaussian process that is whitened by L L L under the boundary conditions ϕ ( G ) = 0 \phi(G)=0 ϕ ( G ) = 0 , with Gaussian space
G G = span ‾ { G ( φ ) : φ ∈ S } ⊂ L 2 ( Ω , P ) . \mathcal{G}_{G} = \overline{\operatorname{span}}\{G(\varphi) : \varphi\in\mathcal{S}\} \subset L_{2}(\Omega,\mathcal{P}). G G = span { G ( φ ) : φ ∈ S } ⊂ L 2 ( Ω , P ) . The Gaussian space is thus intrinsic to the process as initially defined on S \mathcal{S} S (Definition 3.1 ); the same space is obtained by letting the argument range over the whole of H L ′ \mathcal{H}'_{L} H L ′ , because the extended variables of Theorem 3.3 are, by construction, quadratic-mean limits of variables G ( φ ) G(\varphi) G ( φ ) with φ ∈ S \varphi\in\mathcal{S} φ ∈ S . Then, the following properties hold.*
The map ν ↦ G ( ν ) \nu\mapsto G(\nu) ν ↦ G ( ν ) annihilates N ϕ \mathcal{N}_{\phi} N ϕ and factors through P ϕ \mathcal{P}_{\phi} P ϕ . Its restriction
U : H A → G G , U { ν } = G ( ν ) , (26) U:\mathcal{H}_{A}\to\mathcal{G}_{G},\qquad U\{\nu\}=G(\nu), \tag*{(26)} U : H A → G G , U { ν } = G ( ν ) , (26) is a unitary isomorphism, since E { G ( ν ) G ( υ ) } = ⟨ A ϕ ν , υ ⟩ = ⟨ ν , υ ⟩ H L ′ \mathbb{E}\{G(\nu)G(\upsilon)\}=\langle A_{\phi}\nu,\upsilon\rangle=\langle\nu,\upsilon\rangle_{\mathcal{H}'_{L}} E { G ( ν ) G ( υ )} = ⟨ A ϕ ν , υ ⟩ = ⟨ ν , υ ⟩ H L ′ for all ν , υ ∈ H A \nu,\upsilon\in\mathcal{H}_{A} ν , υ ∈ H A .
By composition with the Riesz map A ϕ : H A → H L , ϕ A_{\phi}:\mathcal{H}_{A}\to\mathcal{H}_{L,\phi} A ϕ : H A → H L , ϕ of § 2.2 , the primary native space H L , ϕ \mathcal{H}_{L,\phi} H L , ϕ is congruent to G G \mathcal{G}_{G} G G as well, with A ϕ { ν } ↦ G ( ν ) A_{\phi}\{\nu\}\mapsto G(\nu) A ϕ { ν } ↦ G ( ν ) .
Both spaces are congruent to L 2 L_{2} L 2 through the innovation, in the sense that G ( ν ) = W ( L ϕ − 1 ∗ ν ) G(\nu)=W(L_{\phi}^{-1*}\nu) G ( ν ) = W ( L ϕ − 1 ∗ ν ) and ∥ L ϕ − 1 ∗ ν ∥ L 2 = ∥ ν ∥ H L ′ \|L_{\phi}^{-1*}\nu\|_{L_{2}}=\|\nu\|_{\mathcal{H}'_{L}} ∥ L ϕ − 1 ∗ ν ∥ L 2 = ∥ ν ∥ H L ′ for every ν ∈ H A \nu\in\mathcal{H}_{A} ν ∈ H A .
Proof. Isometry: The covariance identity is (20) of Theorem 3.3 , restricted to H A \mathcal{H}_{A} H A and combined with the Riesz pairing (10). It shows that U U U is a linear isometry from H A \mathcal{H}_{A} H A into L 2 ( Ω , P ) L_{2}(\Omega,\mathcal{P}) L 2 ( Ω , P ) , the action of G G G on the whole of H L ′ \mathcal{H}'_{L} H L ′ being granted by Theorem 3.3 .
Surjectivity: Every φ ∈ S \varphi\in\mathcal{S} φ ∈ S decomposes as φ = P ϕ φ + ϕ \varphi=\mathcal{P}_{\phi}\varphi+\phi φ = P ϕ φ + ϕ with P ϕ φ ∈ H A \mathcal{P}_{\phi}\varphi\in\mathcal{H}_{A} P ϕ φ ∈ H A and ϕ ∈ N ϕ \phi\in\mathcal{N}_{\phi} ϕ ∈ N ϕ , while G ( ϕ ) = 0 G(\phi)=0 G ( ϕ ) = 0 almost surely by Theorem 3.3 . This yields G ( φ ) = G ( P ϕ φ ) = U { P ϕ φ } G(\varphi)=G(\mathcal{P}_{\phi}\varphi)=U\{\mathcal{P}_{\phi}\varphi\} G ( φ ) = G ( P ϕ φ ) = U { P ϕ φ } , so that the range of U U U contains the generating family { G ( φ ) : φ ∈ S } \{G(\varphi):\varphi\in\mathcal{S}\} { G ( φ ) : φ ∈ S } and is therefore dense in G G \mathcal{G}_{G} G G . That range is also closed, because U \mathcal{U} U is an isometry, and the two properties combine into surjectivity.
Innovation form: The identity G ( ν ) = W ( L ϕ − 1 ∗ ν ) G(\nu)=W(L_{\phi}^{-1*}\nu) G ( ν ) = W ( L ϕ − 1 ∗ ν ) is (22) with T = L ϕ − 1 \mathrm{T}=L_{\phi}^{-1} T = L ϕ − 1 , while the norm identity follows from the factorization (8), which gives
∥ L ϕ − 1 ∗ ν ∥ L 2 2 = ⟨ A ϕ ν , ν ⟩ . \left\|L_{\phi}^{-1*}\nu\right\|_{L^2}^2=\langle A_{\phi}\nu,\nu\rangle. L ϕ − 1 ∗ ν L 2 2 = ⟨ A ϕ ν , ν ⟩ .
Proposition 3.5 is the precise sense in which the deterministic and the stochastic constructions coincide: the Riesz structure that [60 ] attaches to L L L and the second-order structure that the present paper attaches to G G G are related by an isometry, while the innovation W W W provides an orthonormal coordinate system for both. In particular, the reproducing kernel of [60 ] , Section 4.2 is the covariance function of G G G , which recovers the classical correspondence of Loève, Parzen and Kailath [22 , 41 ] at the level of generality of the present framework. The pointwise form of that identity, together with the explicit kernel that the boundary conditions produce, is Theorem 4.2 .
While the formal description G = L ϕ − 1 { W } G=L_{\phi}^{-1}\{W\} G = L ϕ − 1 { W } of a GGP looks rather innocuous, it is a remarkably powerful alternative to the classic integral representation of Gaussian stochastic processes. We shall make our point by considering a famous instance of such a representation: Mandelbrot’s definition [34 ] of fractional Brownian motion (fBm) with Hurst index H ∈ ( 0 , 1 ) H\in(0,1) H ∈ ( 0 , 1 ) as
B H ( t ) = 1 Γ ( H + 1 2 ) ∫ R ( ( t − τ ) + H − 1 2 − ( − τ ) + H − 1 2 ) d B ( τ ) (27) B_H(t)=\frac{1}{\Gamma\left(H+\frac{1}{2}\right)}\int_{\mathbb{R}}\left((t-\tau)_+^{H-\frac{1}{2}}-(-\tau)_+^{H-\frac{1}{2}}\right)\,\mathrm{d}B(\tau) \tag*{(27)} B H ( t ) = Γ ( H + 2 1 ) 1 ∫ R ( ( t − τ ) + H − 2 1 − ( − τ ) + H − 2 1 ) d B ( τ ) (27)
where B ( ⋅ ) B(\cdot) B ( ⋅ ) is the Wiener process (considered as a random measure) and d B \mathrm{d}B d B the representation of Gaussian white noise in the classic notation of stochastic calculus. The equivalent description of fBm in the present framework is B H = L ϕ − 1 { W } B_H=L_{\phi}^{-1}\{W\} B H = L ϕ − 1 { W } with L = D γ L=D^\gamma L = D γ , γ = H + 1 2 ∈ ( 1 2 , 3 2 ) \gamma=H+\frac{1}{2}\in\left(\frac{1}{2},\frac{3}{2}\right) γ = H + 2 1 ∈ ( 2 1 , 2 3 ) , N 0 = 1 N_0=1 N 0 = 1 , and ( p 1 , ϕ 1 ) = ( 1 , δ ) (p_1,\phi_1)=(1,\delta) ( p 1 , ϕ 1 ) = ( 1 , δ ) . The operator D γ D^\gamma D γ is the fractional derivative of order γ \gamma γ with Fourier symbol ( i ω ) γ (\mathrm{i}\omega)^\gamma ( i ω ) γ , which induces fractional splines [58 ] . The relevant Green’s function can be found in [60 Table 1] . One readily verifies that the integral kernel in (27) is the Schwartz kernel of the fractional integral operator L ϕ − 1 = D δ − γ L_{\phi}^{-1}=D_{\delta}^{-\gamma} L ϕ − 1 = D δ − γ . This equivalence also yields a simpler description of fBm as the solution of the fractional SDE
D H + 1 2 { B H } = W s . t . B H ( 0 ) = 0 , (28) D^{H+\frac{1}{2}}\{B_H\}=W \quad\mathrm{s.t.}\quad B_H(0)=0, \tag*{(28)} D H + 2 1 { B H } = W s.t. B H ( 0 ) = 0 , (28)
which provides immediate insights on the fractal and regularity properties of these processes [6 ] . As expected, and in accordance with Theorem 4.2 , the correlation function of fBm coincides with the Schwartz kernel of the covariance operator:
E { B H ( t ) B H ( τ ) } = A ϕ { δ τ } ( t ) = σ H 2 2 ( ∣ t ∣ 2 H + ∣ τ ∣ 2 H − ∣ t − τ ∣ 2 H ) (29) \mathbb{E}\{B_H(t)B_H(\tau)\}=A_\phi\{\delta_\tau\}(t)=\frac{\sigma_H^2}{2}\left(|t|^{2H}+|\tau|^{2H}-|t-\tau|^{2H}\right) \tag*{(29)} E { B H ( t ) B H ( τ )} = A ϕ { δ τ } ( t ) = 2 σ H 2 ( ∣ t ∣ 2 H + ∣ τ ∣ 2 H − ∣ t − τ ∣ 2 H ) (29)
where σ H 2 = 1 sin ( π H ) Γ ( 2 H + 1 ) \sigma_H^2=\frac{1}{\sin(\pi H)\Gamma(2H+1)} σ H 2 = s i n ( π H ) Γ ( 2 H + 1 ) 1 is the normalization that corresponds to the Green’s function ρ L ( x ) = x γ − 1 Γ ( γ ) \rho_L(x)=\frac{x^{\gamma-1}}{\Gamma(\gamma)} ρ L ( x ) = Γ ( γ ) x γ − 1 of [60 Table 1] , with the case of Brownian motion being recovered for H = 1 2 H=\frac{1}{2} H = 2 1 with L = D L=D L = D . Appendix A carries out this computation for an arbitrary number N 0 N_0 N 0 of boundary conditions, which yields the family of higher-order fractional Brownian motions, while Section 4.3 transposes it to R d \mathbb{R}^d R d , which yields Lévy’s Brownian field and its fractional extensions.
Link with the Classical Theory
In this section, we take ( S , S ′ ) = ( S ( R d ) , S ′ ( R d ) ) (\mathcal{S},\mathcal{S}')=(\mathcal{S}(\mathbb{R}^d),\mathcal{S}'(\mathbb{R}^d)) ( S , S ′ ) = ( S ( R d ) , S ′ ( R d )) and make the link between our operator-theoretic formulation and the pointwise language of kernels, which is the one that is used in the theory of reproducing-kernel Hilbert spaces (RKHS) [1 , 3 , 42 , 45 ] , in the classical theory of Gaussian processes [25 , 33 , 44 , 62 ] , in scattered-data approximation, where the term “native space” originates [13 , 47 , 48 , 66 ] , and in kernel-based learning [38 , 49 , 52 ] ; the measure-theoretic side of the correspondence between kernels and integral operators is treated in [53 ] .
The basic objects of that language are kernels and the integral operators that they specify, which we define first, with the appropriate safeguards. To control growth, we introduce the space
C s g ( R d ) : = { f ∈ C ( R d ) : ∃ α ≥ 0 such that sup x ∈ R d ∣ f ( x ) ∣ ( 1 + ∥ x ∥ ) α < ∞ } (30) C_{\mathrm{sg}}(\mathbb{R}^d):=\left\{f\in C(\mathbb{R}^d):\ \exists\alpha\ge0\ \text{such that}\ \sup_{x\in\mathbb{R}^d}\frac{|f(x)|}{(1+\lVert x\rVert)^\alpha}<\infty\right\} \tag*{(30)} C sg ( R d ) := { f ∈ C ( R d ) : ∃ α ≥ 0 such that x ∈ R d sup ( 1 + ∥ x ∥ ) α ∣ f ( x ) ∣ < ∞ } (30)
of continuous functions of slow growth, with S ( R d ) ⊂ C s g ( R d ) ⊂ S ′ ( R d ) \mathcal{S}(\mathbb{R}^d)\subset C_{\mathrm{sg}}(\mathbb{R}^d)\subset\mathcal{S}'(\mathbb{R}^d) S ( R d ) ⊂ C sg ( R d ) ⊂ S ′ ( R d ) . Likewise, a bivariate function a : R d × R d → R a:\mathbb{R}^d\times\mathbb{R}^d\to\mathbb{R} a : R d × R d → R (referred to as a kernel) is said to be of slow growth if it is jointly continuous and if ∣ a ( x , y ) ∣ ≤ C ( 1 + ∥ x ∥ ) α ( 1 + ∥ y ∥ ) α |a(x,y)|\le C(1+\lVert x\rVert)^\alpha(1+\lVert y\rVert)^\alpha ∣ a ( x , y ) ∣ ≤ C ( 1 + ∥ x ∥ ) α ( 1 + ∥ y ∥ ) α for some C > 0 C>0 C > 0 and α ≥ 0 \alpha\ge0 α ≥ 0 . With any such kernel, we associate the integral operator
T a : φ ↦ T a { φ } = ∫ R d a ( ⋅ , y ) φ ( y ) d y . (31) T_a:\varphi\mapsto T_a\{\varphi\}=\int_{\mathbb{R}^d}a(\cdot,y)\,\varphi(y)\,\mathrm{d}y. \tag*{(31)} T a : φ ↦ T a { φ } = ∫ R d a ( ⋅ , y ) φ ( y ) d y . (31)
The stated conditions ensure that (31) is well-defined on S ( R d ) \mathcal{S}(\mathbb{R}^{d}) S ( R d ) : the rapid decay of φ \varphi φ against the polynomial bound on a a a makes the integral absolutely convergent for every x ∈ R d x \in\mathbb{R}^{d} x ∈ R d ; the joint continuity of a a a , combined with dominated convergence, yields T a { φ } ∈ C s g ( R d ) T_{a}\{\varphi\} \in C_{\mathrm{sg}}(\mathbb{R}^{d}) T a { φ } ∈ C sg ( R d ) (with the same exponent α \alpha α ); and the bound
∣ ⟨ T a { φ } , ψ ⟩ ∣ ≤ C ∫ R d ∫ R d ( 1 + ∥ x ∥ ) α ( 1 + ∥ y ∥ ) α ∣ ψ ( x ) ∣ ∣ φ ( y ) ∣ d x d y , \left|\langle T_{a}\{\varphi\},\psi\rangle\right| \le C \int_{\mathbb{R}^{d}}\int_{\mathbb{R}^{d}}(1+\lVert x\rVert)^{\alpha}(1+\lVert y\rVert)^{\alpha}|\psi(x)||\varphi(y)|\,\mathrm{d}x\mathrm{d}y, ∣ ⟨ T a { φ } , ψ ⟩ ∣ ≤ C ∫ R d ∫ R d ( 1 + ∥ x ∥ ) α ( 1 + ∥ y ∥ ) α ∣ ψ ( x ) ∣∣ φ ( y ) ∣ d x d y ,
whose right-hand side is controlled by finitely many Schwartz seminorms of φ \varphi φ and ψ \psi ψ , establishes the continuity of T a : S ( R d ) → S ′ ( R d ) T_{a}:\mathcal{S}(\mathbb{R}^{d}) \to\mathcal{S}'(\mathbb{R}^{d}) T a : S ( R d ) → S ′ ( R d ) . The same bounds allow the test function in (31) to be replaced by a measure with finite moments of order α \alpha α —in particular, by a finite atomic measure—so that the evaluations T a { δ y } = a ( ⋅ , y ) T_{a}\{\delta_{y}\}=a(\cdot,y) T a { δ y } = a ( ⋅ , y ) make sense pointwise.
The kernels of interest to us are the two Green’s functions attached to a p p p -admissible operator L L L . Specifically, we assume that the right inverse L − 1 L^{-1} L − 1 and the conditional inverse A A A of B = L ∗ L B=L^{*}L B = L ∗ L (Definition 2.2 ) admit the integral representations
L − 1 : φ ↦ ∫ R d ρ L ( ⋅ , y ) φ ( y ) d y , A : φ ↦ ∫ R d ρ B ( ⋅ , y ) φ ( y ) d y , (32) L^{-1}:\varphi\mapsto\int_{\mathbb{R}^{d}}\rho_{L}(\cdot,y)\varphi(y)\,\mathrm{d}y,\qquad A:\varphi\mapsto\int_{\mathbb{R}^{d}}\rho_{B}(\cdot,y)\varphi(y)\,\mathrm{d}y, \tag*{(32)} L − 1 : φ ↦ ∫ R d ρ L ( ⋅ , y ) φ ( y ) d y , A : φ ↦ ∫ R d ρ B ( ⋅ , y ) φ ( y ) d y , (32)
where ρ L \rho_{L} ρ L and ρ B \rho_{B} ρ B are the (canonical) Green’s functions of L L L and of the normal operator B = L ∗ L B=L^{*}L B = L ∗ L , respectively, and where ρ B \rho_{B} ρ B is symmetric with ρ B ( x , y ) = ρ B ( y , x ) \rho_{B}(x,y)=\rho_{B}(y,x) ρ B ( x , y ) = ρ B ( y , x ) . The representation of L − 1 L^{-1} L − 1 is understood in the sense of the Schwartz kernel theorem, with no regularity requirement on ρ L \rho_{L} ρ L . By contrast, whenever ρ B \rho_{B} ρ B is of slow growth, the conditional inverse A = T ρ B A=T_{\rho_{B}} A = T ρ B falls within the setting of (31) , so that the evaluations A { δ y } = ρ B ( ⋅ , y ) A\{\delta_{y}\}=\rho_{B}(\cdot,y) A { δ y } = ρ B ( ⋅ , y ) make sense pointwise; this is the hypothesis under which the pointwise statements of this section hold.
The first outcome is Theorem 4.2 , which identifies a single bivariate function a φ a_{\varphi} a φ as three objects at once: the Schwartz kernel (generalized impulse response) of the Gram operator A φ A_{\varphi} A φ of (6) ; the reproducing kernel of the primary native space H L , φ \mathcal{H}_{L,\varphi} H L , φ of § 2.2 ; and the covariance function of the generalized Gaussian process G ∼ N ( 0 , A φ ) G\sim\mathcal{N}(0,A_{\varphi}) G ∼ N ( 0 , A φ ) of Theorem 3.3 . It is therefore the pointwise counterpart of the congruence of Proposition 3.5 , and it is also the formula that [60 Section 4.2] leaves open: it identifies the reproducing kernel with the Green’s function ρ B \rho_{B} ρ B of the normal operator in the non-degenerate case N p = { 0 } \mathcal{N}_{p}=\{0\} N p = { 0 } only, whereas the boundary conditions ϕ ( f ) = 0 \phi(f)=0 ϕ ( f ) = 0 modify that kernel as soon as N 0 ≥ 1 N_{0}\ge1 N 0 ≥ 1 .
Definition 4.1 (Reproducing kernel). A Hilbert space H ↪ S ′ ( R d ) \mathcal{H}\hookrightarrow\mathcal{S}'(\mathbb{R}^{d}) H ↪ S ′ ( R d ) is a reproducing-kernel Hilbert space (RKHS) if it is a space of functions on which all the sampling functionals are continuous; that is, if δ x 0 ∈ H ′ \delta_{x_{0}}\in\mathcal{H}' δ x 0 ∈ H ′ for every x 0 ∈ R d x_{0}\in\mathbb{R}^{d} x 0 ∈ R d . The reproducing kernel of such a space is the unique function
h : R d × R d → R such that (i) h ( ⋅ , x 0 ) ∈ H for all x 0 ∈ R d ; (ii) f ( x 0 ) = ⟨ h ( ⋅ , x 0 ) , f ⟩ H for all f ∈ H and x 0 ∈ R d . (33) \begin{aligned}
h &: \mathbb{R}^{d} \times\mathbb{R}^{d} \to\mathbb{R}\ \text{such that}\\
\text{(i)}\quad&h(\cdot,x_{0}) \in\mathcal{H}\ \text{for all }x_{0} \in\mathbb{R}^{d};\\
\text{(ii)}\quad&f(x_{0})=\langle h(\cdot,x_{0}),f\rangle_{\mathcal{H}}\ \text{for all }f \in\mathcal{H}\ \text{and }x_{0} \in\mathbb{R}^{d}. \tag*{(33)}
\end{aligned} h (i) (ii) : R d × R d → R such that h ( ⋅ , x 0 ) ∈ H for all x 0 ∈ R d ; f ( x 0 ) = ⟨ h ( ⋅ , x 0 ) , f ⟩ H for all f ∈ H and x 0 ∈ R d . (33)
The two properties in (33) say that h ( ⋅ , x 0 ) h(\cdot,x_{0}) h ( ⋅ , x 0 ) is the Riesz representative of δ x 0 \delta_{x_{0}} δ x 0 , which is why the existence of the kernel and the continuity of the sampling functionals are equivalent.
Reproducing Kernel and Covariance Function
We now provide the pointwise transcription of the two descriptions of the native space that [60 ] provides: the conditioned parametrization A ϕ = P ϕ ∗ A P ϕ A_{\phi}=P_{\phi}^{*}AP_{\phi} A ϕ = P ϕ ∗ A P ϕ and the factorization A ϕ = L ϕ − 1 L ϕ − ∗ A_{\phi}=L_{\phi}^{-1}L_{\phi}^{-*} A ϕ = L ϕ − 1 L ϕ −∗ (Eqs. (6) and (8)). The first one produces the explicit formula, while the second one exhibits the kernel as a Gram (or autocorrelation) integral.
Theorem 4.2 (Reproducing kernel of the primary native space). Let L : S ( R d ) → L 2 ( R d ) L:\mathcal{S}(\mathbb{R}^{d})\to L_{2}(\mathbb{R}^{d}) L : S ( R d ) → L 2 ( R d ) be p p p -admissible in the sense of Section 2.2, and let ( p , ϕ ) (p,\phi) ( p , ϕ ) be a biorthogonal system with p ⊂ C s g ( R d ) p\subset C_{\mathrm{sg}}(\mathbb{R}^{d}) p ⊂ C sg ( R d ) and ϕ ⊂ H L ′ \phi\subset\mathcal{H}_{L}^{\prime} ϕ ⊂ H L ′ , as in Section 2.2, such that the boundary functionals can be paired with the Green’s functions; that is, such that ⟨ ϕ n , ρ L ( ⋅ , y ) ⟩ \langle\phi_{n},\rho_{L}(\cdot,y)\rangle ⟨ ϕ n , ρ L ( ⋅ , y )⟩ , ⟨ ϕ n , ρ B ( ⋅ , y ) ⟩ \langle\phi_{n},\rho_{B}(\cdot,y)\rangle ⟨ ϕ n , ρ B ( ⋅ , y )⟩ , and ⟨ ϕ m , A { ϕ n } ⟩ \langle\phi_{m},A\{\phi_{n}\}\rangle ⟨ ϕ m , A { ϕ n }⟩ are well-defined for all y ∈ R d y\in\mathbb{R}^{d} y ∈ R d . Then, the following properties hold, Items 2–4 under the additional hypothesis that the Green’s function ρ B \rho_{B} ρ B of (32) is of slow growth.
The conditioned inverse L ϕ − 1 = P ϕ ∗ L − 1 : L 2 ( R d ) → H L , ϕ L_{\phi}^{-1}=P_{\phi}^{*}L^{-1}:L_{2}(\mathbb{R}^{d})\to\mathcal{H}_{L,\phi} L ϕ − 1 = P ϕ ∗ L − 1 : L 2 ( R d ) → H L , ϕ (see (7)) is the integral operator with kernel
g ϕ ( x , y ) = ρ L ( x , y ) − ∑ n = 1 N 0 p n ( x ) q n ( y ) , q n = L − ∗ { ϕ n } , (34) g_{\phi}(x,y)=\rho_{L}(x,y)-\sum_{n=1}^{N_{0}}p_{n}(x)q_{n}(y),\qquad q_{n}=L^{-*}\{\phi_{n}\}, \tag*{(34)} g ϕ ( x , y ) = ρ L ( x , y ) − n = 1 ∑ N 0 p n ( x ) q n ( y ) , q n = L −∗ { ϕ n } , (34) where the functions q n ( y ) = ⟨ ϕ n , ρ L ( ⋅ , y ) ⟩ q_{n}(y)=\langle\phi_{n},\rho_{L}(\cdot,y)\rangle q n ( y ) = ⟨ ϕ n , ρ L ( ⋅ , y )⟩ are the images of the boundary functionals under the adjoint of the Green’s operator. No regularity of ρ L \rho_{L} ρ L is required here: (34) is an identity between Schwartz kernels, which is what allows it to cover the leading examples L = D L=\mathrm{D} L = D and L = D γ L=\mathrm{D}^{\gamma} L = D γ with γ < 1 \gamma<1 γ < 1 , whose Green’s functions are discontinuous at the diagonal.
The Gram operator A ϕ A_{\phi} A ϕ is the integral operator with the symmetric
kernel
a ϕ ( x , y ) = ρ B ( x , y ) − ∑ n = 1 N 0 p n ( x ) v n ( y ) − ∑ n = 1 N 0 v n ( x ) p n ( y ) + ∑ m = 1 N 0 ∑ n = 1 N 0 r m , n p m ( x ) p n ( y ) , (35) a_{\phi}(\mathbf{x},\mathbf{y})=\rho_{B}(\mathbf{x},\mathbf{y})-\sum_{n=1}^{N_0}p_n(\mathbf{x})v_n(\mathbf{y})-\sum_{n=1}^{N_0}v_n(\mathbf{x})p_n(\mathbf{y})+\sum_{m=1}^{N_0}\sum_{n=1}^{N_0}r_{m,n}p_m(\mathbf{x})p_n(\mathbf{y}), \tag*{(35)} a ϕ ( x , y ) = ρ B ( x , y ) − n = 1 ∑ N 0 p n ( x ) v n ( y ) − n = 1 ∑ N 0 v n ( x ) p n ( y ) + m = 1 ∑ N 0 n = 1 ∑ N 0 r m , n p m ( x ) p n ( y ) , (35)
where
v n = A { ϕ n } , i.e., v n ( y ) = ∫ R d ϕ n ( z ) ρ B ( z , y ) d z , (36) v_n=A\{\phi_n\},\qquad\text{i.e.,}\quad v_n(\mathbf{y})=\int_{\mathbb{R}^d}\phi_n(\mathbf{z})\rho_B(\mathbf{z},\mathbf{y})\,d\mathbf{z}, \tag*{(36)} v n = A { ϕ n } , i.e., v n ( y ) = ∫ R d ϕ n ( z ) ρ B ( z , y ) d z , (36)
r m , n = ⟨ ϕ m , A { ϕ n } ⟩ = ∫ R d ∫ R d ϕ m ( x ) ρ B ( x , y ) ϕ n ( y ) d x d y . (37) r_{m,n}=\langle\phi_m,A\{\phi_n\}\rangle=\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}\phi_m(\mathbf{x})\rho_B(\mathbf{x},\mathbf{y})\phi_n(\mathbf{y})\,d\mathbf{x}d\mathbf{y}. \tag*{(37)} r m , n = ⟨ ϕ m , A { ϕ n }⟩ = ∫ R d ∫ R d ϕ m ( x ) ρ B ( x , y ) ϕ n ( y ) d x d y . (37)
The three following properties are equivalent: (a) δ x ∈ H L ′ \delta_x\in\mathcal{H}'_L δ x ∈ H L ′ for every x ∈ R d x\in\mathbb{R}^d x ∈ R d ; (b) g ϕ ( x , ⋅ ) ∈ L 2 ( R d ) g_{\phi}(x,\cdot)\in L_2(\mathbb{R}^d) g ϕ ( x , ⋅ ) ∈ L 2 ( R d ) for every x ∈ R d x\in\mathbb{R}^d x ∈ R d ; and (c) the primary native space H L , ϕ \mathcal{H}_{L,\phi} H L , ϕ is an RKHS. When they hold, the reproducing kernel of H L , ϕ \mathcal{H}_{L,\phi} H L , ϕ is a ϕ a_{\phi} a ϕ , which is then also the autocorrelation of the kernel (34) ,
a ϕ ( x , y ) = ∫ R d g ϕ ( x , z ) g ϕ ( y , z ) d z = ⟨ g ϕ ( x , ⋅ ) , g ϕ ( y , ⋅ ) ⟩ L 2 , (38) a_{\phi}(\mathbf{x},\mathbf{y})=\int_{\mathbb{R}^d}g_{\phi}(\mathbf{x},\mathbf{z})g_{\phi}(\mathbf{y},\mathbf{z})\,d\mathbf{z}=\langle g_{\phi}(\mathbf{x},\cdot),g_{\phi}(\mathbf{y},\cdot)\rangle_{L_2}, \tag*{(38)} a ϕ ( x , y ) = ∫ R d g ϕ ( x , z ) g ϕ ( y , z ) d z = ⟨ g ϕ ( x , ⋅ ) , g ϕ ( y , ⋅ ) ⟩ L 2 , (38)
with a ϕ ( x , x ) = ∥ g ϕ ( x , ⋅ ) ∥ L 2 2 a_{\phi}(x,x)=\|g_{\phi}(x,\cdot)\|_{L_2}^2 a ϕ ( x , x ) = ∥ g ϕ ( x , ⋅ ) ∥ L 2 2 ; and the reproducing kernel of the full native space H L = H L , ϕ ⊕ N p \mathcal{H}_L=\mathcal{H}_{L,\phi}\oplus N_p H L = H L , ϕ ⊕ N p for the inner product (9) is a ϕ ( x , y ) + ∑ n = 1 N 0 p n ( x ) p n ( y ) a_{\phi}(x,y)+\sum_{n=1}^{N_0}p_n(x)p_n(y) a ϕ ( x , y ) + ∑ n = 1 N 0 p n ( x ) p n ( y ) .
Under the conditions of Item 3, the process G ∼ N ( 0 , A ϕ ) G\sim\mathcal{N}(0,A_{\phi}) G ∼ N ( 0 , A ϕ ) of Theorem 3.3 admits the pointwise reading G ( x ) : = G ( δ x ) G(x):=G(\delta_x) G ( x ) := G ( δ x ) , and a ϕ a_{\phi} a ϕ is its covariance function:
E { G ( x ) G ( y ) } = a ϕ ( x , y ) , ( x , y ) ∈ R d × R d . (39) \mathbb{E}\{G(x)G(y)\}=a_{\phi}(x,y),\qquad(x,y)\in\mathbb{R}^d\times\mathbb{R}^d. \tag*{(39)} E { G ( x ) G ( y )} = a ϕ ( x , y ) , ( x , y ) ∈ R d × R d . (39)
The pairing condition on ϕ \phi ϕ is automatic for ϕ ⊂ S ( R d ) \phi\subset\mathcal{S}(\mathbb{R}^d) ϕ ⊂ S ( R d ) , and it holds for the Dirac-type systems ϕ n = δ ( n − 1 ) \phi_n=\delta^{(n-1)} ϕ n = δ ( n − 1 ) and ϕ k = ∂ k δ \phi_k=\partial^k\delta ϕ k = ∂ k δ of Appendix A and of Section 4.3, whose action amounts to the point evaluation of the (sufficiently smooth) kernels and of their derivatives.
Proof. Kernel of the conditioned inverse (Item 1): By [60 Eq. (23)] , we have that L ϕ − 1 = P ϕ ∗ G ∗ = ( Id − R p R ϕ ) L − 1 L_{\phi}^{-1}=P_{\phi}^{*}G^{*}=(\operatorname{Id}-R_pR_{\phi})L^{-1} L ϕ − 1 = P ϕ ∗ G ∗ = ( Id − R p R ϕ ) L − 1 , so that P ϕ ∗ { f } = f − ∑ n = 1 N 0 p n ⟨ ϕ n , f ⟩ P_{\phi}^{*}\{f\}=f-\sum_{n=1}^{N_0}p_n\langle\phi_n,f\rangle P ϕ ∗ { f } = f − ∑ n = 1 N 0 p n ⟨ ϕ n , f ⟩ acts on the first argument of the kernel ρ L \rho_L ρ L of L − 1 L^{-1} L − 1 . Since ⟨ ϕ n , ρ L ( ⋅ , y ) ⟩ = \langle\phi_n,\rho_L(\cdot,y)\rangle= ⟨ ϕ n , ρ L ( ⋅ , y )⟩ = q n ( y ) q_n(y) q n ( y ) by the definition of q n q_n q n , this yields (34). The identity q n = L − 1 ∗ { ϕ n } q_n = L^{-1*}\{\phi_n\} q n = L − 1 ∗ { ϕ n } follows from the fact that the Schwartz kernel of an adjoint operator is the transpose of the original one.
Kernel of the Gram operator (Item 2): We evaluate a ϕ ( ⋅ , y ) = A ϕ { δ y } a_\phi(\cdot,y)=A_\phi\{\delta_y\} a ϕ ( ⋅ , y ) = A ϕ { δ y } by applying the three factors of A ϕ = P ϕ ∗ A P ϕ A_\phi=P_\phi^*AP_\phi A ϕ = P ϕ ∗ A P ϕ in succession. The projector P ϕ = Id − R ϕ R p P_\phi=\operatorname{Id}-R_\phi R_p P ϕ = Id − R ϕ R p gives P ϕ { δ y } = δ y − ∑ n = 1 N 0 p n ( y ) ϕ n P_\phi\{\delta_y\}=\delta_y-\sum_{n=1}^{N_0}p_n(y)\phi_n P ϕ { δ y } = δ y − ∑ n = 1 N 0 p n ( y ) ϕ n , whose image under A A A is ρ B ( ⋅ , y ) − ∑ n = 1 N 0 p n ( y ) v n \rho_B(\cdot,y)-\sum_{n=1}^{N_0}p_n(y)v_n ρ B ( ⋅ , y ) − ∑ n = 1 N 0 p n ( y ) v n with v n v_n v n given by (36). The symmetry of ρ B \rho_B ρ B yields ⟨ ϕ m , ρ B ( ⋅ , y ) ⟩ = v m ( y ) \langle\phi_m,\rho_B(\cdot,y)\rangle=v_m(y) ⟨ ϕ m , ρ B ( ⋅ , y )⟩ = v m ( y ) , while the definition (37) of r m , n r_{m,n} r m , n gives ⟨ ϕ m , v n ⟩ = r m , n \langle\phi_m,v_n\rangle=r_{m,n} ⟨ ϕ m , v n ⟩ = r m , n . The final application of P ϕ ∗ P_\phi^* P ϕ ∗ therefore produces
a ϕ ( ⋅ , y ) = ( ρ B ( ⋅ , y ) − ∑ n = 1 N 0 p n ( y ) v n ) − ∑ m = 1 N 0 p m ( v m ( y ) − ∑ n = 1 N 0 r m , n p n ( y ) ) , (40) a_\phi(\cdot,y)=\left(\rho_B(\cdot,y)-\sum_{n=1}^{N_0}p_n(y)v_n\right)-\sum_{m=1}^{N_0}p_m\left(v_m(y)-\sum_{n=1}^{N_0}r_{m,n}p_n(y)\right), \tag*{(40)} a ϕ ( ⋅ , y ) = ( ρ B ( ⋅ , y ) − n = 1 ∑ N 0 p n ( y ) v n ) − m = 1 ∑ N 0 p m ( v m ( y ) − n = 1 ∑ N 0 r m , n p n ( y ) ) , (40)
which is (35). That expression is symmetric in ( x , y ) (x,y) ( x , y ) because ρ B \rho_B ρ B is symmetric and because r m , n = r n , m r_{m,n}=r_{n,m} r m , n = r n , m , the operator A A A being symmetric.
Reproduction property (Item 3): Let us fix x ∈ R d x\in\mathbb{R}^d x ∈ R d . Since N ϕ ⊂ H ′ \mathcal{N}_\phi\subset\mathcal{H}' N ϕ ⊂ H ′ , the decomposition δ x = P ϕ { δ x } + ∑ n p n ( x ) ϕ n \delta_x=P_\phi\{\delta_x\}+\sum_n p_n(x)\phi_n δ x = P ϕ { δ x } + ∑ n p n ( x ) ϕ n shows that (a) is equivalent to P ϕ { δ x } ∈ H A = ( H L , ϕ ) ′ P_\phi\{\delta_x\}\in\mathcal{H}_A=(\mathcal{H}_{L,\phi})' P ϕ { δ x } ∈ H A = ( H L , ϕ ) ′ , which is precisely the continuity of the sampling functional on H L , ϕ \mathcal{H}_{L,\phi} H L , ϕ ; that is, (a) ⇔ \Leftrightarrow ⇔ (c) by Definition 4.1. To bring in (b), we parametrize H L , ϕ = L ϕ − 1 ( L 2 ) \mathcal{H}_{L,\phi}=L_\phi^{-1}(L_2) H L , ϕ = L ϕ − 1 ( L 2 ) by the innovation. Every f ∈ H L , ϕ f\in\mathcal{H}_{L,\phi} f ∈ H L , ϕ is f = L ϕ − 1 { w } f=L_\phi^{-1}\{w\} f = L ϕ − 1 { w } with w = L f ∈ L 2 ( R d ) w=Lf\in L_2(\mathbb{R}^d) w = L f ∈ L 2 ( R d ) and ∥ f ∥ H L , ϕ = ∥ w ∥ L 2 \|f\|_{\mathcal{H}_{L,\phi}}=\|w\|_{L_2} ∥ f ∥ H L , ϕ = ∥ w ∥ L 2 (by (9), since ϕ ( f ) = 0 \phi(f)=0 ϕ ( f ) = 0 ), so that Item 1 gives f ( x ) = ∫ R d g ϕ ( x , z ) w ( z ) d z f(x)=\int_{\mathbb{R}^d}g_\phi(x,z)w(z)\,dz f ( x ) = ∫ R d g ϕ ( x , z ) w ( z ) d z . The functional w ↦ f ( x ) w\mapsto f(x) w ↦ f ( x ) is therefore bounded on L 2 ( R d ) L_2(\mathbb{R}^d) L 2 ( R d ) if and only if g ϕ ( x , ⋅ ) ∈ L 2 ( R d ) g_\phi(x,\cdot)\in L_2(\mathbb{R}^d) g ϕ ( x , ⋅ ) ∈ L 2 ( R d ) , by the Riesz representation theorem, which is the equivalence of (b) and (c). Let us now assume that (a) holds and let f ∈ H L , ϕ f\in\mathcal{H}_{L,\phi} f ∈ H L , ϕ , so that ϕ ( f ) = 0 \phi(f)=0 ϕ ( f ) = 0 . The biorthogonality then gives ⟨ P ϕ { δ x } , f ⟩ = f ( x ) − ∑ n = 1 N 0 p n ( x ) ⟨ ϕ n , f ⟩ = f ( x ) \langle P_\phi\{\delta_x\},f\rangle=f(x)-\sum_{n=1}^{N_0}p_n(x)\langle\phi_n,f\rangle=f(x) ⟨ P ϕ { δ x } , f ⟩ = f ( x ) − ∑ n = 1 N 0 p n ( x ) ⟨ ϕ n , f ⟩ = f ( x ) , where P ϕ { δ x } ∈ H A P_\phi\{\delta_x\}\in\mathcal{H}_A P ϕ { δ x } ∈ H A . Since A ϕ : H A → H L , ϕ A_\phi:\mathcal{H}_A\to\mathcal{H}_{L,\phi} A ϕ : H A → H L , ϕ is the Riesz map of Section 2.2 (see (10)), and since A ϕ P ϕ = A ϕ A_\phi P_\phi=A_\phi A ϕ P ϕ = A ϕ by idempotence of the projector, this translates into
f ( x ) = ⟨ A ϕ P ϕ { δ x } , f ⟩ H L , ϕ = ⟨ a ϕ ( ⋅ , x ) , f ⟩ H L , ϕ , f(x)=\langle A_\phi P_\phi\{\delta_x\},f\rangle_{\mathcal{H}_{L,\phi}}=\langle a_\phi(\cdot,x),f\rangle_{\mathcal{H}_{L,\phi}}, f ( x ) = ⟨ A ϕ P ϕ { δ x } , f ⟩ H L , ϕ = ⟨ a ϕ ( ⋅ , x ) , f ⟩ H L , ϕ ,
which is the reproduction property (33), the membership a ϕ ( ⋅ , x ) ∈ H L , ϕ a_\phi(\cdot,x)\in\mathcal{H}_{L,\phi} a ϕ ( ⋅ , x ) ∈ H L , ϕ being granted by the range of A ϕ A_\phi A ϕ . This proves that (a) implies (c), while the converse implication is Definition 4.1. The formula for the reproducing kernel of H L \mathcal{H}_L H L is obtained in the same way from the Riesz map J H ′ = A ϕ + R p J_{\mathcal{H}'}=A_\phi+R_p J H ′ = A ϕ + R p of the full space (the augmented operator A + A_+ A + of Section 2.2), whose second block contributes R p { δ y } = ∑ n = 1 N 0 p n p n ( y ) R_p\{\delta_y\}=\sum_{n=1}^{N_0}p_n p_n(y) R p { δ y } = ∑ n = 1 N 0 p n p n ( y ) . Finally, the identity (38) is the pointwise transcription of the factorization (8) : the kernel of a composition is the contraction of the two kernels, that of L ϕ − 1 ∗ L_{\phi}^{-1*} L ϕ − 1 ∗ being the transpose of (34) , and the contraction converges precisely because g ϕ ( x , ⋅ ) ∈ L 2 ( R d ) g_{\phi}(x,\cdot)\in L_{2}(\mathbb{R}^{d}) g ϕ ( x , ⋅ ) ∈ L 2 ( R d ) .
Covariance function (Item 4): By Theorem 3.3 , the action of G G G extends to any test function in H L ′ \mathcal{H}'_{L} H L ′ , which under (a) includes the sampling functionals δ x \delta_x δ x ; the pointwise reading is therefore legitimate. Equation (20) with φ 1 = δ x \varphi_1=\delta_x φ 1 = δ x and φ 2 = δ y \varphi_2=\delta_y φ 2 = δ y then gives E { G ( x ) G ( y ) } = ⟨ A ϕ { δ x } , δ y ⟩ = a ϕ ( y , x ) \mathbb{E}\{G(x)G(y)\}=\langle A_{\phi}\{\delta_x\},\delta_y\rangle=a_{\phi}(y,x) E { G ( x ) G ( y )} = ⟨ A ϕ { δ x } , δ y ⟩ = a ϕ ( y , x ) , which is (39) by symmetry.
Three comments are in order.
First, Formula (35) is a double Taylor-type remainder: the raw Green’s function ρ B \rho_B ρ B is corrected in each variable by the finite-rank term that enforces the boundary conditions, and the double sum restores the symmetry. The characteristic property of a ϕ a_{\phi} a ϕ is that ϕ ( a ϕ ( ⋅ , y ) ) = 0 \phi(a_{\phi}(\cdot,y))=0 ϕ ( a ϕ ( ⋅ , y )) = 0 for every y ∈ R d y\in\mathbb{R}^{d} y ∈ R d , which is the pointwise transcription of the boundary conditions ϕ ( G ) = 0 \phi(G)=0 ϕ ( G ) = 0 of the stochastic differential equation (25) . For N p = { 0 } \mathcal{N}_{p}=\{0\} N p = { 0 } , all correction terms disappear and one recovers a ϕ = ρ B a_{\phi}=\rho_B a ϕ = ρ B , which is the identification of [60 Section 4.2] with ρ B = h \rho_B=h ρ B = h ; in the shift-invariant case, this is the kernel of [60 Eq. (56)] .
Second, the kernel g ϕ g_{\phi} g ϕ of (34) is the impulse response of the innovation model of § 3.3 : once G = L ϕ − 1 { W } G=L_{\phi}^{-1}\{W\} G = L ϕ − 1 { W } is available, (34) is the integral representation G ( x ) = ∫ R d g ϕ ( x , z ) W ( z ) d z G(x)=\int_{\mathbb{R}^{d}}g_{\phi}(x,z)W(z)\,\mathrm{d}z G ( x ) = ∫ R d g ϕ ( x , z ) W ( z ) d z of the process, and (38) is the classical formula that expresses a covariance as the autocorrelation of a shaping kernel. The worked example is fractional Brownian motion: g ϕ g_{\phi} g ϕ is the kernel (81) of Appendix § 7 , which reduces to the integral kernel of Mandelbrot’s representation (27) for N 0 = 1 N_{0}=1 N 0 = 1 , while a ϕ a_{\phi} a ϕ is the covariance function (89) , which reduces to (29) .
Third, the requirement δ x ∈ H L ′ \delta_x\in\mathcal{H}'_{L} δ x ∈ H L ′ is not a technicality but the exact frontier between the two readings of the theory. It is simultaneously the condition under which the native space is a space of functions rather than of distributions, and the condition under which the generalized process has samples G ( x ) G(x) G ( x ) rather than only smoothed observations G ( φ ) G(\varphi) G ( φ ) . Gaussian white noise, for which H L ′ = L 2 ( R d ) \mathcal{H}'_{L}=L_{2}(\mathbb{R}^{d}) H L ′ = L 2 ( R d ) and δ x ∉ L 2 ( R d ) \delta_x\notin L_{2}(\mathbb{R}^{d}) δ x ∈ / L 2 ( R d ) , is the canonical counterexample on both counts.
Conditional Positive Definiteness
The price to pay for a regularization operator L L L whose null space is nontrivial is that the kernel ρ B \rho_B ρ B of the conditional inverse A = ( L ∗ L ) − 1 A=(L^{*}L)^{-1} A = ( L ∗ L ) − 1 cannot be positive-definite. Instead, one has to settle for a weaker form of property that factors out the components that compromise uniqueness. The operator-level version of that notion is the p p p -conditional positive definiteness of § 2.2 . We now give its kernel-level counterpart, which is the form in which the property appears in the literature on radial basis functions and conditionally positive-definite kernels [8 , 20 , 36 , 66 ] .
Definition 4.3 (Conditionally positive-definite kernel). Let p = ( p 1 , … , p N 0 ) p=(p_1,\ldots,p_{N_0}) p = ( p 1 , … , p N 0 ) be a linearly independent family in C s g ( R d ) C_{\mathrm{sg}}(\mathbb{R}^d) C sg ( R d ) , with N p = span { p n } n = 1 N 0 N_p=\operatorname{span}\{p_n\}_{n=1}^{N_0} N p = span { p n } n = 1 N 0 the subspace that it spans. Then, a symmetric kernel h : R d × R d → R h:\mathbb{R}^d \times\mathbb{R}^d \to\mathbb{R} h : R d × R d → R is said to be p p p -conditionally positive-semi-definite if
∑ m = 1 N ∑ n = 1 N z m h ( x m , x n ) z n ≥ 0 (41) \sum_{m=1}^{N}\sum_{n=1}^{N} z_m h(x_m,x_n)z_n \ge0 \tag*{(41)} m = 1 ∑ N n = 1 ∑ N z m h ( x m , x n ) z n ≥ 0 (41) for any N ∈ N N\in\mathbb{N} N ∈ N , any x 1 , … , x N ∈ R d x_1,\ldots,x_N\in\mathbb{R}^d x 1 , … , x N ∈ R d and any z 1 , … , z N ∈ R z_1,\ldots,z_N\in\mathbb{R} z 1 , … , z N ∈ R that are subject to the moment conditions
∑ m = 1 N z m p n ( x m ) = 0 , n = 1 , … , N 0 . (42) \sum_{m=1}^{N} z_m p_n(x_m)=0,\qquad n=1,\ldots,N_0. \tag*{(42)} m = 1 ∑ N z m p n ( x m ) = 0 , n = 1 , … , N 0 . (42) It is said to be p p p -conditionally positive-definite if the inequality (41) is strict for all distinct x 1 , … , x N ∈ R d x_1,\ldots,x_N\in\mathbb{R}^d x 1 , … , x N ∈ R d and all ( z 1 , … , z N ) ∈ R N ∖ { 0 } (z_1,\ldots,z_N)\in\mathbb{R}^N\setminus\{0\} ( z 1 , … , z N ) ∈ R N ∖ { 0 } that satisfy (42) . The unconditional versions of the two properties are obtained for N 0 = 0 N_0=0 N 0 = 0 , in which case (42) is void.
The probability literature drops the qualifier “semi” in (41) . For a shift-invariant kernel h ( x , y ) = ρ ( x − y ) h(x,y)=\rho(x-y) h ( x , y ) = ρ ( x − y ) with N 0 = 0 N_0=0 N 0 = 0 , (41) is exactly the Bochner condition (11) of Theorem 2.3 , applied to ρ \rho ρ with R d \mathbb{R}^d R d in place of S S S ; this special case is the one that underlies the classical RKHS theory.
The two levels—operators and kernels—are equivalent as soon as the kernel is a genuine bivariate function of slow growth.
Theorem 4.4 (Kernel of a positive-semi-definite operator). Let A = T a A=T_a A = T a be a symmetric integral operator of the form (31) whose kernel a a a is of slow growth, and let p ⊂ C s g ( R d ) p\subset C_{\mathrm{sg}}(\mathbb{R}^d) p ⊂ C sg ( R d ) . Then, the ( p (p ( p -conditional) positive semi-definiteness of the operator A : S ( R d ) → S ′ ( R d ) A:S(\mathbb{R}^d)\to S'(\mathbb{R}^d) A : S ( R d ) → S ′ ( R d ) in the sense of § 2.2 is equivalent to the ( p (p ( p -conditional) positive semi-definiteness of its kernel a : R d × R d → R a:\mathbb{R}^d\times\mathbb{R}^d\to\mathbb{R} a : R d × R d → R in the sense of Definition 4.3 .
Proof. The two implications rest on two complementary discretizations: mollified Dirac combs for the direct part, and Riemann sums for the converse.
Mollified Dirac combs: For k ∈ N + k \in\mathbb{N}^{+} k ∈ N + , let u k ( x ) = ( k 2 π ) d e − 1 2 ∥ k x ∥ 2 u_k(\mathbf{x}) = \left(\frac{k}{\sqrt{2\pi}}\right)^{d}e^{-\frac{1}{2}\lVert k\mathbf{x}\rVert^{2}} u k ( x ) = ( 2 π k ) d e − 2 1 ∥ k x ∥ 2 be the rescaled and renormalized Gaussian pulse, which is in S ( R d ) \mathcal{S}(\mathbb{R}^{d}) S ( R d ) with ⟨ u k , 1 ⟩ = 1 \langle u_k,1\rangle=1 ⟨ u k , 1 ⟩ = 1 and which converges to the Dirac distribution in the weak-* topology of M ( R d ) \mathcal{M}(\mathbb{R}^{d}) M ( R d ) as k → ∞ k\to\infty k → ∞ . Given x 1 , … , x N ∈ R d \mathbf{x}_1,\ldots,\mathbf{x}_N\in\mathbb{R}^{d} x 1 , … , x N ∈ R d and z 1 , … , z N ∈ R z_1,\ldots,z_N\in\mathbb{R} z 1 , … , z N ∈ R , we set φ k = ∑ n = 1 N z n u k ( ⋅ − x n ) ∈ S ( R d ) \varphi_k=\sum_{n=1}^{N}z_nu_k(\cdot-\mathbf{x}_n)\in\mathcal{S}(\mathbb{R}^{d}) φ k = ∑ n = 1 N z n u k ( ⋅ − x n ) ∈ S ( R d ) . The family u k ( ⋅ − x m ) ⊗ u k ( ⋅ − x n ) u_k(\cdot-\mathbf{x}_m)\otimes u_k(\cdot-\mathbf{x}_n) u k ( ⋅ − x m ) ⊗ u k ( ⋅ − x n ) is an approximate identity at ( x m , x n ) (\mathbf{x}_m,\mathbf{x}_n) ( x m , x n ) , so that, for a given ϵ > 0 \epsilon>0 ϵ > 0 , we may split the corresponding integral of a a a into the contribution of a ball around ( x m , x n ) (\mathbf{x}_m,\mathbf{x}_n) ( x m , x n ) on which the continuity of a a a confines the integrand to a ( x m , x n ) ± ϵ a(\mathbf{x}_m,\mathbf{x}_n)\pm\epsilon a ( x m , x n ) ± ϵ , and the contribution of the complement, over which the Gaussian decay of u k u_k u k dominates the polynomial bound on a a a and which therefore vanishes as k → ∞ k\to\infty k → ∞ . This yields
lim k → ∞ ⟨ A { φ k } , φ k ⟩ = lim k → ∞ ∫ R d × R d a ( x , y ) ( φ k ⊗ φ k ) ( x , y ) d ( x , y ) = ∑ m = 1 N ∑ n = 1 N z m z n a ( x m , x n ) . (43) \begin{aligned}
\lim_{k\to\infty}\langle A\{\varphi_k\},\varphi_k\rangle
&=\lim_{k\to\infty}\int_{\mathbb{R}^{d}\times\mathbb{R}^{d}}a(\mathbf{x},\mathbf{y})(\varphi_k\otimes\varphi_k)(\mathbf{x},\mathbf{y})\,\mathrm{d}(\mathbf{x},\mathbf{y})\\
&=\sum_{m=1}^{N}\sum_{n=1}^{N}z_mz_na(\mathbf{x}_m,\mathbf{x}_n). \tag*{(43)}
\end{aligned} k → ∞ lim ⟨ A { φ k } , φ k ⟩ = k → ∞ lim ∫ R d × R d a ( x , y ) ( φ k ⊗ φ k ) ( x , y ) d ( x , y ) = m = 1 ∑ N n = 1 ∑ N z m z n a ( x m , x n ) . (43)
Operator ⇒ \Rightarrow ⇒ kernel: In the unconditional case, the left-hand side of (43) is non-negative for every k k k , so that the limit is non-negative as well, which is (41) . In the conditional case, we replace φ k \varphi_k φ k by P ϕ { φ k } = φ k − ∑ n = 1 N c n , k ϕ n ∈ S p ( R d ) P_{\phi}\{\varphi_k\}=\varphi_k-\sum_{n=1}^{N}c_{n,k}\phi_n\in\mathcal{S}_p(\mathbb{R}^{d}) P ϕ { φ k } = φ k − ∑ n = 1 N c n , k ϕ n ∈ S p ( R d ) with c n , k = ⟨ p n , φ k ⟩ c_{n,k}=\langle p_n,\varphi_k\rangle c n , k = ⟨ p n , φ k ⟩ , where ϕ ⊂ S ( R d ) \phi\subset\mathcal{S}(\mathbb{R}^{d}) ϕ ⊂ S ( R d ) is biorthogonal to p p p (Section 2.2). The positivity hypothesis now applies, since P ϕ { φ k } ∈ S p ( R d ) P_{\phi}\{\varphi_k\}\in\mathcal{S}_p(\mathbb{R}^{d}) P ϕ { φ k } ∈ S p ( R d ) by construction. The coefficients satisfy c n , k → ∑ m = 1 N z m p n ( x m ) = 0 c_{n,k}\to\sum_{m=1}^{N}z_mp_n(\mathbf{x}_m)=0 c n , k → ∑ m = 1 N z m p n ( x m ) = 0 , the moment conditions (42) being in force, so that the correction terms − 2 ∑ n c n , k ⟨ A ϕ n , φ k ⟩ + ∑ m ∑ n c m , k c n , k ⟨ A ϕ m , ϕ n ⟩ -2\sum_n c_{n,k}\langle A\phi_n,\varphi_k\rangle+\sum_m\sum_n c_{m,k}c_{n,k}\langle A\phi_m,\phi_n\rangle − 2 ∑ n c n , k ⟨ A ϕ n , φ k ⟩ + ∑ m ∑ n c m , k c n , k ⟨ A ϕ m , ϕ n ⟩ vanish as k → ∞ k\to\infty k → ∞ . The limit of ⟨ A P ϕ φ k , P ϕ φ k ⟩ ≥ 0 \langle AP_{\phi}\varphi_k,P_{\phi}\varphi_k\rangle\geq0 ⟨ A P ϕ φ k , P ϕ φ k ⟩ ≥ 0 is therefore again the right-hand side of (43) .
Kernel ⇒ \Rightarrow ⇒ operator: Let φ ∈ S ( R d ) \varphi\in\mathcal{S}(\mathbb{R}^{d}) φ ∈ S ( R d ) , R > 0 R>0 R > 0 and k ∈ N + k\in\mathbb{N}^{+} k ∈ N + , and let us introduce the Riemann sums
J R , k ( φ ) = ∑ m ∈ [ − k , k ] d ∑ n ∈ [ − k , k ] d z m a ( x m , x n ) z n , z m = ( R k ) d φ ( x m ) , (44) J_{R,k}(\varphi)=\sum_{\mathbf{m}\in[-k,k]^{d}}\sum_{\mathbf{n}\in[-k,k]^{d}}z_m a(\mathbf{x}_m,\mathbf{x}_n)z_n,\qquad z_m=\left(\frac{R}{k}\right)^{d}\varphi(\mathbf{x}_m), \tag*{(44)} J R , k ( φ ) = m ∈ [ − k , k ] d ∑ n ∈ [ − k , k ] d ∑ z m a ( x m , x n ) z n , z m = ( k R ) d φ ( x m ) , (44)
with x m = m R k ∈ R d \mathbf{x}_m=\frac{mR}{k}\in\mathbb{R}^{d} x m = k m R ∈ R d and m ∈ Z d m\in\mathbb{Z}^{d} m ∈ Z d . Let us set R = i R=i R = i and k = i 2 k=i^{2} k = i 2 , so that the mesh 1 i \frac{1}{i} i 1 vanishes while the domain fills R d \mathbb{R}^{d} R d . The integrand F ( x , y ) = φ ( x ) a ( x , y ) φ ( y ) F(\mathbf{x},\mathbf{y})=\varphi(\mathbf{x})a(\mathbf{x},\mathbf{y})\varphi(\mathbf{y}) F ( x , y ) = φ ( x ) a ( x , y ) φ ( y ) is continuous, and the rapid decay of φ \varphi φ against the polynomial bound on a a a gives ∣ F ( x , y ) ∣ ≤ C N ( 1 + ∥ x ∥ ) − N ( 1 + ∥ y ∥ ) − N |F(\mathbf{x},\mathbf{y})|\leq C_N(1+\lVert\mathbf{x}\rVert)^{-N}(1+\lVert\mathbf{y}\rVert)^{-N} ∣ F ( x , y ) ∣ ≤ C N ( 1 + ∥ x ∥ ) − N ( 1 + ∥ y ∥ ) − N for every N N N . Given ϵ > 0 \epsilon>0 ϵ > 0 , we may therefore pick T > 0 T>0 T > 0 such that both the integral of ∣ F ∣ |F| ∣ F ∣ and the Riemann sums of ∣ F ∣ |F| ∣ F ∣ over the complement of [ − T , T ] 2 d [-T,T]^{2d} [ − T , T ] 2 d are below ε \varepsilon ε , uniformly in i i i (the mesh being at most 1); on the compact part, the uniform continuity of F F F makes the sums converge to the integral. Hence lim i → ∞ J i , i 2 ( φ ) = ⟨ A φ , φ ⟩ \lim_{i\to\infty} J_{i,i^2}(\varphi)=\langle A\varphi,\varphi\rangle lim i → ∞ J i , i 2 ( φ ) = ⟨ A φ , φ ⟩ . In the unconditional case, J i , i 2 ( φ ) ≥ 0 J_{i,i^2}(\varphi)\ge0 J i , i 2 ( φ ) ≥ 0 by (41) , and the conclusion follows.
In the conditional case, the weights z m z_m z m of (43) do not fulfil the moment conditions (41) exactly, but only asymptotically, so that a correction is needed. Let us fix N 0 N_0 N 0 points y 1 , … , y N 0 ∈ R d \mathbf{y}_1,\ldots,\mathbf{y}_{N_0}\in\mathbb{R}^d y 1 , … , y N 0 ∈ R d such that the matrix
P = [ p n ( y j ) ] ∈ R N 0 × N 0 \mathbf{P}=[p_n(\mathbf{y}_j)]\in\mathbb{R}^{N_0\times N_0} P = [ p n ( y j )] ∈ R N 0 × N 0
is invertible, which is possible because the p n p_n p n are linearly independent continuous functions. For φ ∈ S p ( R d ) \varphi\in\mathcal{S}_p(\mathbb{R}^d) φ ∈ S p ( R d ) , we then form the residuals ϵ i , n = ∑ m z m p n ( x m ) \epsilon_{i,n}=\sum_m z_m p_n(\mathbf{x}_m) ϵ i , n = ∑ m z m p n ( x m ) and the corrective weights c i = − P − 1 ϵ i \mathbf{c}_i=-\mathbf{P}^{-1}\boldsymbol{\epsilon}_i c i = − P − 1 ϵ i , and we append the atoms y j \mathbf{y}_j y j with weights c i , j c_{i,j} c i , j to the grid. The augmented family fulfils (41) exactly, so that Definition 4.3 applies and yields
J i , i 2 ( φ ) + 2 ∑ j = 1 N 0 c i , j ∑ m z m a ( y j , x m ) + ∑ j = 1 N 0 ∑ j ′ = 1 N 0 c i , j c i , j ′ a ( y j , y j ′ ) ≥ 0. J_{i,i^2}(\varphi)+2\sum_{j=1}^{N_0}c_{i,j}\sum_m z_m a(\mathbf{y}_j,\mathbf{x}_m)+\sum_{j=1}^{N_0}\sum_{j'=1}^{N_0}c_{i,j}c_{i,j'}a(\mathbf{y}_j,\mathbf{y}_{j'})\ge0. J i , i 2 ( φ ) + 2 j = 1 ∑ N 0 c i , j m ∑ z m a ( y j , x m ) + j = 1 ∑ N 0 j ′ = 1 ∑ N 0 c i , j c i , j ′ a ( y j , y j ′ ) ≥ 0.
Now, ϵ i , n → ⟨ p n , φ ⟩ = 0 \epsilon_{i,n}\to\langle p_n,\varphi\rangle=0 ϵ i , n → ⟨ p n , φ ⟩ = 0 as i → ∞ i\to\infty i → ∞ , which gives c i → 0 \mathbf{c}_i\to0 c i → 0 ; moreover, the sums ∑ m ∣ z m ∣ ∣ a ( y j , x m ) ∣ \sum_m|z_m||a(\mathbf{y}_j,\mathbf{x}_m)| ∑ m ∣ z m ∣∣ a ( y j , x m ) ∣ are bounded uniformly in i i i , because a a a is of slow growth and φ \varphi φ is rapidly decaying. The two corrective terms therefore vanish in the limit, which leaves ⟨ A φ , φ ⟩ ≥ 0 \langle A\varphi,\varphi\rangle\ge0 ⟨ A φ , φ ⟩ ≥ 0 for all φ ∈ S p ( R d ) \varphi\in\mathcal{S}_p(\mathbb{R}^d) φ ∈ S p ( R d ) .
Neither of the two implications survives the passage to the strict versions of the properties, and it is worth saying why, since the strict form is the one that governs the invertibility of the interpolation matrices.
In the direction “operator ⇒ \Rightarrow ⇒ kernel”, the obstruction is that a strict inequality is not preserved by the limit (43) . A counterexample on R \mathbb{R} R is the kernel a ( x , y ) = x y e − ( x − y ) 2 a(x,y)=xy\mathrm{e}^{-(x-y)^2} a ( x , y ) = x y e − ( x − y ) 2 , which is of slow growth and which satisfies ⟨ A φ , φ ⟩ = ⟨ A g ψ , ψ ⟩ > 0 \langle A\varphi,\varphi\rangle=\langle A_g\psi,\psi\rangle>0 ⟨ A φ , φ ⟩ = ⟨ A g ψ , ψ ⟩ > 0 for every φ ∈ S ( R ) ∖ { 0 } \varphi\in\mathcal{S}(\mathbb{R})\setminus\{0\} φ ∈ S ( R ) ∖ { 0 } , where ψ ( x ) = x φ ( x ) ∈ S ( R ) \psi(x)=x\varphi(x)\in\mathcal{S}(\mathbb{R}) ψ ( x ) = x φ ( x ) ∈ S ( R ) and where A g A_g A g is the (strictly positive-definite) Gaussian operator with kernel e − ( x − y ) 2 \mathrm{e}^{-(x-y)^2} e − ( x − y ) 2 . The kernel of A A A , by contrast, degenerates at the origin, since a ( 0 , 0 ) = 0 a(0,0)=0 a ( 0 , 0 ) = 0 .
In the direction “kernel ⇒ \Rightarrow ⇒ operator”, the obstruction is that the strict positivity of a kernel only constrains the finite Gram matrices. The illustration is the one-dimensional shift-invariant kernel h sinc ( x , y ) = sinc ( x − y ) h_{\operatorname{sinc}}(x,y)=\operatorname{sinc}(x-y) h sinc ( x , y ) = sinc ( x − y ) , which is positive-definite in the sense of Definition 4.3 because the functions ω ↦ e i ω x m \omega\mapsto\mathrm{e}^{i\omega x_m} ω ↦ e iω x m are linearly independent in L 2 ( [ − π , π ] ) L_2([-\pi,\pi]) L 2 ([ − π , π ]) for distinct x m x_m x m . Its operator, by contrast, is only positive-semi-definite. Indeed, we have that F { sinc } ( ω ) = rect ( ω / ( 2 π ) ) \mathcal{F}\{\operatorname{sinc}\}(\omega)=\operatorname{rect}(\omega/(2\pi)) F { sinc } ( ω ) = rect ( ω / ( 2 π )) , so that
⟨ A sinc φ , φ ⟩ = 1 2 π ∫ R rect ( ω 2 π ) ∣ φ ^ ( ω ) ∣ 2 d ω . (45) \langle A_{\operatorname{sinc}}\varphi,\varphi\rangle=\frac{1}{2\pi}\int_{\mathbb{R}}\operatorname{rect}\left(\frac{\omega}{2\pi}\right)|\widehat{\varphi}(\omega)|^2\,\mathrm{d}\omega. \tag*{(45)} ⟨ A sinc φ , φ ⟩ = 2 π 1 ∫ R rect ( 2 π ω ) ∣ φ ( ω ) ∣ 2 d ω . (45)
which vanishes for any φ ∈ S ( R ) ∖ { 0 } \varphi\in\mathcal{S}(\mathbb{R})\setminus\{0\} φ ∈ S ( R ) ∖ { 0 } whose spectrum is supported outside the band [ − π , π ] [-\pi,\pi] [ − π , π ] .
What does hold, and what the theory of Section 4 actually needs, is an exact characterization of the strict property for the kernel a φ a_\varphi a φ of Theorem 4.2 in terms of the sampling functionals. Let us assume that the conditions of Item 3 are met and let x 1 , … , x M ∈ R d x_1,\ldots,x_M \in\mathbb{R}^d x 1 , … , x M ∈ R d be distinct. For any z ∈ R M z \in\mathbb{R}^M z ∈ R M that fulfils the moment conditions (41), we have that P φ { ∑ m z m δ x m } = ∑ m z m δ x m P_\varphi\left\{\sum_m z_m\delta_{x_m}\right\}=\sum_m z_m\delta_{x_m} P φ { ∑ m z m δ x m } = ∑ m z m δ x m , so that
∑ m = 1 M ∑ n = 1 M z m a φ ( x m , x n ) z n = ∥ ∑ m = 1 M z m δ x m ∥ H L ′ 2 (46) \sum_{m=1}^{M}\sum_{n=1}^{M}z_m a_\varphi(x_m,x_n)z_n=\left\|\sum_{m=1}^{M}z_m\delta_{x_m}\right\|_{\mathcal{H}'_L}^{2} \tag*{(46)} m = 1 ∑ M n = 1 ∑ M z m a φ ( x m , x n ) z n = m = 1 ∑ M z m δ x m H L ′ 2 (46)
by (10). The right-hand side vanishes for some z ≠ 0 z \ne0 z = 0 if and only if the δ x m \delta_{x_m} δ x m are linearly dependent in H L ′ \mathcal{H}'_L H L ′ —the moment conditions being automatic for such a z z z , as one sees by testing against p ⊂ H L p \subset\mathcal{H}_L p ⊂ H L . We have therefore established the following.
Corollary 4.5 (Strict positive definiteness at a set of nodes). Under the conditions of Item 3 of Theorem 4.2, the kernel a φ a_\varphi a φ is strictly p p p -conditionally positive-definite in the sense of Definition 4.3 if and only if the sampling functionals δ x 1 , … , δ x M \delta_{x_1},\ldots,\delta_{x_M} δ x 1 , … , δ x M are linearly independent in H L ′ \mathcal{H}'_L H L ′ , for every finite set of distinct points x 1 , … , x M ∈ R d x_1,\ldots,x_M \in\mathbb{R}^d x 1 , … , x M ∈ R d .
This is the linear independence of the sampling functionals in H L ′ \mathcal{H}'_L H L ′ , which is Condition 1 of the L L L -compatibility used in Section 5 (read for ν m = δ x m \nu_m=\delta_{x_m} ν m = δ x m ). It is also the pointwise instance of the criterion of Theorem 3.3, which states that the covariance matrix of ( G ( φ 1 ) , … , G ( φ N ) ) (G(\varphi_1),\ldots,G(\varphi_N)) ( G ( φ 1 ) , … , G ( φ N )) is positive-definite exactly when the P φ { φ n } P_\varphi\{\varphi_n\} P φ { φ n } are linearly independent in H A \mathcal{H}_A H A . The shift-invariant case is the reassuring one: if A ^ ∈ L 1 ( R d ) \widehat{A}\in L_1(\mathbb{R}^d) A ∈ L 1 ( R d ) with A ^ > 0 \widehat{A}>0 A > 0 almost everywhere, then
∑ m ∑ n z m z n ρ ( x m − x n ) = 1 ( 2 π ) d ∫ R d A ^ ( ω ) ∣ ∑ m z m e − i ⟨ ω , x m ⟩ ∣ 2 d ω \sum_m\sum_n z_m z_n \rho(x_m-x_n)=\frac{1}{(2\pi)^d}\int_{\mathbb{R}^d}\widehat{A}(\omega)\left|\sum_m z_m e^{-i\langle\omega,x_m\rangle}\right|^2\,d\omega m ∑ n ∑ z m z n ρ ( x m − x n ) = ( 2 π ) d 1 ∫ R d A ( ω ) m ∑ z m e − i ⟨ ω , x m ⟩ 2 d ω
is strictly positive, because the exponential sum is real-analytic and therefore vanishes only on a set of measure zero.
Extended Fractional Brownian Fields on R d \mathbb{R}^d R d
We now illustrate the framework with the fractional Laplacian [10 , 28 ] , which is the multidimensional counterpart of the fractional derivative of Section 3.3. The resulting processes are the isotropic fractional Brownian fields : Lévy’s multiparameter Brownian motion [29 ] and its fractional extensions [54 ] . Since all the tools are already in place, every formula below is an instance of Theorem 4.2, whose hypothesis ϕ ⊂ H L ′ \phi\subset\mathcal{H}'_{\mathrm{L}} ϕ ⊂ H L ′ covers the Dirac-type boundary functionals.
The fractional Laplacian L = ( − Δ ) γ / 2 : S ( R d ) → S ′ ( R d ) \mathrm{L}=(-\Delta)^{\gamma/2}: \mathcal{S}(\mathbb{R}^{d}) \to\mathcal{S}'(\mathbb{R}^{d}) L = ( − Δ ) γ /2 : S ( R d ) → S ′ ( R d ) is the shift-invariant operator whose Fourier symbol is L ^ ( ω ) = ∥ ω ∥ γ \widehat{\mathrm{L}}(\omega)=\lVert\omega\rVert^{\gamma} L ( ω ) = ∥ ω ∥ γ , over the range
γ > d 2 , H = γ − d 2 ∉ N , k γ = ⌊ H ⌋ , (47) \gamma> \frac{d}{2}, \qquad H=\gamma-\frac{d}{2}\notin\mathbb{N}, \qquad k_{\gamma}=\lfloor H\rfloor, \tag*{(47)} γ > 2 d , H = γ − 2 d ∈ / N , k γ = ⌊ H ⌋ , (47)
where H H H is the Hurst-type exponent of the resulting field and where the excluded integer values of H H H are the logarithmic (thin-plate) cases discussed at the end of this section. The operator is self-adjoint, with normal operator B = ( − Δ ) γ \mathrm{B}=(-\Delta)^{\gamma} B = ( − Δ ) γ of symbol ∥ ω ∥ 2 γ \lVert\omega\rVert^{2\gamma} ∥ ω ∥ 2 γ . The fractional Laplacian meets the conditions of the multiplier criterion [59 Proposition 5.4] , so that L \mathrm{L} L is p p p -admissible in the sense of Definition 2.2, with attainable null space N B = Π k γ \mathcal{N}_{\mathrm{B}}=\Pi_{k_{\gamma}} N B = Π k γ , the space of polynomials of degree at most k γ k_{\gamma} k γ , whose dimension is N 0 = ( k γ + d d ) N_{0}=\binom{k_{\gamma}+d}{d} N 0 = ( d k γ + d ) . With the usual multi-index notation (∣ k ∣ = k 1 + ⋯ + k d |\boldsymbol{k}|=k_{1}+\cdots+k_{d} ∣ k ∣ = k 1 + ⋯ + k d , k ! = k 1 ! ⋯ k d ! \boldsymbol{k}!=k_{1}!\cdots k_{d}! k ! = k 1 ! ⋯ k d ! , and x k = x 1 k 1 ⋯ x d k d \boldsymbol{x}^{\boldsymbol{k}}=x_{1}^{k_{1}}\cdots x_{d}^{k_{d}} x k = x 1 k 1 ⋯ x d k d ), the biorthogonal system is
p k ( x ) = x k k ! , ϕ k = ∂ k δ , ∣ k ∣ ≤ k γ , (48) p_{\boldsymbol{k}}(x)=\frac{x^{\boldsymbol{k}}}{\boldsymbol{k}!}, \qquad\phi_{\boldsymbol{k}}=\partial^{\boldsymbol{k}}\delta, \qquad|\boldsymbol{k}|\leq k_{\gamma}, \tag*{(48)} p k ( x ) = k ! x k , ϕ k = ∂ k δ , ∣ k ∣ ≤ k γ , (48)
with the convention ⟨ ∂ k δ , f ⟩ = ∂ k f ( 0 ) \langle\partial^{\boldsymbol{k}}\delta,f\rangle=\partial^{\boldsymbol{k}}f(0) ⟨ ∂ k δ , f ⟩ = ∂ k f ( 0 ) , so that the boundary conditions read ∂ k g ( 0 ) = 0 \partial^{\boldsymbol{k}}g(0)=0 ∂ k g ( 0 ) = 0 for ∣ k ∣ ≤ k γ |\boldsymbol{k}|\leq k_{\gamma} ∣ k ∣ ≤ k γ . The primary example is the range k γ = 0 k_{\gamma}=0 k γ = 0 , that is, H ∈ ( 0 , 1 ) H\in(0,1) H ∈ ( 0 , 1 ) : the null space then reduces to the constants, with ( p 1 , ϕ 1 ) = ( 1 , δ ) (p_{1},\phi_{1})=(1,\delta) ( p 1 , ϕ 1 ) = ( 1 , δ ) and the single boundary condition g ( 0 ) = 0 g(0)=0 g ( 0 ) = 0 —Lévy’s field, and the multidimensional analogue of Mandelbrot’s normalization.
Both Green’s functions are Riesz kernels. With the Fourier convention ϕ ^ ( ω ) = ∫ R d ϕ ( x ) e − i ⟨ ω , x ⟩ d x \widehat{\phi}(\omega)=\int_{\mathbb{R}^{d}}\phi(x)e^{-i\langle\omega,x\rangle}\,dx ϕ ( ω ) = ∫ R d ϕ ( x ) e − i ⟨ ω , x ⟩ d x , the classical transform pair of homogeneous distributions reads
F { ∥ ⋅ ∥ λ } ( ω ) = 2 λ + d π d / 2 Γ ( λ + d 2 ) Γ ( − λ 2 ) ∥ ω ∥ − λ − d , (49) \mathcal{F}\{\lVert\cdot\rVert^{\lambda}\}(\omega)=\frac{2^{\lambda+d}\pi^{d/2}\Gamma\left(\frac{\lambda+d}{2}\right)}{\Gamma\left(-\frac{\lambda}{2}\right)}\lVert\omega\rVert^{-\lambda-d}, \tag*{(49)} F {∥ ⋅ ∥ λ } ( ω ) = Γ ( − 2 λ ) 2 λ + d π d /2 Γ ( 2 λ + d ) ∥ ω ∥ − λ − d , (49)
which holds by analytic continuation for λ ∉ { 0 , 2 , 4 , … } ∪ { − d , − d − 2 , … } \lambda\notin\{0,2,4,\ldots\}\cup\{-d,-d-2,\ldots\} λ ∈ / { 0 , 2 , 4 , … } ∪ { − d , − d − 2 , … } , where the excluded values are those for which the pair degenerates and logarithmic factors appear (see [14, Chapter II] and [16, Section 2.4.3]). The choices λ = γ − d \lambda=\gamma-d λ = γ − d and λ = 2 γ − d = 2 H \lambda=2\gamma-d=2H λ = 2 γ − d = 2 H yield
ρ L ( x ) = C γ , d ∥ x ∥ γ − d with C γ , d = Γ ( d − γ 2 ) 2 γ π d / 2 Γ ( γ 2 ) (50) \rho_{\mathrm{L}}(x)=C_{\gamma,d}\lVert x\rVert^{\gamma-d}
\quad\text{with}\quad
C_{\gamma,d}=\frac{\Gamma\left(\frac{d-\gamma}{2}\right)}{2^{\gamma}\pi^{d/2}\Gamma\left(\frac{\gamma}{2}\right)} \tag*{(50)} ρ L ( x ) = C γ , d ∥ x ∥ γ − d with C γ , d = 2 γ π d /2 Γ ( 2 γ ) Γ ( 2 d − γ ) (50)
ρ B ( x ) = c γ , d ∥ x ∥ 2 H with c γ , d = Γ ( − H ) 4 γ π d / 2 Γ ( γ ) , (51) \rho_{\mathrm{B}}(x)=c_{\gamma,d}\lVert x\rVert^{2H}
\quad\text{with}\quad
c_{\gamma,d}=\frac{\Gamma(-H)}{4^{\gamma}\pi^{d/2}\Gamma(\gamma)}, \tag*{(51)} ρ B ( x ) = c γ , d ∥ x ∥ 2 H with c γ , d = 4 γ π d /2 Γ ( γ ) Γ ( − H ) , (51)
where ρ L = F − 1 { ∥ ⋅ ∥ − γ } \rho_{\mathrm{L}}=\mathcal{F}^{-1}\{\lVert\cdot\rVert^{-\gamma}\} ρ L = F − 1 {∥ ⋅ ∥ − γ } and ρ B = F − 1 { ∥ ⋅ ∥ − 2 γ } \rho_{\mathrm{B}}=\mathcal{F}^{-1}\{\lVert\cdot\rVert^{-2\gamma}\} ρ B = F − 1 {∥ ⋅ ∥ − 2 γ } are the canonical (isotropic) Green’s functions of L L L and B B B . The formula for ρ B \rho_{\mathrm{B}} ρ B is valid throughout (47) , since 2 H 2H 2 H avoids the even integers exactly when H ∉ N H\notin\mathbb{N} H ∈ / N . The formula for ρ L \rho_{\mathrm{L}} ρ L additionally requires γ − d ∉ { 0 , 2 , 4 , … } \gamma-d\notin\{0,2,4,\ldots\} γ − d ∈ / { 0 , 2 , 4 , … } ; at those values, ρ L \rho_{\mathrm{L}} ρ L acquires a logarithmic factor, which affects the innovation kernel below but neither ρ B \rho_{\mathrm{B}} ρ B nor the covariance.
Both Green’s functions are powers of the Euclidean norm, and the boundary corrections that follow involve their Taylor polynomials at the origin. We therefore record the closed form of the directional derivatives of f λ = ∥ ⋅ ∥ λ f_\lambda=\lVert\cdot\rVert^\lambda f λ = ∥ ⋅ ∥ λ ,
( h ⋅ ∇ ) m f λ ( u ) = m ! ∑ l = 0 ⌊ m / 2 ⌋ ( λ / 2 m − l ) ( m − l l ) ( 2 ⟨ u , h ⟩ ) m − 2 l ∥ h ∥ 2 l ∥ u ∥ λ − 2 ( m − l ) , (52) (\boldsymbol{h}\cdot\nabla)^m f_\lambda(\boldsymbol{u})=m!\sum_{l=0}^{\lfloor m/2\rfloor}\binom{\lambda/2}{m-l}\binom{m-l}{l}(2\langle\boldsymbol{u},\boldsymbol{h}\rangle)^{m-2l}\lVert\boldsymbol{h}\rVert^{2l}\lVert\boldsymbol{u}\rVert^{\lambda-2(m-l)}, \tag*{(52)} ( h ⋅ ∇ ) m f λ ( u ) = m ! l = 0 ∑ ⌊ m /2 ⌋ ( m − l λ /2 ) ( l m − l ) ( 2 ⟨ u , h ⟩ ) m − 2 l ∥ h ∥ 2 l ∥ u ∥ λ − 2 ( m − l ) , (52)
where ( λ / 2 j ) \binom{\lambda/2}{j} ( j λ /2 ) is the generalized binomial coefficient. Indeed, ∥ u + t h ∥ 2 = ∥ u ∥ 2 + 2 t ⟨ u , h ⟩ + t 2 ∥ h ∥ 2 \lVert\boldsymbol{u}+t\boldsymbol{h}\rVert^2=\lVert\boldsymbol{u}\rVert^2+2t\langle\boldsymbol{u},\boldsymbol{h}\rangle+t^2\lVert\boldsymbol{h}\rVert^2 ∥ u + t h ∥ 2 = ∥ u ∥ 2 + 2 t ⟨ u , h ⟩ + t 2 ∥ h ∥ 2 , so that t ↦ f λ ( u + t h ) t\mapsto f_\lambda(\boldsymbol{u}+t\boldsymbol{h}) t ↦ f λ ( u + t h ) is a power of a quadratic polynomial in t t t ; the binomial series ( 1 + v ) λ / 2 = ∑ j ≥ 0 ( λ / 2 j ) v j (1+v)^{\lambda/2}=\sum_{j\geq0}\binom{\lambda/2}{j}v^j ( 1 + v ) λ /2 = ∑ j ≥ 0 ( j λ /2 ) v j , applied to v = ( 2 t ⟨ u , h ⟩ + t 2 ∥ h ∥ 2 ) / ∥ u ∥ 2 v=(2t\langle\boldsymbol{u},\boldsymbol{h}\rangle+t^2\lVert\boldsymbol{h}\rVert^2)/\lVert\boldsymbol{u}\rVert^2 v = ( 2 t ⟨ u , h ⟩ + t 2 ∥ h ∥ 2 ) / ∥ u ∥ 2 and followed by the expansion of ( 2 ⟨ u , h ⟩ + t ∥ h ∥ 2 ) j (2\langle\boldsymbol{u},\boldsymbol{h}\rangle+t\lVert\boldsymbol{h}\rVert^2)^j ( 2 ⟨ u , h ⟩ + t ∥ h ∥ 2 ) j , gives (51) with l = m − j l=m-j l = m − j counting the powers of t t t that come from the quadratic term. An equivalent form is
( h ⋅ ∇ ) m f λ ( u ) = ( − 1 ) m m ! ∥ h ∥ m ∥ u ∥ λ − m C m ( − λ / 2 ) ( cos θ ) , (\boldsymbol{h}\cdot\nabla)^m f_\lambda(\boldsymbol{u})=(-1)^m m!\lVert\boldsymbol{h}\rVert^m\lVert\boldsymbol{u}\rVert^{\lambda-m}C_m^{(-\lambda/2)}(\cos\theta), ( h ⋅ ∇ ) m f λ ( u ) = ( − 1 ) m m ! ∥ h ∥ m ∥ u ∥ λ − m C m ( − λ /2 ) ( cos θ ) ,
where cos θ = ⟨ u , h ⟩ / ( ∥ u ∥ ∥ h ∥ ) \cos\theta=\langle\boldsymbol{u},\boldsymbol{h}\rangle/(\lVert\boldsymbol{u}\rVert\lVert\boldsymbol{h}\rVert) cos θ = ⟨ u , h ⟩ / (∥ u ∥ ∥ h ∥) and C m ( α ) C_m^{(\alpha)} C m ( α ) is the Gegenbauer polynomial of degree m m m .
The three objects of Theorem 4.2 retain a compact form at every order. Since the Riesz kernels are radial, we have that q k ( y ) = ⟨ φ k , ρ L ( ⋅ − y ) ⟩ = ( ∂ k ρ L ) ( − y ) q_{\boldsymbol{k}}(\boldsymbol{y})=\langle\varphi_{\boldsymbol{k}},\rho_{\mathrm{L}}(\cdot-\boldsymbol{y})\rangle=(\partial^{\boldsymbol{k}}\rho_{\mathrm{L}})(-\boldsymbol{y}) q k ( y ) = ⟨ φ k , ρ L ( ⋅ − y )⟩ = ( ∂ k ρ L ) ( − y ) , so that Item 1 identifies the impulse response of the innovation model as the Taylor remainder
g φ ( x , y ) = ρ L ( x − y ) − ∑ ∣ k ∣ ≤ k γ x k k ! ( ∂ k ρ L ) ( − y ) , (53) g_{\varphi}(\boldsymbol{x},\boldsymbol{y})=\rho_{\mathrm{L}}(\boldsymbol{x}-\boldsymbol{y})-\sum_{\lvert\boldsymbol{k}\rvert\leq k_\gamma}\frac{\boldsymbol{x}^{\boldsymbol{k}}}{\boldsymbol{k}!}(\partial^{\boldsymbol{k}}\rho_{\mathrm{L}})(-\boldsymbol{y}), \tag*{(53)} g φ ( x , y ) = ρ L ( x − y ) − ∣ k ∣ ≤ k γ ∑ k ! x k ( ∂ k ρ L ) ( − y ) , (53)
in which the subtraction of the Taylor polynomial of x ↦ ρ L ( x − y ) \boldsymbol{x}\mapsto\rho_{\mathrm{L}}(\boldsymbol{x}-\boldsymbol{y}) x ↦ ρ L ( x − y ) at the origin is the action of the projector P φ ∗ P_\varphi^* P φ ∗ , exactly as in the one-dimensional case of Appendix A. The multinomial theorem ∑ ∣ k ∣ = m x k k ! ∂ k = 1 m ! ( x ⋅ ∇ ) m \sum_{\lvert\boldsymbol{k}\rvert=m}\frac{\boldsymbol{x}^{\boldsymbol{k}}}{\boldsymbol{k}!}\partial^{\boldsymbol{k}}=\frac{1}{m!}(\boldsymbol{x}\cdot\nabla)^m ∑ ∣ k ∣ = m k ! x k ∂ k = m ! 1 ( x ⋅ ∇ ) m , together with (51) for λ = γ − d \lambda= \gamma- d λ = γ − d , turns (52) into the closed form
g ϕ ( x , y ) = C γ , d ( ∥ x − y ∥ γ − d − ∑ m = 0 k γ ∑ l = 0 ⌊ m / 2 ⌋ ( γ − d 2 m − l ) ( m − l l ) ( − 2 ⟨ x , y ⟩ ) m − 2 l × ∥ x ∥ 2 l ∥ y ∥ γ − d − 2 ( m − l ) ) (54) g_{\phi}(x,y) = C_{\gamma,d}\left(\lVert x-y\rVert^{\gamma-d} - \sum_{m=0}^{k_{\gamma}}\sum_{l=0}^{\lfloor m/2\rfloor} \binom{\frac{\gamma-d}{2}}{m-l}\binom{m-l}{l}\left(-2\langle x,y\rangle\right)^{m-2l} \times\lVert x\rVert^{2l}\lVert y\rVert^{\gamma-d-2(m-l)}\right) \tag*{(54)} g ϕ ( x , y ) = C γ , d ∥ x − y ∥ γ − d − m = 0 ∑ k γ l = 0 ∑ ⌊ m /2 ⌋ ( m − l 2 γ − d ) ( l m − l ) ( − 2 ⟨ x , y ⟩ ) m − 2 l × ∥ x ∥ 2 l ∥ y ∥ γ − d − 2 ( m − l ) (54)
in which the factorials of (51) cancel those of the Taylor expansion. The fractional Brownian field of order γ \gamma γ is the corresponding process B γ = L ϕ − 1 { W } B_{\gamma} = L_{\phi}^{-1}\{W\} B γ = L ϕ − 1 { W } ; that is, the solution of the stochastic partial differential equation
( − Δ ) γ / 2 { B γ } = W subject to ∂ k B γ ( 0 ) = 0 , ∣ k ∣ ≤ k γ . (55) (-\Delta)^{\gamma/2}\{B_{\gamma}\} = W \qquad\text{subject to} \qquad\partial^{k}B_{\gamma}(0) = 0,\quad\lvert k\rvert\le k_{\gamma}. \tag*{(55)} ( − Δ ) γ /2 { B γ } = W subject to ∂ k B γ ( 0 ) = 0 , ∣ k ∣ ≤ k γ . (55)
Equivalently, it is the stochastic integral
B γ ( x ) = ∫ R d g ϕ ( x , z ) W ( z ) d z (56) B_{\gamma}(x) = \int_{\mathbb{R}^{d}} g_{\phi}(x,z) W(z)\,\mathrm{d}z \tag*{(56)} B γ ( x ) = ∫ R d g ϕ ( x , z ) W ( z ) d z (56)
with the kernel (54) , which is the multidimensional counterpart, at arbitrary order, of Mandelbrot’s representation (27) . For k γ = 0 k_{\gamma} = 0 k γ = 0 , the correction reduces to the single term ∥ z ∥ γ − d \lVert z\rVert^{\gamma-d} ∥ z ∥ γ − d , so that B γ ( x ) = C γ , d ∫ R d ( ∥ x − z ∥ γ − d − ∥ z ∥ γ − d ) W ( z ) d z B_{\gamma}(x) = C_{\gamma,d}\int_{\mathbb{R}^{d}}\left(\lVert x-z\rVert^{\gamma-d}-\lVert z\rVert^{\gamma-d}\right)W(z)\,\mathrm{d}z B γ ( x ) = C γ , d ∫ R d ( ∥ x − z ∥ γ − d − ∥ z ∥ γ − d ) W ( z ) d z , which is Lévy’s field. Let us note that the Riesz kernel is symmetric, so that the case d = 1 d = 1 d = 1 returns the well-balanced version of fBm rather than the causal one of (27) ; the two differ by a normalization factor only, since fBm is determined by H H H up to a multiplicative constant. Likewise, v k = A { ϕ k } = ( − 1 ) ∣ k ∣ ∂ k ρ B v_k = A\{\phi_k\} = (-1)^{\lvert k\rvert}\partial^k\rho_B v k = A { ϕ k } = ( − 1 ) ∣ k ∣ ∂ k ρ B and r k , l = ± ( ∂ k + l ρ B ) ( 0 ) = 0 r_{k,l} = \pm(\partial^{k+l}\rho_B)(0) = 0 r k , l = ± ( ∂ k + l ρ B ) ( 0 ) = 0 , because ∂ k + l ρ B \partial^{k+l}\rho_B ∂ k + l ρ B is homogeneous of degree 2 H − ∣ k ∣ − ∣ l ∣ ≥ 2 H − 2 k γ > 0 2H-\lvert k\rvert-\lvert l\rvert\ge2H-2k_{\gamma} > 0 2 H − ∣ k ∣ − ∣ l ∣ ≥ 2 H − 2 k γ > 0 and therefore vanishes at the origin.
The conditioning has a transparent Fourier signature, and this is where the multi-index sums telescope. In the Fourier domain, the projector P ϕ P_{\phi} P ϕ subtracts from ϕ ^ \hat{\phi} ϕ ^ its Taylor polynomial of degree k γ k_{\gamma} k γ at ω = 0 \omega= 0 ω = 0 . For the sampling functional δ x \delta_x δ x , whose Fourier transform is e − i ⟨ ω , x ⟩ \mathrm{e}^{-\mathrm{i}\langle\omega,x\rangle} e − i ⟨ ω , x ⟩ , the multinomial theorem collapses that polynomial—a priori, a sum over the N 0 N_0 N 0 multi-indices—into a function of the single scalar ⟨ ω , x ⟩ \langle\omega,x\rangle ⟨ ω , x ⟩ , so that F { P ϕ δ x } ( ω ) = E k γ ( − i ⟨ ω , x ⟩ ) \mathcal{F}\{P_{\phi}\delta_x\}(\omega) = E_{k_{\gamma}}(-\mathrm{i}\langle\omega,x\rangle) F { P ϕ δ x } ( ω ) = E k γ ( − i ⟨ ω , x ⟩) with the truncated exponential
E K ( t ) : = e t − ∑ m = 0 K t m m ! . (57) E_K(t) := \mathrm{e}^{t} - \sum_{m=0}^{K}\frac{t^m}{m!}. \tag*{(57)} E K ( t ) := e t − m = 0 ∑ K m ! t m . (57)
By Parseval’s relation, the identity a ϕ ( x , y ) = ⟨ A P ϕ { δ y } , P ϕ { δ x } ⟩ a_{\phi}(x,y)=\langle A P_{\phi}\{\delta_y\},P_{\phi}\{\delta_x\}\rangle a ϕ ( x , y ) = ⟨ A P ϕ { δ y } , P ϕ { δ x }⟩ then yields the Fourier form of the covariance function
a ϕ ( x , y ) = 1 ( 2 π ) d ∫ R d E k γ ( − i ⟨ ω , x ⟩ ) E k γ ( − i ⟨ ω , y ⟩ ) ∥ ω ∥ 2 γ d ω , (58) a_{\phi}(x,y)=\frac{1}{(2\pi)^d}\int_{\mathbb{R}^d}\frac{E_{k_{\gamma}}(-i\langle\omega,x\rangle)E_{k_{\gamma}}(-i\langle\omega,y\rangle)}{\lVert\omega\rVert^{2\gamma}}\,\mathrm{d}\omega, \tag*{(58)} a ϕ ( x , y ) = ( 2 π ) d 1 ∫ R d ∥ ω ∥ 2 γ E k γ ( − i ⟨ ω , x ⟩) E k γ ( − i ⟨ ω , y ⟩) d ω , (58)
which is the specialization to sampling functionals of the quadratic form (85) of Appendix A, and from which the qualitative properties of the field can be read off directly.
Proposition 4.6 (Covariance of the fractional Brownian fields). Under the condition (47) , the covariance function of the field (55) is the double Taylor remainder
a ϕ ( x , y ) = c γ , d ( ∥ x − y ∥ 2 H − ∑ m = 0 k γ ( − 1 ) m m ! [ ( x ⋅ ∇ ) m f 2 H ( y ) + ( y ⋅ ∇ ) m f 2 H ( x ) ] ) , (59) a_{\phi}(x,y)=c_{\gamma,d}\left(\lVert x-y\rVert^{2H}-\sum_{m=0}^{k_{\gamma}}\frac{(-1)^m}{m!}\left[(x\cdot\nabla)^m f_{2H}(y)+(y\cdot\nabla)^m f_{2H}(x)\right]\right), \tag*{(59)} a ϕ ( x , y ) = c γ , d ∥ x − y ∥ 2 H − m = 0 ∑ k γ m ! ( − 1 ) m [ ( x ⋅ ∇ ) m f 2 H ( y ) + ( y ⋅ ∇ ) m f 2 H ( x ) ] , (59) where f 2 H = ∥ ⋅ ∥ 2 H f_{2H}=\lVert\cdot\rVert^{2H} f 2 H = ∥ ⋅ ∥ 2 H , where ( h ⋅ ∇ ) m (h\cdot\nabla)^m ( h ⋅ ∇ ) m denotes the m m m -fold directional derivative along h h h , and with c γ , d c_{\gamma,d} c γ , d given by (51) . These derivatives are given in closed form by (52) with λ = 2 H \lambda=2H λ = 2 H , and the factorials cancel upon substitution, so that (59) is an elementary finite combination of ∥ x ∥ \lVert x\rVert ∥ x ∥ , ∥ y ∥ \lVert y\rVert ∥ y ∥ , ⟨ x , y ⟩ \langle x,y\rangle ⟨ x , y ⟩ and ∥ x − y ∥ 2 H \lVert x-y\rVert^{2H} ∥ x − y ∥ 2 H . The field is isotropic and self-similar of exponent H H H at every order, with
E { ∣ B γ ( x ) ∣ 2 } = σ γ , d 2 ∥ x ∥ 2 H , σ γ , d 2 = − 2 ( − 1 ) k γ ( 2 H − 1 k γ ) c γ , d > 0. (60) \mathbb{E}\left\{\lvert B_{\gamma}(x)\rvert^2\right\}=\sigma_{\gamma,d}^2\lVert x\rVert^{2H},\qquad\sigma_{\gamma,d}^2=-2(-1)^{k_{\gamma}}\binom{2H-1}{k_{\gamma}}c_{\gamma,d}>0. \tag*{(60)} E { ∣ B γ ( x ) ∣ 2 } = σ γ , d 2 ∥ x ∥ 2 H , σ γ , d 2 = − 2 ( − 1 ) k γ ( k γ 2 H − 1 ) c γ , d > 0. (60) For k γ = 0 k_{\gamma}=0 k γ = 0 , (59) reduces to the classical form
a ϕ ( x , y ) = σ γ , d 2 2 ( ∥ x ∥ 2 H + ∥ y ∥ 2 H − ∥ x − y ∥ 2 H ) , σ γ , d 2 = 2 Γ ( 1 − H ) H 4 γ π d / 2 Γ ( γ ) . (61) a_{\phi}(x,y)=\frac{\sigma_{\gamma,d}^2}{2}\left(\lVert x\rVert^{2H}+\lVert y\rVert^{2H}-\lVert x-y\rVert^{2H}\right),\qquad\sigma_{\gamma,d}^2=\frac{2\Gamma(1-H)}{H4^{\gamma}\pi^{d/2}\Gamma(\gamma)}. \tag*{(61)} a ϕ ( x , y ) = 2 σ γ , d 2 ( ∥ x ∥ 2 H + ∥ y ∥ 2 H − ∥ x − y ∥ 2 H ) , σ γ , d 2 = H 4 γ π d /2 Γ ( γ ) 2Γ ( 1 − H ) . (61)
Proof. Formula (59) is (35) of Theorem 4.2 with r k , l = 0 r_{k,l}=0 r k , l = 0 and p k ( x ) v k ( y ) = c γ , d ( − x ) k k ! ∂ k f 2 H ( y ) p_k(x)v_k(y)=c_{\gamma,d}\frac{(-x)^k}{k!}\partial^k f_{2H}(y) p k ( x ) v k ( y ) = c γ , d k ! ( − x ) k ∂ k f 2 H ( y ) , in which the multinomial theorem,
∑ ∣ k ∣ = m h k k ! ∂ k = 1 m ! ( h ⋅ ∇ ) m , \sum_{\lvert k\rvert=m}\frac{h^k}{k!}\partial^k=\frac{1}{m!}(h\cdot\nabla)^m, ∣ k ∣ = m ∑ k ! h k ∂ k = m ! 1 ( h ⋅ ∇ ) m ,
collapses the sums over the N 0 N_0 N 0 multi-indices into the ( k γ + 1 ) (k_\gamma+ 1) ( k γ + 1 ) directional terms. The Fourier form (58) then delivers the qualitative properties: the substitution ω ↦ R ω \boldsymbol{\omega} \mapsto R\boldsymbol{\omega} ω ↦ R ω with R R R a rotation gives the isotropy; the dilation ω ↦ ω / s \boldsymbol{\omega} \mapsto\boldsymbol{\omega}/s ω ↦ ω / s gives a ϕ ( s x , s y ) = s 2 H a ϕ ( x , y ) a_\phi(s\boldsymbol{x},s\boldsymbol{y}) = s^{2H}a_\phi(\boldsymbol{x},\boldsymbol{y}) a ϕ ( s x , s y ) = s 2 H a ϕ ( x , y ) (self-similarity of exponent H H H ); and the bounds E k γ ( t ) = O ( t k γ + 1 ) E_{k_\gamma}(t) = O(t^{k_\gamma+1}) E k γ ( t ) = O ( t k γ + 1 ) at the origin and E k γ ( t ) = O ( t k γ ) E_{k_\gamma}(t) = O(t^{k_\gamma}) E k γ ( t ) = O ( t k γ ) at infinity (for imaginary t t t ) show that the integral is finite exactly when k γ < H < k γ + 1 k_\gamma< H < k_\gamma+ 1 k γ < H < k γ + 1 , which is the range (47) . The finiteness of the diagonal makes the pointwise reading legitimate, by Item 3 of Theorem 4.2 .
It remains to evaluate the covariance. Let us set y = x \boldsymbol{y} = \boldsymbol{x} y = x in (59) : the first term vanishes, while the homogeneity of f 2 H f_{2H} f 2 H gives, along the ray,
( x ⋅ ∇ ) m f 2 H ( x ) = d m d s m [ ( 1 + s ) 2 H ] ∣ s = 0 f 2 H ( x ) = 2 H ( 2 H − 1 ) ⋯ ( 2 H − m + 1 ) f 2 H ( x ) , (\boldsymbol{x} \cdot\nabla)^m f_{2H}(\boldsymbol{x}) = \left.\frac{\mathrm{d}^m}{\mathrm{d}s^m}\left[(1+s)^{2H}\right]\right|_{s=0} f_{2H}(\boldsymbol{x})
= 2H(2H-1)\cdots(2H-m+1) f_{2H}(\boldsymbol{x}), ( x ⋅ ∇ ) m f 2 H ( x ) = d s m d m [ ( 1 + s ) 2 H ] s = 0 f 2 H ( x ) = 2 H ( 2 H − 1 ) ⋯ ( 2 H − m + 1 ) f 2 H ( x ) ,
which is also the case h = x \boldsymbol{h} = \boldsymbol{x} h = x of (52) , so that
a ϕ ( x , x ) = − 2 c γ , d ∑ m = 0 k γ ( − 1 ) m ( 2 H m ) ∥ x ∥ 2 H . a_\phi(\boldsymbol{x},\boldsymbol{x}) = -2c_{\gamma,d}\sum_{m=0}^{k_\gamma}(-1)^m \binom{2H}{m}\lVert\boldsymbol{x}\rVert^{2H}. a ϕ ( x , x ) = − 2 c γ , d m = 0 ∑ k γ ( − 1 ) m ( m 2 H ) ∥ x ∥ 2 H .
The alternating-sum identity ∑ m = 0 K ( − 1 ) m ( α m ) = ( − 1 ) K ( α − 1 K ) \sum_{m=0}^{K}(-1)^m\binom{\alpha}{m} = (-1)^K\binom{\alpha-1}{K} ∑ m = 0 K ( − 1 ) m ( m α ) = ( − 1 ) K ( K α − 1 ) then gives (60) . The result is positive: the factors 2 H − 1 , … , 2 H − k γ 2H-1,\ldots,2H-k_\gamma 2 H − 1 , … , 2 H − k γ of the generalized binomial coefficient are positive because 2 H > 2 k γ 2H > 2k_\gamma 2 H > 2 k γ , while the sign of Γ ( − H ) \Gamma(-H) Γ ( − H ) , and hence of c γ , d c_{\gamma,d} c γ , d , is ( − 1 ) k γ + 1 (-1)^{k_\gamma+1} ( − 1 ) k γ + 1 .
For d = 1 d = 1 d = 1 , Proposition 4.6 reduces to Proposition A.2 of Appendix § 7 , whose variance formula (92) the alternating-sum identity above condenses into the single generalized binomial coefficient of (60) , as one verifies with the duplication and reflection formulas of the Gamma function. In particular, the d = 1 d = 1 d = 1 case of (59) is the covariance of the n n n th-order fractional Brownian motion of Perrin et al. [43 ] with n = k γ + 1 n = k_\gamma+ 1 n = k γ + 1 . By contrast, for d ≥ 2 d \ge2 d ≥ 2 and H > 1 H > 1 H > 1 , the classical literature treats the isotropic self-similar Gaussian fields as generalized random fields that are defined only modulo the polynomials of degree k γ k_\gamma k γ , whether through the spectral representation [11 , 15 , 32 ] or through the generalized covariances of Matheron’s intrinsic random functions of order k k k [35 ] . The conditioning by ϕ \phi ϕ removes this ambiguity, and this is what gives access to the pointwise formula (59) . To the best of our knowledge, the explicit description that (54) , (56) and (59) provide for d ≥ 2 d \ge2 d ≥ 2 and k γ ≥ 1 k_\gamma\ge1 k γ ≥ 1 is new. The operator-based characterizations of the fractional Brownian fields [54 ] specify these processes through their whitening operator and their spectral measure rather than through a pointwise covariance, and the closed forms that are available in the literature are the case k γ = 0 k_\gamma= 0 k γ = 0 of (60), which is Lévy’s field, and the one-dimensional formula of [43 ] .
Anticipating § 5 , Theorem 5.3 will identify the MMSE estimator of the field from the noisy samples y m = B γ ( x m ) + ϵ m y_m = B_\gamma(x_m) + \epsilon_m y m = B γ ( x m ) + ϵ m as the constrained spline
g ~ λ = ∑ m = 1 M a m a ϕ ( ⋅ , x m ) , a = ( G ~ + λ I ) − 1 y , λ = σ 2 , (62) \widetilde{g}_\lambda= \sum_{m=1}^{M} a_m a_\phi(\cdot,x_m), \qquad\mathbf{a} = (\widetilde{\mathbf{G}}+\lambda\mathbf{I})^{-1}\mathbf{y}, \qquad\lambda= \sigma^2, \tag*{(62)} g λ = m = 1 ∑ M a m a ϕ ( ⋅ , x m ) , a = ( G + λ I ) − 1 y , λ = σ 2 , (62)
with [ G ~ ] m , n = a ϕ ( x m , x n ) [\widetilde{\mathbf{G}}]_{m,n} = a_\phi(x_m,x_n) [ G ] m , n = a ϕ ( x m , x n ) . These are the polyharmonic (or fractional thin-plate) splines of Duchon [12 ] , whose radial basis functions ∥ ⋅ − x m ∥ 2 H \lVert\cdot-x_m\rVert^{2H} ∥ ⋅ − x m ∥ 2 H are Π k γ \Pi_{k_\gamma} Π k γ -conditionally positive-definite in the sense of Definition 4.3 , the boundary corrections being carried by a ϕ a_\phi a ϕ . The excluded integer values of H H H in (46) are precisely those of the classical thin-plate splines, whose radial functions (e.g., ∥ x ∥ 2 log ∥ x ∥ \lVert x\rVert^2\log\lVert x\rVert ∥ x ∥ 2 log ∥ x ∥ for γ = 2 \gamma=2 γ = 2 and d = 2 d=2 d = 2 ) acquire the logarithmic factor. The interpolation counterpart (Corollary 5.5 ) applies as soon as the deterministic observations ∂ k B γ ( 0 ) = 0 \partial^k B_\gamma(0)=0 ∂ k B γ ( 0 ) = 0 , ∣ k ∣ ≤ k γ \lvert k\rvert\leq k_\gamma ∣ k ∣ ≤ k γ , are part of the data.
Generalized Splines as MMSE Estimators of GGPs
We now have all the elements to make the connection between the minimum-error estimation of a signal under the Gaussian hypothesis—also known as the Wiener estimator —and the spline reconstruction techniques of [60 Section 4] . To that end, we consider the linear measurement model
g ↦ y = ν ( g ) + ϵ ∈ R M , (63) g \mapsto\mathbf{y} = \boldsymbol{\nu}(g)+\boldsymbol{\epsilon}\in\mathbb{R}^{M}, \tag*{(63)} g ↦ y = ν ( g ) + ϵ ∈ R M , (63)
where g g g (our signal) is a realization of a Gaussian process G ∼ N ( 0 , A ) G\sim\mathcal{N}(0,A) G ∼ N ( 0 , A ) in S ′ \mathcal{S}' S ′ and where ν = ( ν m ) \boldsymbol{\nu}=(\nu_m) ν = ( ν m ) with ν 1 , … , ν M ∈ H L ′ \nu_1,\ldots,\nu_M\in\mathcal{H}'_L ν 1 , … , ν M ∈ H L ′ is a continuous linear measurement operator ν : H L → R M \boldsymbol{\nu}:\mathcal{H}_L\to\mathbb{R}^{M} ν : H L → R M . The second component of the model, ϵ ∈ R M \boldsymbol{\epsilon}\in\mathbb{R}^{M} ϵ ∈ R M , is an additive disturbance term (discrete measurement noise) that does not depend on the signal and whose components are i.i.d. Gaussian with zero mean and variance σ 2 \sigma^2 σ 2 .
Estimation of Non-Degenerate GGPs
In the interest of clarity, we start with the “conventional” scenario A = ( L ∗ L ) − 1 : S → S ′ A=(L^*L)^{-1}:\mathcal{S}\to\mathcal{S}' A = ( L ∗ L ) − 1 : S → S ′ with L L L invertible (N 0 = 0 N_0=0 N 0 = 0 ), for which we have explicitly solved the deterministic inverse problem in [60 Section 4.3.1] . Specifically, we know that the Tikhonov estimator of g g g in (63) from the noisy measurements y \mathbf{y} y is the generalized smoothing spline
g λ = arg min g ∈ H L ( ∥ y − ν ( g ) ∥ 2 2 + λ ∥ L g ∥ L 2 2 ) = ∑ m = 1 M a m A { ν m } with ( a m ) = ( G + λ I ) − 1 y , (64) \begin{aligned}
g_{\lambda}=\underset{g\in\mathcal{H}_{L}}{\operatorname{arg\,min}}\left(\left\|\mathbf{y}-\boldsymbol{\nu}(g)\right\|_{2}^{2}+\lambda\left\|Lg\right\|_{L_{2}}^{2}\right) \\
&=\sum_{m=1}^{M}a_{m}A\{\nu_{m}\}\quad\text{with}\quad(a_{m})=(\mathbf{G}+\lambda\mathbf{I})^{-1}\mathbf{y}, \tag*{(64)}
\end{aligned} g λ = g ∈ H L arg min ( ∥ y − ν ( g ) ∥ 2 2 + λ ∥ Lg ∥ L 2 2 ) = m = 1 ∑ M a m A { ν m } with ( a m ) = ( G + λ I ) − 1 y , (64)
where the matrix G ∈ R M × M \mathbf{G}\in\mathbb{R}^{M\times M} G ∈ R M × M with [ G ] m , n = ⟨ A ν m , ν n ⟩ = ⟨ ν m , ν n ⟩ H L ′ [\mathbf{G}]_{m,n}=\langle A\nu_{m},\nu_{n}\rangle=\langle\nu_{m},\nu_{n}\rangle_{\mathcal{H}_{L}^{\prime}} [ G ] m , n = ⟨ A ν m , ν n ⟩ = ⟨ ν m , ν n ⟩ H L ′ is symmetric positive (semi-)definite.
Let us note that the matrix inverse in (64) is always well-defined for λ > 0 \lambda>0 λ > 0 . For the limit case of an exact fit with λ = 0 \lambda=0 λ = 0 , we need the additional assumption that G \mathbf{G} G is positive-definite, which is equivalent to the ν m \nu_{m} ν m being linearly independent. When this latter assumption fails, it suffices to reduce the (noise-free) data set by selecting a subset of r = rank ( G ) < M r=\operatorname{rank}(\mathbf{G})<M r = rank ( G ) < M measurements y ~ ∈ R r \widetilde{\mathbf{y}}\in\mathbb{R}^{r} y ∈ R r such that the reduced Gram matrix G ~ ∈ R r × r \widetilde{\mathbf{G}}\in\mathbb{R}^{r\times r} G ∈ R r × r is of full rank. This can be implemented systematically by replacing the ill-posed inverse of G \mathbf{G} G by the generalized Moore–Penrose inverse G + \mathbf{G}^{+} G + (see [2 pp. 284–295] ). Since A + = A − 1 \mathbf{A}^{+}=\mathbf{A}^{-1} A + = A − 1 when A \mathbf{A} A is invertible, we can therefore replace the right-hand-side of (64) by its foolproof variant a = ( G + + λ I ) + y \mathbf{a}=(\mathbf{G}^{+}+\lambda\mathbf{I})^{+}\mathbf{y} a = ( G + + λ I ) + y .
We now turn to the Bayesian counterpart: the MMSE estimator of g ( φ ) g(\varphi) g ( φ ) , for a fixed φ ∈ H L ′ \varphi\in\mathcal{H}_{L}^{\prime} φ ∈ H L ′ , given the measurements y \mathbf{y} y . In probabilistic terms, the model (63) reads Y = Y 0 + ϵ \mathbf{Y}=\mathbf{Y}_{0}+\boldsymbol{\epsilon} Y = Y 0 + ϵ with Y 0 = ( G ( ν 1 ) , … , G ( ν M ) ) \mathbf{Y}_{0}=(G(\nu_{1}),\ldots,G(\nu_{M})) Y 0 = ( G ( ν 1 ) , … , G ( ν M )) and ϵ ∼ N ( 0 , σ 2 I ) \boldsymbol{\epsilon}\sim\mathcal{N}(0,\sigma^{2}\mathbf{I}) ϵ ∼ N ( 0 , σ 2 I ) independent of G G G , while the quantity to estimate is the scalar random variable X = G ( φ ) X=G(\varphi) X = G ( φ ) . By Theorem 3.3, which covers the test functions φ , ν 1 , … , ν M ∈ H L ′ \varphi,\nu_{1},\ldots,\nu_{M}\in\mathcal{H}_{L}^{\prime} φ , ν 1 , … , ν M ∈ H L ′ , the vector ( X , Y 0 ) (X,\mathbf{Y}_{0}) ( X , Y 0 ) is Gaussian with the covariances ⟨ A φ , φ ⟩ \langle A\varphi,\varphi\rangle ⟨ A φ , φ ⟩ , ⟨ A φ , ν m ⟩ \langle A\varphi,\nu_{m}\rangle ⟨ A φ , ν m ⟩ , and ⟨ A ν m , ν n ⟩ \langle A\nu_{m},\nu_{n}\rangle ⟨ A ν m , ν n ⟩ ; the independence of the noise then gives
Z = ( X , Y ) ∼ N ( 0 , C Z ) , C Z = ( σ X 2 c T c G + σ 2 I ) , (65) \mathbf{Z}=(X,\mathbf{Y})\sim\mathcal{N}(0,\mathbf{C}_{Z}),\qquad
\mathbf{C}_{Z}=
\begin{pmatrix}
\sigma_{X}^{2} & \mathbf{c}^{\mathsf{T}}\\
\mathbf{c} & \mathbf{G}+\sigma^{2}\mathbf{I}
\end{pmatrix}, \tag*{(65)} Z = ( X , Y ) ∼ N ( 0 , C Z ) , C Z = ( σ X 2 c c T G + σ 2 I ) , (65)
with σ X 2 = ⟨ A φ , φ ⟩ \sigma_{X}^{2}=\langle A\varphi,\varphi\rangle σ X 2 = ⟨ A φ , φ ⟩ , [ c ] m = ⟨ A { ν m } , φ ⟩ [\mathbf{c}]_{m}=\langle A\{\nu_{m}\},\varphi\rangle [ c ] m = ⟨ A { ν m } , φ ⟩ , and [ G ] m , n = ⟨ A ν m , ν n ⟩ [\mathbf{G}]_{m,n}=\langle A\nu_{m},\nu_{n}\rangle [ G ] m , n = ⟨ A ν m , ν n ⟩ . Two features of (65) are noteworthy: the vector c = ν ∗ ( φ ) \mathbf{c}=\boldsymbol{\nu}^{*}(\varphi) c = ν ∗ ( φ ) collects the evaluations at φ \varphi φ of the Riesz representers ν ∗ = ( A { ν 1 } , … , A { ν M } ) \boldsymbol{\nu}^{*}=(A\{\nu_{1}\},\ldots,A\{\nu_{M}\}) ν ∗ = ( A { ν 1 } , … , A { ν M }) of the measurement functionals, and G \mathbf{G} G is the very Gram matrix of (64) . The conditional law of one block of a Gaussian vector given the other block is Gaussian, with mean and variance that are read off the partitioned covariance [5 Eqs. (2.81)–(2.82)] . Here, p X ∣ Y = y p_{X\mid\mathbf{Y}=\mathbf{y}} p X ∣ Y = y is the univariate Gaussian with mean
E { X ∣ y } = c T ( G + σ 2 I ) − 1 y = ν ∗ ( φ ) T ( G + σ 2 I ) − 1 y (66) \mathbb{E}\{X\mid\mathbf{y}\}=\mathbf{c}^{\mathsf{T}}(\mathbf{G}+\sigma^{2}\mathbf{I})^{-1}\mathbf{y}=\boldsymbol{\nu}^{*}(\varphi)^{\mathsf{T}}(\mathbf{G}+\sigma^{2}\mathbf{I})^{-1}\mathbf{y} \tag*{(66)} E { X ∣ y } = c T ( G + σ 2 I ) − 1 y = ν ∗ ( φ ) T ( G + σ 2 I ) − 1 y (66)
and variance σ X ∣ y 2 = ⟨ A φ , φ ⟩ − ν ∗ ( φ ) T ( G + σ 2 I ) − 1 ν ∗ ( φ ) \sigma^2_{X\mid y} = \langle A\varphi,\varphi\rangle- \boldsymbol{\nu}^{*}(\varphi)^{\mathsf{T}}(\mathbf{G}+\sigma^2\mathbf{I})^{-1}\boldsymbol{\nu}^{*}(\varphi) σ X ∣ y 2 = ⟨ A φ , φ ⟩ − ν ∗ ( φ ) T ( G + σ 2 I ) − 1 ν ∗ ( φ ) . Since the conditional mean is the MMSE estimator whatever the distribution, the comparison with (63) yields the following.
Theorem 5.1 (Generalized Tikhonov-Wahba-Wiener theorem). We consider the following setting.
The signal g = { g ( φ ) } φ ∈ S ∈ S ′ \mathbf{g}=\{g(\varphi)\}_{\varphi\in\mathcal{S}}\in\mathcal{S}' g = { g ( φ ) } φ ∈ S ∈ S ′ to be recovered is a realization of a generalized Gaussian process G ∼ N ( 0 , C L ) \mathbf{G}\sim\mathcal{N}(0,\mathbf{C}_{L}) G ∼ N ( 0 , C L ) in S ′ \mathcal{S}' S ′ with C L \mathbf{C}_{L} C L positive definite.
There exists an invertible whitening/regularization operator L L L such that C L = A = ( L ∗ L ) − 1 : S → S ′ \mathbf{C}_{L}=\mathbf{A}=(L^{*}L)^{-1}:\mathcal{S}\to\mathcal{S}' C L = A = ( L ∗ L ) − 1 : S → S ′ . This operator induces the native Hilbert space H L = ( S , ∥ L ⋅ ∥ L 2 ) \mathcal{H}_{L}=(\mathcal{S},\|L\cdot\|_{L^{2}}) H L = ( S , ∥ L ⋅ ∥ L 2 ) of Section 2.2, whose continuous dual H L ′ \mathcal{H}'_{L} H L ′ is equipped with the inner product ⟨ ν 1 , ν 2 ⟩ H L ′ = ⟨ A ν 1 , ν 2 ⟩ \langle\nu_{1},\nu_{2}\rangle_{\mathcal{H}'_{L}}=\langle A\nu_{1},\nu_{2}\rangle ⟨ ν 1 , ν 2 ⟩ H L ′ = ⟨ A ν 1 , ν 2 ⟩ .
The observed data are y = ν ( g ) + ϵ ∈ R M \mathbf{y}=\boldsymbol{\nu}(\mathbf{g})+\boldsymbol{\epsilon}\in\mathbb{R}^{M} y = ν ( g ) + ϵ ∈ R M , with ν = ( ν 1 , … , ν M ) ∈ ( H L ′ ) M \boldsymbol{\nu}=(\nu_{1},\ldots,\nu_{M})\in(\mathcal{H}'_{L})^{M} ν = ( ν 1 , … , ν M ) ∈ ( H L ′ ) M and ϵ \boldsymbol{\epsilon} ϵ a realization of a white Gaussian noise with variance σ 2 \sigma^{2} σ 2 .
Then, for any φ ∈ H L ′ ⊃ S \varphi\in\mathcal{H}'_{L}\supset\mathcal{S} φ ∈ H L ′ ⊃ S , the MMSE estimator of g ( φ ) g(\varphi) g ( φ ) given y \mathbf{y} y is
g M M S E ( φ ∣ y ) = E { G ( φ ) ∣ y } = g λ ( φ ) with λ = σ 2 , (67) g_{\mathrm{MMSE}}(\varphi\mid\mathbf{y})=\mathbb{E}\{G(\varphi)\mid\mathbf{y}\}=g_{\lambda}(\varphi)\quad\text{with}\quad\lambda=\sigma^{2}, \tag*{(67)} g MMSE ( φ ∣ y ) = E { G ( φ ) ∣ y } = g λ ( φ ) with λ = σ 2 , (67) where g λ ∈ H L g_{\lambda}\in\mathcal{H}_{L} g λ ∈ H L is the generalized smoothing spline defined by (63).
Since the result in Theorem 5.1 holds for any φ ∈ S \varphi\in\mathcal{S} φ ∈ S , it gives a precise meaning to the statement that the generalized spline g λ = { g λ ( φ ) : φ ∈ S } ∈ S ′ g_{\lambda}=\{g_{\lambda}(\varphi):\varphi\in\mathcal{S}\}\in\mathcal{S}' g λ = { g λ ( φ ) : φ ∈ S } ∈ S ′ with λ = σ 2 \lambda=\sigma^{2} λ = σ 2 is the optimal estimate of the generalized Gaussian process G = { G ( φ ) : φ ∈ S } \mathbf{G}=\{G(\varphi):\varphi\in\mathcal{S}\} G = { G ( φ ) : φ ∈ S } in S ′ \mathcal{S}' S ′ given the measurements y \mathbf{y} y . As for the case where H L \mathcal{H}_{L} H L is a RKHS of ordinary functions f : R d → R f:\mathbb{R}^{d}\to\mathbb{R} f : R d → R , we have that δ x 0 ∈ H L ′ \delta_{x_{0}}\in\mathcal{H}'_{L} δ x 0 ∈ H L ′ for any x 0 ∈ R d x_{0}\in\mathbb{R}^{d} x 0 ∈ R d . Consequently, we can plug φ = δ x 0 \varphi=\delta_{x_{0}} φ = δ x 0 in (66), which yields Wahba’s equivalence for classic Gaussian processes G = { G ( x ) } x ∈ R d \mathbf{G}=\{G(x)\}_{x\in\mathbb{R}^{d}} G = { G ( x ) } x ∈ R d whitened by L L L ; namely, that E { G ( x 0 ) ∣ y } = g λ ( x 0 ) \mathbb{E}\{G(x_{0})\mid\mathbf{y}\}=g_{\lambda}(x_{0}) E { G ( x 0 ) ∣ y } = g λ ( x 0 ) with λ = σ 2 \lambda=\sigma^{2} λ = σ 2 .
Estimation of GGPs with Non-Trivial Null Space
The variational side of the estimation is governed by the representer theorem [60 Theorem 4.3] , which states that the solution of a regularized inverse problem over H L \mathcal{H}_{L} H L with measurement functionals ν 1 , … , ν M ∈ H L ′ \nu_{1},\ldots,\nu_{M}\in\mathcal{H}'_{L} ν 1 , … , ν M ∈ H L ′ lies in the finite-dimensional space that is spanned by { A ϕ { ν m } } m = 1 M ∪ { p n } n = 1 N 0 \{A\phi\{\nu_{m}\}\}_{m=1}^{M}\cup\{p_{n}\}_{n=1}^{N_{0}} { A ϕ { ν m } } m = 1 M ∪ { p n } n = 1 N 0 . This requires the measurement functionals ν = ( ν 1 , … , ν M ) \boldsymbol{\nu}=(\nu_{1},\ldots,\nu_{M}) ν = ( ν 1 , … , ν M ) to be L L L -compatible [60 Definition 4.2] : the ν m \nu_{m} ν m are linearly independent elements of H L ′ \mathcal{H}'_{L} H L ′ , and the matrix P ∈ R M × N 0 \mathbf{P}\in\mathbb{R}^{M\times N_{0}} P ∈ R M × N 0 with [ P ] m , n = ⟨ ν m , p n ⟩ [\mathbf{P}]_{m,n}=\langle\nu_{m},p_{n}\rangle [ P ] m , n = ⟨ ν m , p n ⟩ is of rank N 0 N_{0} N 0 . We now present two variants of Theorem 5.1 for GGPs that are whitened by a p p p -admissible operator L \mathrm{L} L . The first is the direct generalization of a weaker variant of (67) for ordinary stochastic processes that states that L \mathrm{L} L -smoothing splines yield the best linear unbiased (BLU) estimator of a stochastic process { X ( t ) } \{X(t)\} { X ( t )} whitened by L \mathrm{L} L under the assumption of a fixed-effect model that includes an additional deterministic drift p ( ⋅ ) ∈ N p p(\cdot) \in\mathcal{N}_p p ( ⋅ ) ∈ N p [3 , 63 ] . In our formalism, the generalized version of this model reads
{ G = ∑ n = 1 N 0 θ n p n + X Y m = G ( ν m ) + ϵ m , m = 1 , … , M , (68) \begin{cases}
G = \sum_{n=1}^{N_0} \theta_n p_n + X \\
Y_m = G(\nu_m) + \epsilon_m,\qquad m=1,\ldots,M, \tag*{(68)}
\end{cases} { G = ∑ n = 1 N 0 θ n p n + X Y m = G ( ν m ) + ϵ m , m = 1 , … , M , (68)
where the θ n \theta_n θ n are fixed but unknown, ϵ = ( ϵ m ) ∼ N ( 0 , σ 2 I ) \epsilon= (\epsilon_m) \sim\mathcal{N}(0,\sigma^2 I) ϵ = ( ϵ m ) ∼ N ( 0 , σ 2 I ) , and where X ∼ N ( 0 , A ϕ ) X \sim\mathcal{N}(0,A_\phi) X ∼ N ( 0 , A ϕ ) is a GGP whose covariance operator A ϕ : S → S ′ A_\phi:\mathcal{S}\to\mathcal{S}' A ϕ : S → S ′ is the Gram operator (6) that is built from A = ( L ∗ L ) − 1 A=(\mathrm{L}^*\mathrm{L})^{-1} A = ( L ∗ L ) − 1 where L \mathrm{L} L is p p p -admissible with p = ( p 1 , … , p N 0 ) p=(p_1,\ldots,p_{N_0}) p = ( p 1 , … , p N 0 ) .
The BLU estimator of G ( φ ) G(\varphi) G ( φ ) in (68) given the observed data Y = y Y=y Y = y has the form g B L U ( φ ∣ y ) = β T y g_{\mathrm{BLU}}(\varphi\mid y)=\boldsymbol{\beta}^{\mathsf{T}}y g BLU ( φ ∣ y ) = β T y (linearity), with its coefficient-vector β = β ( φ ) ∈ R M \boldsymbol{\beta}=\boldsymbol{\beta}(\varphi)\in\mathbb{R}^{M} β = β ( φ ) ∈ R M being selected such as to minimize E { [ g B L U ( φ ∣ y ) − G ( φ ) ] 2 } \mathbb{E}\{[g_{\mathrm{BLU}}(\varphi\mid y)-G(\varphi)]^2\} E {[ g BLU ( φ ∣ y ) − G ( φ ) ] 2 } , subject to the zero-bias constraint E { g B L U ( φ ∣ y ) − G ( φ ) ∣ θ } = 0 \mathbb{E}\{g_{\mathrm{BLU}}(\varphi\mid y)-G(\varphi)\mid\boldsymbol{\theta}\}=0 E { g BLU ( φ ∣ y ) − G ( φ ) ∣ θ } = 0 for every θ ∈ R N 0 \boldsymbol{\theta}\in\mathbb{R}^{N_0} θ ∈ R N 0 . The relation of this estimator to splines is stated in Theorem 5.2. In the classical setting, the equivalence is stated in [63 Theorems 1.5.1–1.5.2] , with the proof deferred to the explicit computation of [27 ] ; an alternative route derives the smoothing spline as the limit of Bayes estimators under an improper prior on the null-space coefficients [63 Theorem 1.5.3] . The proof below is self-contained: it reduces the equivalence to the symmetry of a single matrix, without any explicit inversion.
Theorem 5.2 (Kimeldorf–Wahba [27 ] ). Let ν \nu ν be L \mathrm{L} L -compatible in the sense defined above, let σ 2 > 0 \sigma^2>0 σ 2 > 0 , and let g λ g_\lambda g λ be the generalized smoothing spline defined by
g λ = arg min g ∈ H L ( ∥ y − ν ( g ) ∥ 2 2 + λ ∥ L g ∥ 2 2 ) . (69) g_\lambda=\underset{g\in\mathcal{H}_{\mathrm{L}}}{\operatorname{arg\,min}}\left(\lVert y-\nu(g)\rVert_2^2+\lambda\lVert\mathrm{L}g\rVert_2^2\right). \tag*{(69)} g λ = g ∈ H L arg min ( ∥ y − ν ( g ) ∥ 2 2 + λ ∥ L g ∥ 2 2 ) . (69) Then, for any φ ∈ H L ′ \varphi\in\mathcal{H}_{\mathrm{L}}' φ ∈ H L ′ and λ = σ 2 \lambda=\sigma^2 λ = σ 2 , g λ ( φ ) = g B L U ( φ ∣ y ) g_\lambda(\varphi)=g_{\mathrm{BLU}}(\varphi\mid y) g λ ( φ ) = g BLU ( φ ∣ y ) where g B L U ( φ ∣ y ) g_{\mathrm{BLU}}(\varphi\mid y) g BLU ( φ ∣ y ) is the best linear unbiased estimator of G ( φ ) G(\varphi) G ( φ ) in (68) given y y y .
Proof. Throughout, we use the matrix P = [ ⟨ ν m , p n ⟩ ] ∈ R M × N 0 \mathbf{P}=\left[\langle\nu_m,p_n\rangle\right]\in\mathbb{R}^{M\times N_0} P = [ ⟨ ν m , p n ⟩ ] ∈ R M × N 0 (of rank N 0 N_0 N 0 , by L \mathrm{L} L -compatibility), the Gram matrix G ~ = [ ⟨ A ϕ { ν m } , ν n ⟩ ] ∈ R M × M \widetilde{\mathbf{G}}=\left[\langle A_\phi\{\nu_m\},\nu_n\rangle\right]\in\mathbb{R}^{M\times M} G = [ ⟨ A ϕ { ν m } , ν n ⟩ ] ∈ R M × M , and the two φ \varphi φ -dependent vectors c ~ = ( ⟨ A ϕ { ν m } , φ ⟩ ) ∈ R M \widetilde{\mathbf{c}}=\left(\langle A_\phi\{\nu_m\},\varphi\rangle\right)\in\mathbb{R}^{M} c = ( ⟨ A ϕ { ν m } , φ ⟩ ) ∈ R M and p ( φ ) = ( ⟨ p n , φ ⟩ ) ∈ R N 0 p(\varphi)=\left(\langle p_n,\varphi\rangle\right)\in\mathbb{R}^{N_0} p ( φ ) = ( ⟨ p n , φ ⟩ ) ∈ R N 0 .
Normal equations of the BLU estimator: In the model (67), we have that Y = P θ + ν ( X ) + ϵ Y = \mathbf{P}\theta+ \nu(X) + \epsilon Y = P θ + ν ( X ) + ϵ and E { G ( φ ) } = θ T p ( φ ) \mathbb{E}\{G(\varphi)\} = \theta^{\mathsf{T}}p(\varphi) E { G ( φ )} = θ T p ( φ ) , so that the bias of g B L U ( φ ∣ y ) = β T y g_{\mathrm{BLU}}(\varphi|y) = \beta^{\mathsf{T}}y g BLU ( φ ∣ y ) = β T y is ( P T β − p ( φ ) ) T θ (\mathbf{P}^{\mathsf{T}}\beta- p(\varphi))^{\mathsf{T}}\theta ( P T β − p ( φ ) ) T θ . The requirement that it vanishes for every θ ∈ R N 0 \theta\in\mathbb{R}^{N_0} θ ∈ R N 0 is the linear constraint P T β = p ( φ ) \mathbf{P}^{\mathsf{T}}\beta= p(\varphi) P T β = p ( φ ) , which is feasible because P \mathbf{P} P has rank N 0 N_0 N 0 . Under that constraint, the deterministic part cancels out from the error, which reduces to β T ( ν ( X ) + ϵ ) − X ( φ ) \beta^{\mathsf{T}}(\nu(X)+\epsilon)-X(\varphi) β T ( ν ( X ) + ϵ ) − X ( φ ) , with variance
β T ( G ~ + σ 2 I ) β − 2 β T c ~ + ⟨ A φ φ , φ ⟩ . \beta^{\mathsf{T}}(\widetilde{\mathbf{G}}+\sigma^2\mathbf{I})\beta- 2\beta^{\mathsf{T}}\widetilde{c}+\langle A_{\varphi}\varphi,\varphi\rangle. β T ( G + σ 2 I ) β − 2 β T c + ⟨ A φ φ , φ ⟩ .
where the second-order statistics of X X X are given by (20). The minimization of this quadratic form subject to P T β = p ( φ ) \mathbf{P}^{\mathsf{T}}\beta= p(\varphi) P T β = p ( φ ) yields, by Lagrange duality, the linear system
K ( β γ ) = ( c ~ p ( φ ) ) , K : = ( G ~ + σ 2 I P P T 0 ) . (70) \mathbf{K}\begin{pmatrix}\beta\\\gamma\end{pmatrix}
=
\begin{pmatrix}\widetilde{c}\\p(\varphi)\end{pmatrix},
\qquad
\mathbf{K}:=
\begin{pmatrix}
\widetilde{\mathbf{G}}+\sigma^2\mathbf{I} & \mathbf{P}\\
\mathbf{P}^{\mathsf{T}} & \mathbf{0}
\end{pmatrix}. \tag*{(70)} K ( β γ ) = ( c p ( φ ) ) , K := ( G + σ 2 I P T P 0 ) . (70)
whose symmetric matrix K \mathbf{K} K is invertible because G ~ + σ 2 I \widetilde{\mathbf{G}}+\sigma^2\mathbf{I} G + σ 2 I is positive-definite and P \mathbf{P} P is of rank N 0 N_0 N 0 .
Normal equations of the smoothing spline: By the representer theorem of [60 ] , as recalled in § 2.2 , the minimizer g λ g_\lambda g λ is unique and can be written as g λ = ∑ m = 1 M a m A φ { ν m } + ∑ n = 1 N 0 b n p n g_\lambda=\sum_{m=1}^{M}a_mA_\varphi\{\nu_m\}+\sum_{n=1}^{N_0}b_np_n g λ = ∑ m = 1 M a m A φ { ν m } + ∑ n = 1 N 0 b n p n . The two building blocks of the loss are then ν ( g λ ) = G ~ a + P b \nu(g_\lambda)=\widetilde{\mathbf{G}}a+\mathbf{P}b ν ( g λ ) = G a + P b and ∥ L g λ ∥ L 2 2 = a T G ~ a \|Lg_\lambda\|_{L_2}^2=a^{\mathsf{T}}\widetilde{\mathbf{G}}a ∥ L g λ ∥ L 2 2 = a T G a , where the second identity follows from L A φ = L φ − 1 ∗ LA_\varphi=L_\varphi^{-1*} L A φ = L φ − 1 ∗ and the factorization (8). The zeroing of the gradient of ∥ y − G ~ a − P b ∥ 2 2 + λ a T G ~ a \|y-\widetilde{\mathbf{G}}a-\mathbf{P}b\|_2^2+\lambda a^{\mathsf{T}}\widetilde{\mathbf{G}}a ∥ y − G a − P b ∥ 2 2 + λ a T G a gives the stationarity conditions G ~ [ ( G ~ + λ I ) a + P b − y ] = 0 \widetilde{\mathbf{G}}[(\widetilde{\mathbf{G}}+\lambda\mathbf{I})a+\mathbf{P}b-y]=0 G [( G + λ I ) a + P b − y ] = 0 and P T [ G ~ a + P b − y ] = 0 \mathbf{P}^{\mathsf{T}}[\widetilde{\mathbf{G}}a+\mathbf{P}b-y]=0 P T [ G a + P b − y ] = 0 , and one verifies directly that these are satisfied by the solution of
K ( a b ) = ( y 0 ) with λ = σ 2 , (71) \mathbf{K}\begin{pmatrix}a\\b\end{pmatrix}
=
\begin{pmatrix}y\\0\end{pmatrix}
\quad\text{with }\lambda=\sigma^2, \tag*{(71)} K ( a b ) = ( y 0 ) with λ = σ 2 , (71)
since the latter gives ( G ~ + λ I ) a + P b = y (\widetilde{\mathbf{G}}+\lambda\mathbf{I})a+\mathbf{P}b=y ( G + λ I ) a + P b = y and P T a = 0 \mathbf{P}^{\mathsf{T}}a=0 P T a = 0 , hence P T [ G ~ a + P b − y ] = − λ P T a = 0 \mathbf{P}^{\mathsf{T}}[\widetilde{\mathbf{G}}a+\mathbf{P}b-y]=-\lambda\mathbf{P}^{\mathsf{T}}a=0 P T [ G a + P b − y ] = − λ P T a = 0 . The loss is convex, so that stationarity suffices, and the evaluation of the solution reads g λ ( φ ) = a T c ~ + b T p ( φ ) g_\lambda(\varphi)=a^{\mathsf{T}}\widetilde{c}+b^{\mathsf{T}}p(\varphi) g λ ( φ ) = a T c + b T p ( φ ) .
Equivalence: The two estimators are driven by the same symmetric invertible matrix K \mathbf{K} K , with the roles of the data and of the test function exchanged. Specifically,
g λ ( φ ) = ( c ~ p ( φ ) ) T K − 1 ( y 0 ) = ( y 0 ) T K − 1 ( c ~ p ( φ ) ) = β T y = g B L U ( φ ∣ y ) , g_\lambda(\varphi)=
\begin{pmatrix}\widetilde{c}\\p(\varphi)\end{pmatrix}^{\mathsf{T}}
\mathbf{K}^{-1}
\begin{pmatrix}y\\0\end{pmatrix}
=
\begin{pmatrix}y\\0\end{pmatrix}^{\mathsf{T}}
\mathbf{K}^{-1}
\begin{pmatrix}\widetilde{c}\\p(\varphi)\end{pmatrix}
=\beta^{\mathsf{T}}y=g_{\mathrm{BLU}}(\varphi|y), g λ ( φ ) = ( c p ( φ ) ) T K − 1 ( y 0 ) = ( y 0 ) T K − 1 ( c p ( φ ) ) = β T y = g BLU ( φ ∣ y ) ,
where the middle equality is the symmetry of K − 1 \mathbf{K}^{-1} K − 1 .
Theorem 5.2 is remarkable in that it shows the complete equivalence between generalized smoothing splines and BLU estimators for GGPs. The same theorem clarifies the status of the polyharmonic splines of Duchon [12 ] for d ≥ 3 d \ge3 d ≥ 3 , whose polynomial part ranges over Π m − 1 ⊃ N p \Pi_{m-1} \supset\mathcal{N}_p Π m − 1 ⊃ N p [59 Remark 6.2] : the components in the excess polynomials are fixed effects in the sense of (68) , so that these splines are BLU estimators, but they are not the MMSE estimator of any GGP; conditioning the process on Π m − 1 \Pi_{m-1} Π m − 1 leads instead to the constrained spline of Theorem 5.3 , which annihilates these polynomials.
Our second variant of Theorem 5.1 finalizes this functional picture by providing the corresponding MMSE (or Bayesian) estimator in full generality. The latter also fits the description of a generalized spline, but of a constrained type that fulfills prescribed boundary conditions. For p = { 0 } p = \{0\} p = { 0 } (trivial null space), we recover Theorem 5.1 , with the MMSE and BLU estimators being equivalent.
Theorem 5.3 (MMSE estimation of a GGP). We consider the following setting.
The signal g = { g ( φ ) } φ ∈ S ∈ S ′ g = \{g(\varphi)\}_{\varphi\in\mathcal{S}} \in\mathcal{S}' g = { g ( φ ) } φ ∈ S ∈ S ′ to be recovered is a realization of a generalized Gaussian process G ∼ N ( 0 , C L , ϕ ) G \sim\mathcal{N}(0, C_{L,\phi}) G ∼ N ( 0 , C L , ϕ ) in S ′ \mathcal{S}' S ′ whose covariance operator C L , ϕ = A ϕ C_{L,\phi} = A_{\phi} C L , ϕ = A ϕ is the Gram operator (6) of a p p p -admissible operator L L L ; by Theorem 3.3, the action of G G G extends to test functions φ ∈ H L ′ \varphi\in\mathcal{H}'_L φ ∈ H L ′ .
The observed data are y = ν ( g ) + ϵ ∈ R M \mathbf{y} = \boldsymbol{\nu}(g) + \boldsymbol{\epsilon} \in\mathbb{R}^{M} y = ν ( g ) + ϵ ∈ R M , with ν = ( ν 1 , … , ν M ) ∈ ( H L ′ ) M \boldsymbol{\nu} = (\nu_1, \ldots, \nu_M) \in(\mathcal{H}'_L)^M ν = ( ν 1 , … , ν M ) ∈ ( H L ′ ) M and ϵ \boldsymbol{\epsilon} ϵ a realization of a white Gaussian noise with variance σ 2 \sigma^2 σ 2 .
Then, for any φ ∈ H L ′ ⊃ S \varphi\in\mathcal{H}'_L \supset\mathcal{S} φ ∈ H L ′ ⊃ S , the MMSE estimator of g ( φ ) g(\varphi) g ( φ ) given y = ( y m ) \mathbf{y} = (y_m) y = ( y m ) is
g M M S E ( φ ∣ y ) = E { G ( φ ) ∣ y } = g ~ λ ( φ ) = ⟨ g ~ λ , φ ⟩ with λ = σ 2 , (72) g_{\mathrm{MMSE}}(\varphi\mid\mathbf{y}) = \mathbb{E}\{G(\varphi)\mid\mathbf{y}\} = \widetilde{g}_{\lambda}(\varphi) = \langle\widetilde{g}_{\lambda}, \varphi\rangle\quad\text{with} \quad\lambda= \sigma^2, \tag*{(72)} g MMSE ( φ ∣ y ) = E { G ( φ ) ∣ y } = g λ ( φ ) = ⟨ g λ , φ ⟩ with λ = σ 2 , (72) where g ~ λ ∈ H L \widetilde{g}_{\lambda} \in\mathcal{H}_L g λ ∈ H L is the constrained generalized spline specified by
g ~ λ = arg min g ∈ H L ( ∑ m = 1 M ∣ y m − ⟨ ν m , g ⟩ ∣ 2 + λ ∥ L g ∥ L 2 2 ) s.t. ϕ ( g ) = 0 (73) \widetilde{g}_{\lambda} = \underset{g \in\mathcal{H}_L}{\operatorname{arg\,min}}\left(\sum_{m=1}^{M}\left|y_m-\langle\nu_m,g\rangle\right|^2+\lambda\lVert Lg\rVert_{L_2}^2\right) \quad\text{s.t.} \quad\phi(g)=0 \tag*{(73)} g λ = g ∈ H L arg min ( m = 1 ∑ M ∣ y m − ⟨ ν m , g ⟩ ∣ 2 + λ ∥ Lg ∥ L 2 2 ) s.t. ϕ ( g ) = 0 (73) = ∑ m = 1 M a m A ϕ { ν m } with ( a m ) = ( G ~ + λ I ) − 1 y , (74) = \sum_{m=1}^{M} a_m A_{\phi}\{\nu_m\} \quad\text{with} \quad(a_m)=(\widetilde{G}+\lambda I)^{-1}\mathbf{y}, \tag*{(74)} = m = 1 ∑ M a m A ϕ { ν m } with ( a m ) = ( G + λ I ) − 1 y , (74) where G ~ ∈ R M × M \widetilde{G} \in\mathbb{R}^{M \times M} G ∈ R M × M with [ G ~ ] m , n = ⟨ A ϕ ν m , ν n ⟩ [\widetilde{G}]_{m,n}=\langle A_{\phi}\nu_m,\nu_n\rangle [ G ] m , n = ⟨ A ϕ ν m , ν n ⟩ .
Proof. Let φ ~ = P ϕ { φ } \tilde{\varphi}=P_{\phi}\{\varphi\} φ ~ = P ϕ { φ } and ν ~ m = P ϕ { ν m } \tilde{\nu}_{m}=P_{\phi}\{\nu_{m}\} ν ~ m = P ϕ { ν m } denote the projections onto H A \mathcal{H}_{A} H A . By Theorem 3.3 , G ( φ ) = G ( φ ~ ) G(\varphi)=G(\tilde{\varphi}) G ( φ ) = G ( φ ~ ) and G ( ν m ) = G ( ν ~ m ) G(\nu_{m})=G(\tilde{\nu}_{m}) G ( ν m ) = G ( ν ~ m ) almost surely, so that the measurement model is almost surely y = ν ~ ( g ) + ϵ y=\tilde{\nu}(g)+\epsilon y = ν ~ ( g ) + ϵ and the quantity to estimate is G ( φ ~ ) G(\tilde{\varphi}) G ( φ ~ ) . We now read this model on the dual pair ( H L , ϕ , H A ) (\mathcal{H}_{L,\phi},\mathcal{H}_{A}) ( H L , ϕ , H A ) , whose Riesz map is A ϕ A_{\phi} A ϕ (see (10) ). On this pair, L : H L , ϕ → L 2 L:\mathcal{H}_{L,\phi}\to L_{2} L : H L , ϕ → L 2 is invertible with inverse L ϕ − 1 L^{-1}_{\phi} L ϕ − 1 , the norm is ∥ g ~ ∥ H L , ϕ = ∥ L g ~ ∥ L 2 \|\tilde{g}\|_{\mathcal{H}_{L,\phi}}=\|L\tilde{g}\|_{L_{2}} ∥ g ~ ∥ H L , ϕ = ∥ L g ~ ∥ L 2 by (9) , and the restriction of G G G to H A \mathcal{H}_{A} H A is the non-degenerate GGP with covariance A ϕ = ( L ∗ L ) − 1 A_{\phi}=(L^{*}L)^{-1} A ϕ = ( L ∗ L ) − 1 , by (8) and (20) . The derivation of Theorem 5.1 uses nothing but these three ingredients, and it therefore applies verbatim: with λ = σ 2 \lambda=\sigma^{2} λ = σ 2 ,
E { G ( φ ) ∣ y } = E { G ( φ ~ ) ∣ y } = g ~ λ ( φ ~ ) , \mathbb{E}\{G(\varphi)\mid y\}=\mathbb{E}\{G(\tilde{\varphi})\mid y\}=\tilde{g}_{\lambda}(\tilde{\varphi}), E { G ( φ ) ∣ y } = E { G ( φ ~ ) ∣ y } = g ~ λ ( φ ~ ) ,
g ~ λ = ∑ m = 1 M a m A ϕ { ν ~ m } with a = ( G ~ + λ I ) − 1 y , \tilde{g}_{\lambda}=\sum_{m=1}^{M}a_{m}A_{\phi}\{\tilde{\nu}_{m}\}\quad\text{with}\quad\mathbf{a}=(\tilde{\mathbf{G}}+\lambda\mathbf{I})^{-1}\mathbf{y}, g ~ λ = m = 1 ∑ M a m A ϕ { ν ~ m } with a = ( G ~ + λ I ) − 1 y ,
where g ~ λ \tilde{g}_{\lambda} g ~ λ is the minimizer of ∑ m = 1 M ∣ y m − ⟨ ν ~ m , g ~ ⟩ ∣ 2 + λ ∥ L g ~ ∥ L 2 2 \sum_{m=1}^{M}|y_{m}-\langle\tilde{\nu}_{m},\tilde{g}\rangle|^{2}+\lambda\|L\tilde{g}\|_{L_{2}}^{2} ∑ m = 1 M ∣ y m − ⟨ ν ~ m , g ~ ⟩ ∣ 2 + λ ∥ L g ~ ∥ L 2 2 over H L , ϕ = { g ∈ H L : ϕ ( g ) = 0 } \mathcal{H}_{L,\phi}=\{g\in\mathcal{H}_{L}:\phi(g)=0\} H L , ϕ = { g ∈ H L : ϕ ( g ) = 0 } . It remains to remove the tildes. Since N ϕ \mathcal{N}_{\phi} N ϕ annihilates H L , ϕ \mathcal{H}_{L,\phi} H L , ϕ , we have that ⟨ ν ~ m , g ~ ⟩ = ⟨ ν m , g ~ ⟩ \langle\tilde{\nu}_{m},\tilde{g}\rangle=\langle\nu_{m},\tilde{g}\rangle ⟨ ν ~ m , g ~ ⟩ = ⟨ ν m , g ~ ⟩ and g ~ λ ( φ ~ ) = g ~ λ ( φ ) \tilde{g}_{\lambda}(\tilde{\varphi})=\tilde{g}_{\lambda}(\varphi) g ~ λ ( φ ~ ) = g ~ λ ( φ ) , which identifies the minimization problem with (73) . Likewise, A ϕ { ν ~ m } = A ϕ { ν m } A_{\phi}\{\tilde{\nu}_{m}\}=A_{\phi}\{\nu_{m}\} A ϕ { ν ~ m } = A ϕ { ν m } and ⟨ A ϕ { ν ~ m } , ν ~ n ⟩ = ⟨ A ϕ { ν m } , ν n ⟩ = [ G ~ ] m , n \langle A_{\phi}\{\tilde{\nu}_{m}\},\tilde{\nu}_{n}\rangle=\langle A_{\phi}\{\nu_{m}\},\nu_{n}\rangle=[\tilde{\mathbf{G}}]_{m,n} ⟨ A ϕ { ν ~ m } , ν ~ n ⟩ = ⟨ A ϕ { ν m } , ν n ⟩ = [ G ~ ] m , n , because A ϕ = A ϕ P ϕ A_{\phi}=A_{\phi}P_{\phi} A ϕ = A ϕ P ϕ by (6) , which gives (74) . Let us note that, for λ > 0 \lambda>0 λ > 0 , no linear independence of the ν ~ m \tilde{\nu}_{m} ν ~ m is needed, since G ~ + λ I \tilde{\mathbf{G}}+\lambda\mathbf{I} G ~ + λ I is always invertible.
To the best of our knowledge, Theorem 5.3 is new at this level of generality, precisely because its formulation requires the proper definition of the corresponding GGP.
Remark 5.4 (Exact fit). Theorem 5.3 remains valid in the noise-free scenario σ 2 = λ = 0 \sigma^{2}=\lambda=0 σ 2 = λ = 0 , provided that the Gram matrix G ~ \tilde{\mathbf{G}} G ~ is invertible. Indeed, the conditioning argument of the proof only requires the invertibility of C Y ~ = G ~ + σ 2 I \mathbf{C}_{\tilde{Y}}=\tilde{\mathbf{G}}+\sigma^{2}\mathbf{I} C Y ~ = G ~ + σ 2 I , while the coefficients become a = G ~ − 1 y \mathbf{a}=\tilde{\mathbf{G}}^{-1}\mathbf{y} a = G ~ − 1 y , so that the estimator fits the data exactly, with ⟨ ν m , g ~ 0 ⟩ = [ G ~ a ] m = y m \langle\nu_{m},\tilde{g}_{0}\rangle=[\tilde{\mathbf{G}}\mathbf{a}]_{m}=y_{m} ⟨ ν m , g ~ 0 ⟩ = [ G ~ a ] m = y m . The corresponding variational characterization is the constrained interpolation problem
g ~ 0 = arg min g ∈ H L ∥ L g ∥ L 2 s.t. ν ( g ) = y and ϕ ( g ) = 0 , \tilde{g}_{0}=\underset{g\in\mathcal{H}_{L}}{\arg\min}\ \|Lg\|_{L_{2}}\quad\text{s.t.}\quad\boldsymbol{\nu}(g)=\mathbf{y}\ \text{and}\ \phi(g)=0, g ~ 0 = g ∈ H L arg min ∥ Lg ∥ L 2 s.t. ν ( g ) = y and ϕ ( g ) = 0 ,
which follows from [60 Theorem 4.1] applied to H L , ϕ \mathcal{H}_{L,\phi} H L , ϕ , whose Riesz map is A ϕ A_{\phi} A ϕ .
For the limit case λ → 0 \lambda\to0 λ → 0 , we can formulate a simpler version of Theorem 5.3 that fills the gap that was stated in the introduction by showing that every generalized spline interpolant is the MMSE estimator of a corresponding GGP. To that end, we impose the condition N ϕ ⊂ N ν = span { ν m } m = 1 M N_{\phi} \subset N_{\nu} = \operatorname{span}\{\nu_m\}_{m=1}^{M} N ϕ ⊂ N ν = span { ν m } m = 1 M , which means that the observations of the process must be rich enough to span the boundary conditions ϕ ( G ) = 0 \phi(G) = 0 ϕ ( G ) = 0 .
Corollary 5.5 (MMSE interpolation of a GGP). Let Y m = G ( ν m ) Y_m = G(\nu_m) Y m = G ( ν m ) , m = 1 , … , M m = 1,\ldots,M m = 1 , … , M be a finite collection of observations of a generalized Gaussian process G ∼ N ( 0 , A ϕ ) G \sim\mathcal{N}(0,A_{\phi}) G ∼ N ( 0 , A ϕ ) in S ′ \mathcal{S}' S ′ that is whitened by a p p p -admissible operator L L L , where the functionals ν 1 , … , ν M ∈ H L ′ \nu_1,\ldots,\nu_M \in\mathcal{H}'_L ν 1 , … , ν M ∈ H L ′ are linearly independent. Then, under the condition that N ϕ ⊂ span { ν m } m = 1 M N_{\phi} \subset\operatorname{span}\{\nu_m\}_{m=1}^{M} N ϕ ⊂ span { ν m } m = 1 M , the MMSE interpolator of G G G given the observed data Y = y Y = y Y = y is the generalized spline interpolant
g 0 = arg min g ∈ H L ∥ L g ∥ L 2 s.t. ν ( g 0 ) = y . (75) g_0 = \underset{g \in\mathcal{H}_L}{\operatorname{arg\,min}}\ \|Lg\|_{L_2}
\quad\text{s.t.} \quad\boldsymbol{\nu}(g_0) = \mathbf{y}. \tag*{(75)} g 0 = g ∈ H L arg min ∥ Lg ∥ L 2 s.t. ν ( g 0 ) = y . (75) It is such that E { G ( φ ) ∣ y } = g 0 ( φ ) \mathbb{E}\{G(\varphi)\mid\mathbf{y}\} = g_0(\varphi) E { G ( φ ) ∣ y } = g 0 ( φ ) for any φ ∈ H L ′ ⊃ S \varphi\in\mathcal{H}'_L \supset\mathcal{S} φ ∈ H L ′ ⊃ S and almost every realization y \mathbf{y} y of the data.
Proof. The hypothesis provides a matrix P ~ ∈ R M × N 0 \widetilde{\mathbf{P}} \in\mathbb{R}^{M \times N_0} P ∈ R M × N 0 such that P ~ T ν = ϕ \widetilde{\mathbf{P}}^{\mathsf{T}}\boldsymbol{\nu} = \boldsymbol{\phi} P T ν = ϕ ; since P ~ T P = ϕ ( p ) = I N 0 \widetilde{\mathbf{P}}^{\mathsf{T}}\mathbf{P} = \boldsymbol{\phi}(\mathbf{p}) = \mathbf{I}_{N_0} P T P = ϕ ( p ) = I N 0 , the matrix P = ν ( p ) \mathbf{P} = \boldsymbol{\nu}(\mathbf{p}) P = ν ( p ) has rank N 0 N_0 N 0 and [60 Proposition 4.4] applies with the choice ϕ ~ = ϕ \widetilde{\boldsymbol{\phi}} = \boldsymbol{\phi} ϕ = ϕ . It provides the change of coordinates ν = Q ν ~ + P ϕ \boldsymbol{\nu} = \mathbf{Q}\widetilde{\boldsymbol{\nu}} + \mathbf{P}\boldsymbol{\phi} ν = Q ν + P ϕ , in which the data are equivalent to the pair ( y ~ , P ~ T y ) = ( Q ~ T y , ϕ ( G ) ) (\widetilde{\mathbf{y}},\widetilde{\mathbf{P}}^{\mathsf{T}}\mathbf{y}) = (\widetilde{\mathbf{Q}}^{\mathsf{T}}\mathbf{y},\boldsymbol{\phi}(G)) ( y , P T y ) = ( Q T y , ϕ ( G )) , with ν ~ = Q ~ T ν ∈ H A M − N 0 \widetilde{\boldsymbol{\nu}} = \widetilde{\mathbf{Q}}^{\mathsf{T}}\boldsymbol{\nu} \in\mathcal{H}^{M-N_0}_A ν = Q T ν ∈ H A M − N 0 . The second block vanishes almost surely (Theorem 3.3 ), so that it carries no information—the σ \sigma σ -algebras generated by Y Y Y and by Y ~ = ν ~ ( G ) \widetilde{Y} = \widetilde{\boldsymbol{\nu}}(G) Y = ν ( G ) coincide—while, on the deterministic side, it turns the interpolation constraints ν ( g ) = y \boldsymbol{\nu}(g) = \mathbf{y} ν ( g ) = y into ν ~ ( g ) = y ~ \widetilde{\boldsymbol{\nu}}(g) = \widetilde{\mathbf{y}} ν ( g ) = y together with the boundary conditions ϕ ( g ) = 0 \boldsymbol{\phi}(g) = 0 ϕ ( g ) = 0 . In particular, the feasible set of the interpolation problem is almost surely contained in H L , ϕ \mathcal{H}_{L,\boldsymbol{\phi}} H L , ϕ , so that g 0 g_0 g 0 coincides with the constrained interpolant of the reduced data y ~ \widetilde{\mathbf{y}} y .
It then suffices to apply Theorem 5.3 with an exact fit (Remark 5.4) to the reduced, noise-free measurement model Y ~ = ν ~ ( G ) \widetilde{Y} = \widetilde{\boldsymbol{\nu}}(G) Y = ν ( G ) . The corresponding Gram matrix G ~ = [ ⟨ A ϕ { ν ~ m } , ν ~ n ⟩ ] ∈ R ( M − N 0 ) × ( M − N 0 ) \widetilde{\mathbf{G}} = [\langle A_{\phi}\{\widetilde{\nu}_m\},\widetilde{\nu}_n\rangle] \in\mathbb{R}^{(M-N_0)\times(M-N_0)} G = [⟨ A ϕ { ν m } , ν n ⟩] ∈ R ( M − N 0 ) × ( M − N 0 ) is invertible, because the ν ~ m \widetilde{\nu}_m ν m are linearly independent (Q ~ \widetilde{\mathbf{Q}} Q has full column rank and the ν m \nu_m ν m are linearly independent) and A ϕ A_{\phi} A ϕ is the Riesz map of H A \mathcal{H}_A H A . Hence E { G ( φ ) ∣ Y ~ = y ~ } = g ~ 0 ( φ ) \mathbb{E}\{G(\varphi)\mid\widetilde{\mathbf{Y}} = \widetilde{\mathbf{y}}\} = \widetilde{g}_0(\varphi) E { G ( φ ) ∣ Y = y } = g 0 ( φ ) , where g ~ 0 \widetilde{g}_0 g 0 is the constrained interpolant of y ~ \widetilde{\mathbf{y}} y ; that is, g ~ 0 = g 0 \widetilde{g}_0 = g_0 g 0 = g 0 . □
Conclusion
We conclude the paper by emphasizing the crucial role of the regularization/whitening operator L L L and of its native space H L \mathcal{H}_L H L . As for the deterministic part of the story, the generalized forms of L L L -splines are the solutions of regularized linear inverse problems in H L \mathcal{H}_L H L . Their stochastic counterparts are precisely the GGPs whitened by L. (The delicate part in this statement is the necessity to invert L, which requires the specification of boundary conditions.) The pleasing outcome is the mathematical congruence between the deterministic and stochastic worlds. The present framework provides a rigorous support to statements such as “fractional splines are optimal estimators for fractional Brownian motion.” Likewise, we can now identify the polynomial splines of degree 2 n − 1 2n-1 2 n − 1 as the optimal estimators of Gaussian processes, which, up to some boundary conditions (the technical part of the story), are the n n n -fold integration of a white noise, or, for more classic folks, the ( n − 1 ) (n-1) ( n − 1 ) -fold integration of a Brownian motion. With the help of abstraction and nuclear spaces, we ultimately end up with an equivalence in the continuum that is as strong as in finite dimensions.
Higher-Order fBm and Fractional Splines
Section 3.3 presented fractional Brownian motion (fBm) as the prototypical generalized Gaussian process with a one-dimensional null space. We now carry out the same construction for an arbitrary order, which yields the family of higher-order fractional Brownian motions [43 ] , and we make the three objects around which the theory revolves fully explicit: the conditioned inverse L ϕ − 1 L_\phi^{-1} L ϕ − 1 , the covariance operator A ϕ A_\phi A ϕ , and the covariance function of the process. The biorthogonal system is the one that [60 Table 1] attaches to fractional splines, so that the boundary conditions are the vanishing of the function and of its first ( N 0 − 1 ) (N_0-1) ( N 0 − 1 ) derivatives at the origin.
Operator, Boundary Conditions, and Admissible Range
The relevant whitening operator is the fractional derivative L = D γ : S ( R ) → S ′ ( R ) L = D^\gamma: \mathcal{S}(\mathbb{R}) \to\mathcal{S}'(\mathbb{R}) L = D γ : S ( R ) → S ′ ( R ) of order γ > 1 2 \gamma> \frac{1}{2} γ > 2 1 , whose Fourier symbol is L ^ ( ω ) = ( i ω ) γ \widehat{L}(\omega) = (i\omega)^\gamma L ( ω ) = ( iω ) γ . We also set
H = γ − 1 2 , (76) H = \gamma- \frac{1}{2}, \tag*{(76)} H = γ − 2 1 , (76)
which is the Hurst-type exponent of the resulting process. The biorthogonal system of [60 Table 1] is
p n ( x ) = x n − 1 ( n − 1 ) ! , ϕ n = δ ( n − 1 ) , n = 1 , … , N 0 , (77) p_n(x) = \frac{x^{n-1}}{(n-1)!}, \qquad\phi_n = \delta^{(n-1)}, \qquad n = 1,\ldots,N_0, \tag*{(77)} p n ( x ) = ( n − 1 )! x n − 1 , ϕ n = δ ( n − 1 ) , n = 1 , … , N 0 , (77)
where we adopt the convention ⟨ δ ( n − 1 ) , f ⟩ = f ( n − 1 ) ( 0 ) \langle\delta^{(n-1)}, f\rangle= f^{(n-1)}(0) ⟨ δ ( n − 1 ) , f ⟩ = f ( n − 1 ) ( 0 ) , which is the one that makes ( p , ϕ ) (p,\phi) ( p , ϕ ) biorthogonal with ⟨ p m , ϕ n ⟩ = δ m , n \langle p_m,\phi_n\rangle= \delta_{m,n} ⟨ p m , ϕ n ⟩ = δ m , n . The null space N p = \mathcal{N}_p = N p = span { 1 , x , … , x N 0 − 1 } \operatorname{span}\{1,x,\ldots,x^{N_0-1}\} span { 1 , x , … , x N 0 − 1 } is the space of polynomials of degree less than N 0 N_0 N 0 , while the boundary conditions that specify the conditioned inverse read
ϕ ( g ) = 0 ⟺ g ( 0 ) = g ′ ( 0 ) = ⋯ = g ( N 0 − 1 ) ( 0 ) = 0. (78) \phi(g)=0 \Longleftrightarrow g(0)=g'(0)=\cdots=g^{(N_0-1)}(0)=0. \tag*{(78)} ϕ ( g ) = 0 ⟺ g ( 0 ) = g ′ ( 0 ) = ⋯ = g ( N 0 − 1 ) ( 0 ) = 0. (78)
These are the standard homogeneous initial conditions for the function and its derivatives, and they reduce to g ( 0 ) = 0 g(0)=0 g ( 0 ) = 0 for N 0 = 1 N_0=1 N 0 = 1 , which is Mandelbrot’s normalization of fBm.
Remark A.1 (Admissible range). The pair ( γ , N 0 ) (\gamma,N_0) ( γ , N 0 ) cannot be chosen independently. The quadratic form (84) below is finite for every φ ∈ S ( R ) \varphi\in\mathcal{S}(\mathbb{R}) φ ∈ S ( R ) if and only if
N 0 − 1 2 < γ < N 0 + 1 2 , (79) N_0-\frac{1}{2}<\gamma<N_0+\frac{1}{2}, \tag*{(79)} N 0 − 2 1 < γ < N 0 + 2 1 , (79)
which identifies N 0 = ⌊ γ + 1 2 ⌋ N_0=\lfloor\gamma+\frac{1}{2}\rfloor N 0 = ⌊ γ + 2 1 ⌋ as the integer that is nearest to γ \gamma γ , in agreement with the general rule N B = Π k γ N_{\mathrm{B}}=\Pi_{k_\gamma} N B = Π k γ with k γ = ⌊ γ − d 2 ⌋ k_\gamma=\lfloor\gamma-\frac{d}{2}\rfloor k γ = ⌊ γ − 2 d ⌋ of [59 ] and with the entry for D α D^\alpha D α of [60 Table 1] . The lower bound is what neutralizes the singularity of ∣ ω ∣ − 2 γ |\omega|^{-2\gamma} ∣ ω ∣ − 2 γ at the origin, while the upper bound is what keeps the correction term integrable at infinity; equivalently, the upper bound is the condition under which the monomial x N 0 − 1 x^{N_0-1} x N 0 − 1 belongs to the attainable null space of D γ ∗ D γ D^{\gamma*}D^\gamma D γ ∗ D γ . The two boundary cases γ ∈ N + 1 2 \gamma\in\mathbb{N}+\frac{1}{2} γ ∈ N + 2 1 are excluded, and they are precisely the ones for which the constant c γ c_\gamma c γ of (87) below becomes singular and the Green’s function acquires a logarithmic factor. For γ = m ∈ N \gamma=m\in\mathbb{N} γ = m ∈ N , (78) yields N 0 = m N_0=m N 0 = m , which is the classical polynomial-spline setting; for the classical range H ∈ ( 0 , 1 ) H\in(0,1) H ∈ ( 0 , 1 ) of fBm, it yields N 0 = 1 N_0=1 N 0 = 1 for every H H H .
The Conditioned Inverse and its Fourier Characterization
The Green’s operator of D γ D^\gamma D γ is the Riemann–Liouville fractional integral D − γ : w ↦ ρ L ∗ w D^{-\gamma}:w\mapsto\rho_L*w D − γ : w ↦ ρ L ∗ w with ρ L ( x ) = x + γ − 1 Γ ( γ ) \rho_L(x)=\frac{x_+^{\gamma-1}}{\Gamma(\gamma)} ρ L ( x ) = Γ ( γ ) x + γ − 1 [60 Table 1] , while the conditioning is carried by the projector P ϕ ∗ = Id − R p R ϕ P_\phi^*=\operatorname{Id}-R_pR_\phi P ϕ ∗ = Id − R p R ϕ of § 2.2 , whose action on a sufficiently smooth function is the subtraction of its Taylor polynomial at the origin:
P ϕ ∗ { f } ( x ) = f ( x ) − ∑ n = 1 N 0 x n − 1 ( n − 1 ) ! f ( n − 1 ) ( 0 ) . (80) P_\phi^*\{f\}(x)=f(x)-\sum_{n=1}^{N_0}\frac{x^{n-1}}{(n-1)!}f^{(n-1)}(0). \tag*{(80)} P ϕ ∗ { f } ( x ) = f ( x ) − n = 1 ∑ N 0 ( n − 1 )! x n − 1 f ( n − 1 ) ( 0 ) . (80)
The conditioned inverse L ϕ − 1 = P ϕ ∗ D − γ L_{\phi}^{-1}=P_{\phi}^{*}D^{-\gamma} L ϕ − 1 = P ϕ ∗ D − γ is therefore an integral operator with the (non-convolutional) kernel
ρ γ , ϕ ( t , τ ) = ( t − τ ) + γ − 1 Γ ( γ ) − ∑ n = 1 N 0 t n − 1 ( n − 1 ) ! ( − τ ) + γ − n Γ ( γ − n + 1 ) , (81) \rho_{\gamma,\phi}(t,\tau)=\frac{(t-\tau)_{+}^{\gamma-1}}{\Gamma(\gamma)}-\sum_{n=1}^{N_{0}}\frac{t^{n-1}}{(n-1)!}\frac{(-\tau)_{+}^{\gamma-n}}{\Gamma(\gamma-n+1)}, \tag*{(81)} ρ γ , ϕ ( t , τ ) = Γ ( γ ) ( t − τ ) + γ − 1 − n = 1 ∑ N 0 ( n − 1 )! t n − 1 Γ ( γ − n + 1 ) ( − τ ) + γ − n , (81)
which is obtained by applying (80) to t ↦ ρ L ( t − τ ) t\mapsto\rho_{L}(t-\tau) t ↦ ρ L ( t − τ ) , and which is the instance of the general formula (34) of Theorem 4.2 that corresponds to q n ( τ ) = ( − τ ) + γ − n Γ ( γ − n + 1 ) q_{n}(\tau)=\frac{(-\tau)_{+}^{\gamma-n}}{\Gamma(\gamma-n+1)} q n ( τ ) = Γ ( γ − n + 1 ) ( − τ ) + γ − n . For N 0 = 1 N_{0}=1 N 0 = 1 , (81) reduces to 1 Γ ( γ ) ( ( t − τ ) + γ − 1 − ( − τ ) + γ − 1 ) \frac{1}{\Gamma(\gamma)}\left((t-\tau)_{+}^{\gamma-1}-(-\tau)_{+}^{\gamma-1}\right) Γ ( γ ) 1 ( ( t − τ ) + γ − 1 − ( − τ ) + γ − 1 ) , which is the integral kernel of Mandelbrot’s representation (27) . The higher-order kernels are its natural extension, with one correction term per boundary condition.
The conditioning breaks the shift-invariance, so that L ϕ − 1 L_{\phi}^{-1} L ϕ − 1 has no frequency response of its own. Its adjoint, however, has a transparent Fourier description. Let ϕ ^ = F { ϕ } \widehat{\phi}=\mathcal{F}\{\phi\} ϕ = F { ϕ } and let
ϕ ^ N 0 ( ω ) = ∑ n = 1 N 0 ϕ ^ ( n − 1 ) ( 0 ) ( n − 1 ) ! ω n − 1 (82) \widehat{\phi}_{N_{0}}(\omega)=\sum_{n=1}^{N_{0}}\frac{\widehat{\phi}^{(n-1)}(0)}{(n-1)!}\omega^{n-1} \tag*{(82)} ϕ N 0 ( ω ) = n = 1 ∑ N 0 ( n − 1 )! ϕ ( n − 1 ) ( 0 ) ω n − 1 (82)
be the Taylor polynomial of ϕ ^ \widehat{\phi} ϕ of degree ( N 0 − 1 ) (N_{0}-1) ( N 0 − 1 ) at the origin. Since F { φ n } ( ω ) = ( − i ω ) n − 1 \mathcal{F}\{\varphi_{n}\}(\omega)=(-i\omega)^{n-1} F { φ n } ( ω ) = ( − iω ) n − 1 and ⟨ p n , ϕ ⟩ = i n − 1 ( n − 1 ) ! ϕ ^ ( n − 1 ) ( 0 ) \langle p_{n},\phi\rangle=\frac{i^{n-1}}{(n-1)!}\widehat{\phi}^{(n-1)}(0) ⟨ p n , ϕ ⟩ = ( n − 1 )! i n − 1 ϕ ( n − 1 ) ( 0 ) , the projector P ϕ = Id − R ϕ R p P_{\phi}=\operatorname{Id}-R_{\phi}R_{p} P ϕ = Id − R ϕ R p acts in the Fourier domain as
F { P ϕ ϕ } ( ω ) = ϕ ^ ( ω ) − ϕ ^ N 0 ( ω ) , (83) \mathcal{F}\{P_{\phi}\phi\}(\omega)=\widehat{\phi}(\omega)-\widehat{\phi}_{N_{0}}(\omega), \tag*{(83)} F { P ϕ ϕ } ( ω ) = ϕ ( ω ) − ϕ N 0 ( ω ) , (83)
which is the exact Fourier-domain counterpart of (80) . Together with L ϕ − 1 ∗ = D − γ ∗ P ϕ L_{\phi}^{-1*}=D^{-\gamma*}P_{\phi} L ϕ − 1 ∗ = D − γ ∗ P ϕ , this yields the announced characterization
F { L ϕ − 1 ∗ ϕ } ( ω ) = ϕ ^ ( ω ) − ϕ ^ N 0 ( ω ) ( − i ω ) γ , ( − i ω ) γ = ( i ω ) γ . (84) \mathcal{F}\{L_{\phi}^{-1*}\phi\}(\omega)=\frac{\widehat{\phi}(\omega)-\widehat{\phi}_{N_{0}}(\omega)}{(-i\omega)^{\gamma}},\qquad(-i\omega)^{\gamma}=(i\omega)^{\gamma}. \tag*{(84)} F { L ϕ − 1 ∗ ϕ } ( ω ) = ( − iω ) γ ϕ ( ω ) − ϕ N 0 ( ω ) , ( − iω ) γ = ( iω ) γ . (84)
The interpretation is that the conditioning removes the first N 0 N_{0} N 0 Taylor coefficients of ϕ ^ \widehat{\phi} ϕ at the origin, which is exactly what is needed for the division by ( i ω ) γ (i\omega)^{\gamma} ( iω ) γ to remain well-defined. In signal-processing terms, the operator P ϕ P_{\phi} P ϕ endows the test function with N 0 N_{0} N 0 vanishing moments.
The Covariance Operator
By combining (84) with the factorization A ϕ = L ϕ − 1 L ϕ − 1 ∗ A_{\phi}=L_{\phi}^{-1}L_{\phi}^{-1*} A ϕ = L ϕ − 1 L ϕ − 1 ∗ (see (8) ), we obtain the Fourier-domain description of the covariance operator as the sesquilinear form
⟨ A ϕ φ , ψ ⟩ = 1 2 π ∫ R ( φ ^ ( ω ) − φ ^ N 0 ( ω ) ) ( ψ ^ ( ω ) − ψ ^ N 0 ( ω ) ) ∣ ω ∣ 2 γ d ω (85) \langle A_{\phi}\varphi,\psi\rangle=\frac{1}{2\pi}\int_{\mathbb{R}}\frac{\left(\widehat{\varphi}(\omega)-\widehat{\varphi}_{N_0}(\omega)\right)\left(\widehat{\psi}(\omega)-\widehat{\psi}_{N_0}(\omega)\right)}{|\omega|^{2\gamma}}\,\mathrm{d}\omega \tag*{(85)} ⟨ A ϕ φ , ψ ⟩ = 2 π 1 ∫ R ∣ ω ∣ 2 γ ( φ ( ω ) − φ N 0 ( ω ) ) ( ψ ( ω ) − ψ N 0 ( ω ) ) d ω (85)
which is finite for all φ , ψ ∈ S ( R ) \varphi,\psi\in\mathcal{S}(\mathbb{R}) φ , ψ ∈ S ( R ) under the condition (78). The integrand is non-negative for ψ = φ \psi=\varphi ψ = φ , and it vanishes identically exactly on N ϕ = span { δ ( n − 1 ) } n = 1 N 0 \mathcal{N}_{\phi}=\operatorname{span}\{\delta^{(n-1)}\}_{n=1}^{N_0} N ϕ = span { δ ( n − 1 ) } n = 1 N 0 , whose members have the polynomial Fourier transforms F { φ n } ( ω ) = ( − i ω ) n − 1 \mathcal{F}\{\varphi_n\}(\omega)=(-\mathrm{i}\omega)^{n-1} F { φ n } ( ω ) = ( − i ω ) n − 1 : this is the degeneracy of Theorem 3.3 in concrete form. For φ , ψ ∈ H A \varphi,\psi\in\mathcal{H}_A φ , ψ ∈ H A , the corrections disappear, because ⟨ p n , φ ⟩ = 0 \langle p_n,\varphi\rangle=0 ⟨ p n , φ ⟩ = 0 is equivalent to φ ^ ( n − 1 ) ( 0 ) = 0 \widehat{\varphi}^{(n-1)}(0)=0 φ ( n − 1 ) ( 0 ) = 0 , and (84) reduces to ⟨ A φ , ψ ⟩ = 1 2 π ∫ R φ ^ ψ ^ ∣ ω ∣ 2 γ d ω \langle A_{\varphi},\psi\rangle=\frac{1}{2\pi}\int_{\mathbb{R}}\frac{\widehat{\varphi}\widehat{\psi}}{|\omega|^{2\gamma}}\,\mathrm{d}\omega ⟨ A φ , ψ ⟩ = 2 π 1 ∫ R ∣ ω ∣ 2 γ φ ψ d ω , which is the concrete form of (21).
The Covariance Function
The process B γ = L φ − 1 { W } ∼ N ( 0 , C L , ϕ ) B_{\gamma}=L_{\varphi}^{-1}\{W\}\sim\mathcal{N}(0,C_{L,\phi}) B γ = L φ − 1 { W } ∼ N ( 0 , C L , ϕ ) is the solution of the fractional stochastic differential equation
D γ { B γ } = W subject to B γ ( n − 1 ) ( 0 ) = 0 , n = 1 , … , N 0 , (86) D^{\gamma}\{B_{\gamma}\}=W\qquad\text{subject to}\qquad B_{\gamma}^{(n-1)}(0)=0,\quad n=1,\ldots,N_0, \tag*{(86)} D γ { B γ } = W subject to B γ ( n − 1 ) ( 0 ) = 0 , n = 1 , … , N 0 , (86)
which is the concrete instance of (25). For γ > 1 2 \gamma>\frac{1}{2} γ > 2 1 , the native space H D γ \mathcal{H}_{D^{\gamma}} H D γ is a reproducing-kernel Hilbert space of continuous functions, with δ t ∈ H D γ ′ \delta_t\in\mathcal{H}_{D^{\gamma}}' δ t ∈ H D γ ′ for every t ∈ R t\in\mathbb{R} t ∈ R ; by Item 3 of Theorem 4.2, this is equivalent to the finiteness of the diagonal value a ϕ ( t , t ) = σ γ 2 ∣ t ∣ 2 H a_{\phi}(t,t)=\sigma_{\gamma}^{2}|t|^{2H} a ϕ ( t , t ) = σ γ 2 ∣ t ∣ 2 H computed in (91) below. The covariance function
r γ ( t , τ ) = E [ B γ ( t ) B γ ( τ ) ] = ⟨ A ϕ { δ τ } , δ t ⟩ (87) r_{\gamma}(t,\tau)=\mathbb{E}\left[B_{\gamma}(t)B_{\gamma}(\tau)\right]=\langle A_{\phi}\{\delta_{\tau}\},\delta_t\rangle \tag*{(87)} r γ ( t , τ ) = E [ B γ ( t ) B γ ( τ ) ] = ⟨ A ϕ { δ τ } , δ t ⟩ (87)
is well-defined. Let
ρ γ ( x ) = c γ ∣ x ∣ 2 γ − 1 , c γ = − 1 2 sin ( π 2 ( 2 γ − 1 ) ) Γ ( 2 γ ) = 1 2 cos ( π γ ) Γ ( 2 γ ) , (88) \rho_{\gamma}(x)=c_{\gamma}|x|^{2\gamma-1},\qquad c_{\gamma}=\frac{-1}{2\sin\left(\frac{\pi}{2}(2\gamma-1)\right)\Gamma(2\gamma)}=\frac{1}{2\cos(\pi\gamma)\Gamma(2\gamma)}, \tag*{(88)} ρ γ ( x ) = c γ ∣ x ∣ 2 γ − 1 , c γ = 2 sin ( 2 π ( 2 γ − 1 ) ) Γ ( 2 γ ) − 1 = 2 cos ( π γ ) Γ ( 2 γ ) 1 , (88)
be the Green’s function of D γ ∗ D γ D^{\gamma*}D^{\gamma} D γ ∗ D γ of [60 Table 1] , whose derivatives are
ρ γ ( k ) ( x ) = c γ Γ ( 2 γ ) Γ ( 2 γ − k ) ∣ x ∣ 2 γ − 1 − k ( sgn x ) k . \rho_{\gamma}^{(k)}(x)=c_{\gamma}\frac{\Gamma(2\gamma)}{\Gamma(2\gamma-k)}|x|^{2\gamma-1-k}(\operatorname{sgn}x)^k. ρ γ ( k ) ( x ) = c γ Γ ( 2 γ − k ) Γ ( 2 γ ) ∣ x ∣ 2 γ − 1 − k ( sgn x ) k .
Proposition A.2 (Covariance of the higher-order fBm). Under the condition (78), the covariance function of the process (85) is
r γ ( t , τ ) = ρ γ ( t − τ ) − ∑ n = 1 N 0 ( − t ) n − 1 ( n − 1 ) ! ρ γ ( n − 1 ) ( τ ) − ∑ n = 1 N 0 ( − τ ) n − 1 ( n − 1 ) ! ρ γ ( n − 1 ) ( t ) . (89) r_{\gamma}(t,\tau)=\rho_{\gamma}(t-\tau)-\sum_{n=1}^{N_0}\frac{(-t)^{n-1}}{(n-1)!}\rho_{\gamma}^{(n-1)}(\tau)-\sum_{n=1}^{N_0}\frac{(-\tau)^{n-1}}{(n-1)!}\rho_{\gamma}^{(n-1)}(t). \tag*{(89)} r γ ( t , τ ) = ρ γ ( t − τ ) − n = 1 ∑ N 0 ( n − 1 )! ( − t ) n − 1 ρ γ ( n − 1 ) ( τ ) − n = 1 ∑ N 0 ( n − 1 )! ( − τ ) n − 1 ρ γ ( n − 1 ) ( t ) . (89)
It is the double Taylor remainder of ρ γ ( t − τ ) \rho_\gamma(t-\tau) ρ γ ( t − τ ) , in the sense that it is obtained by subtracting from ρ γ ( t − τ ) \rho_\gamma(t-\tau) ρ γ ( t − τ ) its Taylor polynomials of degree ( N 0 − 1 ) (N_0-1) ( N 0 − 1 ) at t = 0 t=0 t = 0 and at τ = 0 \tau=0 τ = 0 .
Proof. This is the general formula (35) of Theorem 4.2 specialized to L = D γ L=D^\gamma L = D γ , whose double sum vanishes here. We give the direct computation. The identity A ϕ = P ϕ ∗ A P ϕ A_\phi=P_\phi^*AP_\phi A ϕ = P ϕ ∗ A P ϕ of (6) gives r γ ( t , τ ) = ⟨ A P ϕ { δ τ } , P ϕ { δ t } ⟩ r_\gamma(t,\tau)=\langle A P_\phi\{\delta_\tau\},P_\phi\{\delta_t\}\rangle r γ ( t , τ ) = ⟨ A P ϕ { δ τ } , P ϕ { δ t }⟩ with P ϕ { δ t } = δ t − ∑ n = 1 N 0 p n ( t ) ϕ n P_\phi\{\delta_t\}=\delta_t-\sum_{n=1}^{N_0}p_n(t)\phi_n P ϕ { δ t } = δ t − ∑ n = 1 N 0 p n ( t ) ϕ n . The expansion of the bilinear form produces four groups of terms, which we evaluate by means of ⟨ A δ τ , δ t ⟩ = ρ γ ( t − τ ) \langle A\delta_\tau,\delta_t\rangle=\rho_\gamma(t-\tau) ⟨ A δ τ , δ t ⟩ = ρ γ ( t − τ ) , ⟨ A δ τ , ϕ n ⟩ = ρ γ ( n − 1 ) ( − τ ) \langle A\delta_\tau,\phi_n\rangle=\rho_\gamma^{(n-1)}(-\tau) ⟨ A δ τ , ϕ n ⟩ = ρ γ ( n − 1 ) ( − τ ) , ⟨ A ϕ m , δ t ⟩ = ( − 1 ) m − 1 ρ γ ( m − 1 ) ( t ) \langle A\phi_m,\delta_t\rangle=(-1)^{m-1}\rho_\gamma^{(m-1)}(t) ⟨ A ϕ m , δ t ⟩ = ( − 1 ) m − 1 ρ γ ( m − 1 ) ( t ) , and ⟨ A ϕ m , ϕ n ⟩ = ( − 1 ) m − 1 ρ γ ( m + n − 2 ) ( 0 ) \langle A\phi_m,\phi_n\rangle=(-1)^{m-1}\rho_\gamma^{(m+n-2)}(0) ⟨ A ϕ m , ϕ n ⟩ = ( − 1 ) m − 1 ρ γ ( m + n − 2 ) ( 0 ) . The last group vanishes, because ρ γ \rho_\gamma ρ γ is ( m + n − 2 ) (m+n-2) ( m + n − 2 ) times differentiable at the origin with a vanishing derivative there whenever m + n < 2 γ + 1 m+n<2\gamma+1 m + n < 2 γ + 1 , a condition that (79) guarantees for all m , n ≤ N 0 m,n\le N_0 m , n ≤ N 0 . In the notation of Theorem 4.2 , this is the statement that the Gram coefficients (37) all vanish, so that r m , n = 0 r_{m,n}=0 r m , n = 0 . The evenness of ρ γ \rho_\gamma ρ γ gives ρ γ ( n − 1 ) ( − τ ) = ( − 1 ) n − 1 ρ γ ( n − 1 ) ( τ ) \rho_\gamma^{(n-1)}(-\tau)=(-1)^{n-1}\rho_\gamma^{(n-1)}(\tau) ρ γ ( n − 1 ) ( − τ ) = ( − 1 ) n − 1 ρ γ ( n − 1 ) ( τ ) , which turns the two middle groups into the two sums of (89) .
One verifies directly on (89) that ∂ t n − 1 r γ ( t , τ ) ∣ t = 0 = 0 \left.\partial_t^{n-1}r_\gamma(t,\tau)\right|_{t=0}=0 ∂ t n − 1 r γ ( t , τ ) t = 0 = 0 for n = 1 , … , N 0 n=1,\ldots,N_0 n = 1 , … , N 0 , which is the covariance transcription of the boundary conditions (78) .
For N 0 = 1 N_0=1 N 0 = 1 , that is, for γ ∈ ( 1 2 , 3 2 ) \gamma\in\left(\frac{1}{2},\frac{3}{2}\right) γ ∈ ( 2 1 , 2 3 ) and H ∈ ( 0 , 1 ) H\in(0,1) H ∈ ( 0 , 1 ) , formula (89) reduces to the classical covariance of fBm,
r γ ( t , τ ) = σ H 2 2 ( ∣ t ∣ 2 H + ∣ τ ∣ 2 H − ∣ t − τ ∣ 2 H ) , σ H 2 = − 2 c γ = 1 sin ( π H ) Γ ( 2 H + 1 ) (90) r_\gamma(t,\tau)=\frac{\sigma_H^2}{2}\left(|t|^{2H}+|\tau|^{2H}-|t-\tau|^{2H}\right),\qquad\sigma_H^2=-2c_\gamma=\frac{1}{\sin(\pi H)\Gamma(2H+1)} \tag*{(90)} r γ ( t , τ ) = 2 σ H 2 ( ∣ t ∣ 2 H + ∣ τ ∣ 2 H − ∣ t − τ ∣ 2 H ) , σ H 2 = − 2 c γ = sin ( π H ) Γ ( 2 H + 1 ) 1 (90)
which also fixes the normalization constant that is implicit in (29) : it is the one that corresponds to the choice ρ L ( x ) = x + γ − 1 Γ ( γ ) \rho_L(x)=\frac{x_+^{\gamma-1}}{\Gamma(\gamma)} ρ L ( x ) = Γ ( γ ) x + γ − 1 of [60 Table 1] . For H = 1 2 H=\frac{1}{2} H = 2 1 , one gets σ H 2 = 1 \sigma_H^2=1 σ H 2 = 1 and r 1 ( t , τ ) = min ( t , τ ) r_1(t,\tau)=\min(t,\tau) r 1 ( t , τ ) = min ( t , τ ) for t , τ ≥ 0 t,\tau\ge0 t , τ ≥ 0 , which is Brownian motion. For N 0 = 2 N_0=2 N 0 = 2 , that is, for γ ∈ ( 3 2 , 5 2 ) \gamma\in\left(\frac{3}{2},\frac{5}{2}\right) γ ∈ ( 2 3 , 2 5 ) and H ∈ ( 1 , 2 ) H\in(1,2) H ∈ ( 1 , 2 ) , we get
r γ ( t , τ ) = c γ ( ∣ t − τ ∣ 2 H − ∣ t ∣ 2 H − ∣ τ ∣ 2 H + 2 H ( t ∣ τ ∣ 2 H − 1 sgn ( τ ) + τ ∣ t ∣ 2 H − 1 sgn ( t ) ) ) (91) r_\gamma(t,\tau)=c_\gamma\left(|t-\tau|^{2H}-|t|^{2H}-|\tau|^{2H}+2H\left(t|\tau|^{2H-1}\operatorname{sgn}(\tau)+\tau|t|^{2H-1}\operatorname{sgn}(t)\right)\right) \tag*{(91)} r γ ( t , τ ) = c γ ( ∣ t − τ ∣ 2 H − ∣ t ∣ 2 H − ∣ τ ∣ 2 H + 2 H ( t ∣ τ ∣ 2 H − 1 sgn ( τ ) + τ ∣ t ∣ 2 H − 1 sgn ( t ) ) ) (91)
where the two additional terms are the ones that enforce B γ ′ ( 0 ) = 0 B'_\gamma(0)=0 B γ ′ ( 0 ) = 0 .
Whatever the order, the process remains self-similar of exponent H H H : the substitution τ = t \tau= t τ = t in (89) gives
E { ∣ B γ ( t ) ∣ 2 } = σ γ 2 ∣ t ∣ 2 H , σ γ 2 = − 2 c γ Γ ( 2 γ ) ∑ n = 1 N 0 ( − 1 ) n − 1 ( n − 1 ) ! Γ ( 2 γ − n + 1 ) , (92) \mathbb{E}\left\{ \left|B_{\gamma}(t)\right|^2 \right\} = \sigma_{\gamma}^2 |t|^{2H}, \qquad\sigma_{\gamma}^2 = -2c_{\gamma}\Gamma(2\gamma)\sum_{n=1}^{N_0}\frac{(-1)^{n-1}}{(n-1)!\Gamma(2\gamma-n+1)}, \tag*{(92)} E { ∣ B γ ( t ) ∣ 2 } = σ γ 2 ∣ t ∣ 2 H , σ γ 2 = − 2 c γ Γ ( 2 γ ) n = 1 ∑ N 0 ( n − 1 )! Γ ( 2 γ − n + 1 ) ( − 1 ) n − 1 , (92)
which is consistent with σ γ 2 = σ H 2 \sigma_{\gamma}^2 = \sigma_H^2 σ γ 2 = σ H 2 for N 0 = 1 N_0 = 1 N 0 = 1 . Formula (89) coincides with the covariance that Perrin et al. obtained for their n n n th-order extension of fractional Brownian motion [43 ] , with n = N 0 n = N_0 n = N 0 and with the same normalization: the vanishing of the process and of its first ( N 0 − 1 ) (N_0 - 1) ( N 0 − 1 ) derivatives at the origin. The transfer principle of [51 ] relates these processes to the classical fBm.
The Associated Splines
Theorem 5.3 now applies verbatim. Given the noisy samples y m = B γ ( t m ) + ϵ m y_m = B_{\gamma}(t_m) + \epsilon_m y m = B γ ( t m ) + ϵ m at M M M distinct locations t 1 , … , t M t_1,\ldots,t_M t 1 , … , t M , the MMSE estimator of B γ B_{\gamma} B γ is the constrained fractional spline
g ~ λ = ∑ m = 1 M a m r γ ( ⋅ , t m ) , a = ( G ~ + λ I ) − 1 y , λ = σ 2 , (93) \widetilde{g}_{\lambda} = \sum_{m=1}^{M} a_m r_{\gamma}(\cdot,t_m), \qquad\mathbf{a} = (\widetilde{\mathbf{G}}+\lambda\mathbf{I})^{-1}\mathbf{y}, \qquad\lambda= \sigma^2, \tag*{(93)} g λ = m = 1 ∑ M a m r γ ( ⋅ , t m ) , a = ( G + λ I ) − 1 y , λ = σ 2 , (93)
with [ G ~ ] m , n = r γ ( t m , t n ) [\widetilde{\mathbf{G}}]_{m,n} = r_{\gamma}(t_m,t_n) [ G ] m , n = r γ ( t m , t n ) . The basis functions are the covariance function itself, in accordance with the congruence of Proposition 3.5 . For γ = m ∈ N \gamma= m \in\mathbb{N} γ = m ∈ N , one has N 0 = m N_0 = m N 0 = m , c γ = ( − 1 ) m 2 ( 2 m − 1 ) ! c_{\gamma} = \frac{(-1)^m}{2(2m-1)!} c γ = 2 ( 2 m − 1 )! ( − 1 ) m , and (93) is the polynomial smoothing spline of degree ( 2 m − 1 ) (2m-1) ( 2 m − 1 ) that satisfies the boundary conditions (78) ; for non-integer γ \gamma γ , it is the fractional spline of [58 , 61 ] . This gives a precise meaning, and a proof, to the statement that fractional splines are the optimal estimators of fractional Brownian motion [6 , 56 ] .
The interpolation counterpart (Corollary 5.5 ) is more demanding, since its hypothesis N φ ⊂ N ν \mathcal{N}_{\varphi} \subset\mathcal{N}_{\nu} N φ ⊂ N ν requires the boundary functionals to be reproducible from the observations. For N 0 = 1 N_0 = 1 N 0 = 1 , it suffices to include the sample at t = 0 t = 0 t = 0 , whose value B γ ( 0 ) = 0 B_{\gamma}(0) = 0 B γ ( 0 ) = 0 is deterministic. For N 0 ≥ 2 N_0 \ge2 N 0 ≥ 2 , the derivatives δ ( n − 1 ) \delta^{(n-1)} δ ( n − 1 ) cannot be reproduced by finitely many point evaluations, so that the samples must be complemented by the derivative measurements B γ ( n − 1 ) ( 0 ) = 0 B_{\gamma}^{(n-1)}(0) = 0 B γ ( n − 1 ) ( 0 ) = 0 for n = 2 , … , N 0 n = 2,\ldots,N_0 n = 2 , … , N 0 . Once this is done, every fractional spline interpolant of the augmented data is an MMSE interpolator.
Acknowledgment
The author used Claude (Anthropic) as an interactive assistant during the preparation of this paper, for proofreading, for checking and simplifying proofs, and for the numerical verification of the formulas of the appendix and Section 4.3. All results were verified by the author, who takes full responsibility for the content. This research was supported by the Swiss National Science Foundation (SNSF) under Grant 200020_219356.