keywords: 41A25; 46E22; 65N15; 65N35
Introduction
Partial differential equations (PDEs) constitute a cornerstone in the mathematical modeling of physical and engineering systems. In recent years, the advent of deep learning techniques has facilitated the emergence of novel methodologies for solving PDEs [1 , 2 , 3 , 4 , 5 , 6 ] . The notable efficacy of these methods stems from their mesh-free nature and their capacity to alleviate the curse of dimensionality. However, their application is constrained by two primary bottlenecks: limited accuracy and high computational expense. These limitations fundamentally originate from the nonconvex optimization landscape encountered during neural network training. Several approaches address this difficulty by fixing the inner-layer parameters [7 , 8 , 9 ] . For linear PDEs, optimizing the output-layer coefficients then reduces to a linear least-squares problem, thereby avoiding the complexities associated with nonconvex optimization. A prominent instance is the random feature method (RFM) [8 ] , which uses a random feature model [10 ] to represent the approximate solution. This model is a two-layer neural network whose hidden-layer parameters are randomly sampled and then held fixed. Specifically, the corresponding function class is
Σ m ϕ : = { u α : u α ( x ) = 1 m ∑ i = 1 m α i ϕ ( x , θ i ) , α ∈ R m } \Sigma_m^\phi:= \left\{u_\alpha: u_\alpha(x) = \frac{1}{\sqrt{m}}\sum_{i=1}^{m}\alpha_i\phi(x,\theta_i),\ \alpha\in\mathbb{R}^{m}\right\} Σ m ϕ := { u α : u α ( x ) = m 1 i = 1 ∑ m α i ϕ ( x , θ i ) , α ∈ R m }
where θ 1 , … , θ m \theta_1,\ldots,\theta_m θ 1 , … , θ m are the sampled feature parameters and α \alpha α contains the trainable output coefficients. RFM has exhibited exceptional performance in diverse numerical experiments spanning structural analysis, interface problems, and time-dependent problems [8 , 11 , 12 , 13 ] . Notably, in many cases, the achieved errors decay at an exponential or near-exponential rate as the number of random features or collocation points increases. Despite these empirical successes, a comprehensive theoretical explanation for such convergence rates remains incomplete. Closing this theoretical gap is essential both to substantiate the empirical efficacy of RFM and to extend its applicability to a wider range of challenging problems.
Throughout this paper, we consider linear boundary value problems of the form
{ L u ∗ = f ∗ in Ω , B u ∗ = g ∗ on ∂ Ω . \begin{cases}
\mathcal{L}u^* = f^* & \text{in }\Omega,\\
\mathcal{B}u^* = g^* & \text{on }\partial\Omega.
\end{cases} { L u ∗ = f ∗ B u ∗ = g ∗ in Ω , on ∂ Ω.
where L \mathcal{L} L is a linear differential operator, B \mathcal{B} B is a linear boundary operator, and u ∗ u^* u ∗ is the target solution. Our analysis of RFM distinguishes two sources of error. The approximation error is the distance from u ∗ u^* u ∗ to the random feature model. The estimation error arises from recovering u ∗ u^* u ∗ from finitely many noiseless observations of f ∗ f^* f ∗ and g ∗ g^* g ∗ . Since random features were introduced as a computational surrogate for kernel methods [10 ] , we analyze both problems through the induced kernels and their associated integral operators. For finite-feature approximation, we construct kernels on parameter space from the Sobolev inner products of the features and from the weighted inner products of their images under the differential and boundary operators. For each error metric, our duality result reformulates the regularized finite-feature approximation problem as noiseless KRR on the parameter space, where the sampled parameters serve as observation sites for the transformed target. Through this correspondence, the noiseless recovery lower bound and the KRR upper bound in [14 ] yield the lower and upper bounds for the worst-case approximation error in each metric. For finite collocation estimation, we construct a spatial kernel for which the rescaled interior and boundary observations are point values of the transformed target. The collocation lower bound then follows from the noiseless recovery bound in [14 Theorem 4.1] . For the collocation upper bound, we adapt the concentration argument in the proof of the upper bound in [14 ] to account for the separate sampling of interior and boundary points. We control each empirical operator under its own sampling distribution and combine the resulting interior and boundary operator deviation bounds to estimate the population residual of the regularized estimator over the full RKHS. Although finite-feature approximation admits a direct analysis, we use duality to study it within the same noiseless KRR framework as recovery from collocation observations. Our approach based on duality yields the same best approximation error as the direct approach (see Remark 5). Both the approximation and collocation bounds depend on the eigenvalues of the corresponding integral operators. To make the bounds explicit, we relate the decay of the associated eigenvalues to the regularity of the activation function. We use Volkov’s singular-value estimates [15 ] together with our ambient regularity lemmas to derive upper bounds for the eigenvalues governing Sobolev approximation and PDE collocation.
Our main results are summarized below, with precise statements and assumptions given in § 4 .
Approximation in Sobolev norms and the residual seminorm. We establish upper and lower bounds for the worst-case approximation error over the unit ball of H k \mathcal{H}_k H k . Under the respective assumptions of Theorems 1 and 2 for Sobolev approximation and Theorems 3 and 4 for residual approximation, the bounds take the form
μ ~ m + 1 ( r ) ≤ sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ Φ ~ m ∥ u − u ∗ ∥ H r ( Ω ) 2 ≲ μ ~ l m ( r ) \widetilde{\mu}_{m+1}^{(r)} \le\sup_{\lVert u^*\rVert_{\mathcal{H}_k}\le1}\inf_{u\in\widetilde{\Phi}_m}\lVert u-u^*\rVert_{\mathcal{H}_r(\Omega)}^2 \lesssim\widetilde{\mu}_{l_m}^{(r)} μ m + 1 ( r ) ≤ ∥ u ∗ ∥ H k ≤ 1 sup u ∈ Φ m inf ∥ u − u ∗ ∥ H r ( Ω ) 2 ≲ μ l m ( r )
μ ~ m + 1 ( R η ) ≤ sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ Φ ~ m ∥ u − u ∗ ∥ R η 2 ≲ μ ~ l m ( R η ) . \widetilde{\mu}_{m+1}^{(\mathcal{R}_{\eta})} \le\sup_{\lVert u^*\rVert_{\mathcal{H}_k}\le1}\inf_{u\in\widetilde{\Phi}_m}\lVert u-u^*\rVert_{\mathcal{R}_{\eta}}^2 \lesssim\widetilde{\mu}_{l_m}^{(\mathcal{R}_{\eta})}. μ m + 1 ( R η ) ≤ ∥ u ∗ ∥ H k ≤ 1 sup u ∈ Φ m inf ∥ u − u ∗ ∥ R η 2 ≲ μ l m ( R η ) .
For each error measure, the upper bound holds with probability at least 1 − δ 1-\delta 1 − δ under its corresponding feature sampling distribution, whereas the lower bounds hold for every fixed admissible choice of feature parameters. Here, μ ~ j ( r ) \widetilde{\mu}_j^{(r)} μ j ( r ) and μ ~ j ( R η ) \widetilde{\mu}_j^{(\mathcal{R}_{\eta})} μ j ( R η ) denote the j j j -th largest positive eigenvalues of the integral operators defined in (41) and (51) , respectively. The index l m l_m l m is specified in the respective upper-bound theorem with l m ≳ log ( m / δ ) l_m \gtrsim\log(m/\delta) l m ≳ log ( m / δ ) .
u ^ n , η , λ : = argmin u ∈ H k { R ^ n , η ( u ) + λ ∥ u ∥ H k 2 } , λ > 0 , \hat{u}_{n,\eta,\lambda}:=\underset{u\in\mathcal{H}_k}{\operatorname{argmin}}\left\{\widehat{\mathcal{R}}_{n,\eta}(u)+\lambda\|u\|_{\mathcal{H}_k}^{2}\right\},\qquad\lambda>0, u ^ n , η , λ := u ∈ H k argmin { R n , η ( u ) + λ ∥ u ∥ H k 2 } , λ > 0 ,
where R ^ n , η \widehat{\mathcal{R}}_{n,\eta} R n , η is the weighted empirical PDE residual defined in (10) , constructed from n n n noiseless interior and boundary observations. Under the sampling and regularization conditions of Theorem 5, the estimator u ^ n , η , λ \hat{u}_{n,\eta,\lambda} u ^ n , η , λ satisfies, with probability at least 1 − δ 1-\delta 1 − δ ,
sup ∥ u ∗ ∥ H k ≤ 1 ∥ u ^ n , η , λ − u ∗ ∥ R η 2 ≲ μ l n P D E , l n ≲ n log ( n / δ ) . \sup_{\|u^*\|_{\mathcal{H}_k}\leq1}\|\hat{u}_{n,\eta,\lambda}-u^*\|_{\mathcal{R}_\eta}^{2}\lesssim\mu_{l_n}^{\mathrm{PDE}},\qquad l_n\lesssim\frac{n}{\log(n/\delta)}. ∥ u ∗ ∥ H k ≤ 1 sup ∥ u ^ n , η , λ − u ∗ ∥ R η 2 ≲ μ l n PDE , l n ≲ log ( n / δ ) n .
Here μ j P D E \mu_j^{\mathrm{PDE}} μ j PDE denotes the j j j -th largest positive eigenvalue of the integral operator defined in (57) . The index l n l_n l n is specified in the theorem. For every fixed admissible collection of collocation observations, we also establish
inf G ∈ M n P D E A sup ∥ u ∗ ∥ H k ≤ 1 ∥ G ( S n ( u ∗ ) ) − u ∗ ∥ R η 2 ≳ μ n + 1 P D E . \inf_{G\in\mathcal{M}_{n}^{\mathrm{PDE}}\atop{\vphantom{\Large A}}}\ \sup_{\|u^*\|_{\mathcal{H}_k}\leq1}\|G(S_n(u^*))-u^*\|_{\mathcal{R}_\eta}^{2}\gtrsim\mu_{n+1}^{\mathrm{PDE}}. A G ∈ M n PDE inf ∥ u ∗ ∥ H k ≤ 1 sup ∥ G ( S n ( u ∗ )) − u ∗ ∥ R η 2 ≳ μ n + 1 PDE .
Here, S n ( u ∗ ) S_n(u^*) S n ( u ∗ ) denotes the observations, and M n P D E \mathcal{M}_{n}^{\mathrm{PDE}} M n PDE is the class of reconstruction procedures based only on these observations.
A lower bound under joint feature and observation restrictions. We consider reconstruction from n n n collocation observations with outputs restricted to a random feature model spanned by m m m features. Let E m , n E_{m,n} E m , n denote the smallest worst-case squared residual error over the unit ball of H k \mathcal{H}_k H k among all such reconstruction procedures. For fixed positive proportions of interior and boundary observations, we establish
E m , n ≳ max { μ m + 1 P D E , μ n + 1 P D E } . E_{m,n}\gtrsim\max\{\mu_{m+1}^{\mathrm{PDE}},\mu_{n+1}^{\mathrm{PDE}}\}. E m , n ≳ max { μ m + 1 PDE , μ n + 1 PDE } .
for every fixed admissible choice of feature parameters and collocation points, with an implicit constant independent of m m m and n n n . This bound identifies the separate limitations imposed by the feature space and the available observations and applies, in particular, to the RFM estimator.
Dependence of eigenvalue decay on activation regularity. We derive eigenvalue upper bounds for features of the form ϕ ( x , θ ) = σ ( k ⊤ x + b ) \phi(x,\theta)=\sigma(k^{\top}x+b) ϕ ( x , θ ) = σ ( k ⊤ x + b ) . For the PDE eigenvalues, we specialize to second-order uniformly elliptic equations with Dirichlet boundary conditions. Under the finite-regularity assumptions of Theorem 8 , with p ≥ max { 2 , r } p\geq\max\{2,r\} p ≥ max { 2 , r } , we obtain
μ ~ j ( r ) = o ( j − 1 − 2 ( p − r ) d ) , μ j P D E = o ( j − 1 − 2 ( p − 2 ) d ) . \tilde{\mu}_{j}^{(r)}=o\left(j^{-1-\frac{2(p-r)}{d}}\right),\qquad\mu_{j}^{\mathrm{PDE}}=o\left(j^{-1-\frac{2(p-2)}{d}}\right). μ ~ j ( r ) = o ( j − 1 − d 2 ( p − r ) ) , μ j PDE = o ( j − 1 − d 2 ( p − 2 ) ) .
Under the analytic assumptions of the same theorem, there exists c > 0 c>0 c > 0 such that
μ ~ j ( r ) ≲ e − c j 1 / d , μ j P D E ≲ e − c j 1 / d . \tilde{\mu}_{j}^{(r)}\lesssim e^{-cj^{1/d}},\qquad\mu_{j}^{\mathrm{PDE}}\lesssim e^{-cj^{1/d}}. μ ~ j ( r ) ≲ e − c j 1/ d , μ j PDE ≲ e − c j 1/ d .
Related Work
This section reviews related work on random-feature approximation, spectral analysis of kernel methods, and kernel collocation for PDEs.
Random-feature approximation. Early universal approximation results cover networks with randomly generated hidden nodes [16 ] . For uniformly bounded features, a fixed target admits a high-probability L 2 L^2 L 2 approximation bound of order O ( m − 1 / 2 ) O(m^{-1/2}) O ( m − 1/2 ) if its integral representation has a coefficient function whose ratio to the sampling density is bounded [17 ] . For ReLUk ^k k features with well-distributed fixed parameters on S d S^d S d , the H r H^r H r approximation error is of order O ( m − 1 / 2 − ( 2 ( k − r ) + 1 ) / ( 2 d ) ) O(m^{-1/2-(2(k-r)+1)/(2d)}) O ( m − 1/2 − ( 2 ( k − r ) + 1 ) / ( 2 d ) ) for 0 ≤ r ≤ k 0 \le r \le k 0 ≤ r ≤ k and targets in H ( d + 2 k + 1 ) / 2 ( Ω ) H^{(d+2k+1)/2}(\Omega) H ( d + 2 k + 1 ) /2 ( Ω ) on a bounded Lipschitz domain Ω \Omega Ω [18 Theorem 2.2] . For parameters sampled independently and uniformly on S d S^d S d , the corresponding high-probability H r H^r H r bound incurs a logarithmic loss in the number of features [18 Theorem 6.2] . [19 ] studies best L p L^p L p approximation on an interval by random projection networks with fixed hidden parameters and smooth nonpolynomial activations. [20 ] analyze RFM for one-dimensional second-order elliptic problems and derive error estimates that depend on the regularity of the solution. In multiple dimensions, they establish high-probability approximation bounds for trigonometric features [21 ] . Our analysis starts from a general feature map and its induced RKHS. Under the stated spectral and sampling assumptions, we derive upper and lower bounds for the worst-case approximation error over the RKHS unit ball, both in Sobolev norms and in the PDE residual seminorm. For each error metric, the bounds are expressed in terms of the eigenvalues of the corresponding operator on the parameter space. This formulation applies without prescribing a particular activation function. We then use the regularity of the activation function to obtain explicit upper bounds on the eigenvalues.
Spectral analysis of kernel methods. For random-feature approximation and kernel quadrature, the eigenvalues of the associated integral operator yield upper and lower bounds on the number of samples needed to attain a prescribed accuracy. The upper bounds can be attained by sampling from a suitable nonuniform distribution [22 ] . A duality framework connects random-feature approximation and estimation [23 ] . In our setting, the sampled feature parameters serve as observation points in a noiseless KRR problem on parameter space. This formulation allows us to bound the finite-feature approximation error using KRR estimates that depend on eigenvalue decay and target regularity [14 ] . Sobolev and analytic regularity of an integral kernel also yield decay estimates for the singular values of the corresponding operator [15 ] . We draw on these results to analyze approximation in Sobolev norms and the PDE residual seminorm, recovery from interior and boundary observations, and the dependence of eigenvalue bounds on activation regularity.
Kernel collocation for PDEs. Radial basis function collocation is a classical kernel method for solving PDEs [24 , 25 ] . Native-space interpolation theory relates approximation errors to fill distance and kernel smoothness [26 ] . For second-order linear elliptic equations with Dirichlet boundary conditions, constrained and weighted least-squares kernel collocation methods admit Sobolev error bounds under suitable smoothness and point-density conditions [27 ] . For nonlinear and parametric PDEs, error analyses of kernel and Gaussian process methods use RKHS embeddings, PDE stability, and Sobolev interpolation estimates [28 ] . Our analysis complements these results by deriving upper and lower bounds in terms of operator eigenvalues for two distinct problems: approximation by m m m features and recovery over the full RKHS from n n n collocation observations. When reconstruction is subject to both restrictions, the worst-case squared residual error is at least the larger of the two lower bounds.
Notations
We use N + \mathbb{N}^{+} N + to denote the set of positive integers and set N = { 0 } ∪ N + \mathbb{N}=\{0\}\cup\mathbb{N}^{+} N = { 0 } ∪ N + . We shall use ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ to denote an operator norm without specifying the input and output spaces for brevity. For any Banach spaces X \mathcal{X} X and Y \mathcal{Y} Y , let B ( X ) \mathcal{B}(\mathcal{X}) B ( X ) denote the bounded linear operators on X \mathcal{X} X and F ( X ; Y ) \mathcal{F}(\mathcal{X};\mathcal{Y}) F ( X ; Y ) denote the functions { f : X → Y } \{f:\mathcal{X}\to\mathcal{Y}\} { f : X → Y } . Let P ( X ) \mathcal{P}(\mathcal{X}) P ( X ) denote the set of probability measures over X \mathcal{X} X . For any ρ ∈ P ( X ) \rho\in\mathcal{P}(\mathcal{X}) ρ ∈ P ( X ) , we denote by L p ( X , ρ ) L^{p}(\mathcal{X},\rho) L p ( X , ρ ) the classical L p L^{p} L p space, and often write L p ( ρ ) L^{p}(\rho) L p ( ρ ) when the underlying space X \mathcal{X} X is clear from context. For any f , g ∈ L 2 ( ρ ) f,g\in L^{2}(\rho) f , g ∈ L 2 ( ρ ) , we write ⟨ f , g ⟩ ρ = ∫ X f ( x ) g ( x ) d ρ ( x ) \langle f,g\rangle_{\rho}=\int_{\mathcal{X}}f(x)g(x)\,\mathrm{d}\rho(x) ⟨ f , g ⟩ ρ = ∫ X f ( x ) g ( x ) d ρ ( x ) for the inner product, and use ∥ ⋅ ∥ ρ \|\cdot\|_{\rho} ∥ ⋅ ∥ ρ to denote the associated norm. For a bounded domain Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d and a nonnegative integer r r r , let H r ( Ω ) = W r , 2 ( Ω ) H^{r}(\Omega)=W^{r,2}(\Omega) H r ( Ω ) = W r , 2 ( Ω ) denote the Sobolev space with respect to Lebesgue measure, equipped with the inner product ⟨ f , g ⟩ H r ( Ω ) : = ∑ ∣ β ∣ ≤ r ∫ Ω D β f ( x ) D β g ( x ) d x \langle f,g\rangle_{H^{r}(\Omega)}:=\sum_{|\beta|\le r}\int_{\Omega}D^{\beta}f(x)D^{\beta}g(x)\,\mathrm{d}x ⟨ f , g ⟩ H r ( Ω ) := ∑ ∣ β ∣ ≤ r ∫ Ω D β f ( x ) D β g ( x ) d x , where β \beta β is a multi-index and D β D^{\beta} D β denotes the corresponding weak derivative. We denote the induced norm by ∥ ⋅ ∥ H r ( Ω ) \|\cdot\|_{H^{r}(\Omega)} ∥ ⋅ ∥ H r ( Ω ) . In particular, H 0 ( Ω ) = L 2 ( Ω ) H^{0}(\Omega)=L^{2}(\Omega) H 0 ( Ω ) = L 2 ( Ω ) . Let H 1 , H 2 \mathcal{H}_{1},\mathcal{H}_{2} H 1 , H 2 be two Hilbert spaces, and v ∈ H 1 , u ∈ H 2 v\in\mathcal{H}_{1},u\in\mathcal{H}_{2} v ∈ H 1 , u ∈ H 2 . The operator u ⊗ v : H 1 ↦ H 2 u\otimes v:\mathcal{H}_{1}\mapsto\mathcal{H}_{2} u ⊗ v : H 1 ↦ H 2 is defined as ( u ⊗ v ) x = ⟨ v , x ⟩ H 1 u (u\otimes v)x=\langle v,x\rangle_{\mathcal{H}_{1}}u ( u ⊗ v ) x = ⟨ v , x ⟩ H 1 u for x ∈ H 1 x\in\mathcal{H}_{1} x ∈ H 1 . We use I I I to denote a general identity operator without specifying the domain for brevity and let I d I_{d} I d be the d × d d\times d d × d identity matrix. We use big-O notations like O ( ⋅ ) O(\cdot) O ( ⋅ ) and o ( ⋅ ) o(\cdot) o ( ⋅ ) to hide constants and O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) to further hide logarithmic factors. We also adopt the notation a ≲ b a\lesssim b a ≲ b to indicate a = O ( b ) a=O(b) a = O ( b ) and write a ≍ b a\asymp b a ≍ b when both a ≲ b a\lesssim b a ≲ b and b ≲ a b\lesssim a b ≲ a hold.
The paper is organized as follows. Section § 2 provides the necessary preliminaries for our analysis. Section § 3 introduces the problem setting. Our main results are presented in Section § 4 , with the corresponding analysis provided in Section § 5 . Conclusions follow in Section § 6 . Auxiliary results for our proofs are provided in Appendix § 7 and the proofs of activation-dependent eigenvalue bounds are deferred to Appendix B.
Preliminaries
In this section, we recall some basic definitions and results concerning reproducing kernel Hilbert spaces, kernel ridge regression, and degrees of freedom, which will be used in the subsequent sections.
Kernel and RKHS
Let X \mathcal{X} X be a measurable input space, and ρ ∈ P ( X ) \rho\in\mathcal{P}(\mathcal{X}) ρ ∈ P ( X ) denote the reference probability measure. A function k : X × X → R k:\mathcal{X}\times\mathcal{X}\to\mathbb{R} k : X × X → R is said to be a (positive semidefinite) kernel if 1) it is symmetric, i.e. k ( x , x ′ ) = k ( x ′ , x ) , ∀ x , x ′ ∈ X k(x,x')=k(x',x),\forall x,x'\in\mathcal{X} k ( x , x ′ ) = k ( x ′ , x ) , ∀ x , x ′ ∈ X ; and 2) ∀ n ≥ 1 \forall n\ge1 ∀ n ≥ 1 , x 1 , … , x n ∈ X x_{1},\ldots,x_{n}\in\mathcal{X} x 1 , … , x n ∈ X , the kernel matrix ( k ( x i , x j ) ) i , j ∈ R n × n (k(x_{i},x_{j}))_{i,j}\in\mathbb{R}^{n\times n} ( k ( x i , x j ) ) i , j ∈ R n × n is positive semidefinite. Given a kernel k k k , there exists a unique Hilbert space H \mathcal{H} H such that k ( x , ⋅ ) ∈ H , ∀ x ∈ X k(x,\cdot)\in\mathcal{H},\forall x\in\mathcal{X} k ( x , ⋅ ) ∈ H , ∀ x ∈ X ; and ⟨ f , k ( x , ⋅ ) ⟩ H = f ( x ) , ∀ f ∈ H , x ∈ X \langle f,k(x,\cdot)\rangle_{\mathcal{H}}=f(x),\forall f\in\mathcal{H},x\in\mathcal{X} ⟨ f , k ( x , ⋅ ) ⟩ H = f ( x ) , ∀ f ∈ H , x ∈ X . The second property is known as the reproducing property, and thus H \mathcal{H} H is often referred to as a reproducing kernel Hilbert space (RKHS) [29 ] . The RKHS can also be characterized using the associated integral operator T : L 2 ( X , ρ ) ↦ L 2 ( X , ρ ) \mathcal{T}:L^{2}(\mathcal{X},\rho)\mapsto L^{2}(\mathcal{X},\rho) T : L 2 ( X , ρ ) ↦ L 2 ( X , ρ ) , given by
T f = ∫ X k ( ⋅ , x ′ ) f ( x ′ ) d ρ ( x ′ ) . (1) \mathcal{T}f=\int_{\mathcal{X}}k(\cdot,x')f(x')\,\mathrm{d}\rho(x'). \tag*{(1)} T f = ∫ X k ( ⋅ , x ′ ) f ( x ′ ) d ρ ( x ′ ) . (1)
Throughout this paper, we assume ∫ k ( x , x ) d ρ ( x ) < ∞ \int k(x,x)\,\mathrm{d}\rho(x)<\infty ∫ k ( x , x ) d ρ ( x ) < ∞ , which guarantees that T \mathcal{T} T is a self-adjoint, positive, and compact operator; see [30 Theorem 4.27] . By the spectral theory of compact operators, we have the following decomposition:
T = ∑ j = 1 ∞ μ j e j ⊗ e j , (2) \mathcal{T}=\sum_{j=1}^{\infty}\mu_j e_j\otimes e_j, \tag*{(2)} T = j = 1 ∑ ∞ μ j e j ⊗ e j , (2)
where { μ j } j = 1 ∞ \{\mu_j\}_{j=1}^{\infty} { μ j } j = 1 ∞ are the eigenvalues in a decreasing order and { e j } j = 1 ∞ \{e_j\}_{j=1}^{\infty} { e j } j = 1 ∞ are the corresponding orthonormal eigenfunctions. Without loss of generality, we assume that { e j } j = 1 ∞ \{e_j\}_{j=1}^{\infty} { e j } j = 1 ∞ forms a complete basis of L 2 ( X , ρ ) L^{2}(\mathcal{X},\rho) L 2 ( X , ρ ) . Otherwise we can always focus on the subspace spanned by the eigenfunctions associated with nonzero eigenvalues. We impose the following assumption on the eigenvalue sequences considered in this paper.
Assumption 1 Each eigenvalue sequence { μ j } j ≥ 1 \{\mu_j\}_{j\geq1} { μ j } j ≥ 1 considered in this paper is arranged in nonincreasing order and satisfies one of the following conditions:
Algebraic decay:
μ j ≍ j − t for some t > 1. \mu_j\asymp j^{-t}\qquad\text{for some }t>1. μ j ≍ j − t for some t > 1. Stretched-exponential decay:
μ j ≲ τ − j 1 / d for some τ > 1 and d ∈ N + , \mu_j\lesssim\tau^{-j^{1/d}}\qquad\text{for some }\tau>1\text{ and }d\in\mathbb{N}_{+}, μ j ≲ τ − j 1/ d for some τ > 1 and d ∈ N + , where d d d is the spatial dimension, which will be specified in each setting.
We focus on these two decay regimes, for which Lemma 20 provides explicit estimates of the degree of freedom. Other eigenvalue decay regimes may be analyzed using similar arguments.
We further extend H \mathcal{H} H to the following interpolation space with arbitrary s ∈ R s\in\mathbb{R} s ∈ R .
Definition 1 (Interpolation space) For any s ∈ R s\in\mathbb{R} s ∈ R and f ∈ L 2 ( X , ρ ) f\in L^{2}(\mathcal{X},\rho) f ∈ L 2 ( X , ρ ) , we define ∥ f ∥ H s = ∥ T − s 2 f ∥ ρ \|f\|_{\mathcal{H}^{s}}=\|\mathcal{T}^{-\frac{s}{2}}f\|_{\rho} ∥ f ∥ H s = ∥ T − 2 s f ∥ ρ and the interpolation space H s = { f : ∥ f ∥ H s < ∞ } \mathcal{H}^{s}=\{f:\|f\|_{\mathcal{H}^{s}}<\infty\} H s = { f : ∥ f ∥ H s < ∞ } .
By the spectral decomposition (2) , H s \mathcal{H}^{s} H s can be written in the following form
H s = { ∑ j = 1 ∞ a j μ j s / 2 e j : ∑ j = 1 ∞ a j 2 < ∞ } , ∥ ∑ j = 1 ∞ a j μ j s / 2 e j ∥ H s 2 = ∑ j = 1 ∞ a j 2 . (3) \mathcal{H}^{s}=\left\{\sum_{j=1}^{\infty}a_j\mu_j^{s/2}e_j:\sum_{j=1}^{\infty}a_j^2<\infty\right\},\qquad\left\|\sum_{j=1}^{\infty}a_j\mu_j^{s/2}e_j\right\|_{\mathcal{H}^{s}}^2=\sum_{j=1}^{\infty}a_j^2. \tag*{(3)} H s = { j = 1 ∑ ∞ a j μ j s /2 e j : j = 1 ∑ ∞ a j 2 < ∞ } , j = 1 ∑ ∞ a j μ j s /2 e j H s 2 = j = 1 ∑ ∞ a j 2 . (3)
It is obvious that H 1 = H \mathcal{H}^{1}=\mathcal{H} H 1 = H , H 0 = L 2 ( X , ρ ) \mathcal{H}^{0}=L^{2}(\mathcal{X},\rho) H 0 = L 2 ( X , ρ ) , and H s \mathcal{H}^{s} H s provides a natural interpolation between them, and beyond (i.e. s > 1 s>1 s > 1 or s < 0 s<0 s < 0 ). For more discussion, we refer to [14 Section 2] .
Weighted KRR
Suppose that x 1 , … , x n x_{1},\ldots,x_{n} x 1 , … , x n are i.i.d. samples drawn from a sampling distribution ρ ′ \rho^{\prime} ρ ′ , possibly different from the reference distribution ρ \rho ρ . We impose the following assumption on every pair of reference and sampling measures used throughout the paper.
Assumption 2 Let ν \nu ν and ν ′ \nu^{\prime} ν ′ be probability measures on the same measurable space. The sampling measure ν ′ \nu^{\prime} ν ′ is absolutely continuous with respect to the reference measure ν \nu ν . Moreover, its Radon–Nikodym derivative r = d ν ′ d ν r=\frac{\mathrm{d}\nu^{\prime}}{\mathrm{d}\nu} r = d ν d ν ′ satisfies r > 0 r>0 r > 0 ν \nu ν -almost everywhere.
We apply Assumption 2 to ( ρ , ρ ′ ) (\rho,\rho^{\prime}) ( ρ , ρ ′ ) on X \mathcal{X} X , with q = d ρ ′ d ρ q=\frac{\mathrm{d}\rho^{\prime}}{\mathrm{d}\rho} q = d ρ d ρ ′ . For a target function f ∗ ∈ H f^{*}\in\mathcal{H} f ∗ ∈ H , we consider the following KRR:
f ^ n , λ = argmin f ∈ H [ 1 n ∑ i = 1 n 1 q ( x i ) ( f ( x i ) − f ∗ ( x i ) ) 2 + λ ∥ f ∥ H 2 ] . \widehat{f}_{n,\lambda}=\underset{f\in\mathcal{H}}{\operatorname{argmin}}\left[\frac{1}{n}\sum_{i=1}^{n}\frac{1}{q(x_{i})}\left(f(x_{i})-f^{*}(x_{i})\right)^{2}+\lambda\lVert f\rVert_{\mathcal{H}}^{2}\right]. f n , λ = f ∈ H argmin [ n 1 i = 1 ∑ n q ( x i ) 1 ( f ( x i ) − f ∗ ( x i ) ) 2 + λ ∥ f ∥ H 2 ] .
The weighted empirical risk is an unbiased estimate of the true risk and standard KRR is recovered by setting q ( x ) ≡ 1 q(x)\equiv1 q ( x ) ≡ 1 . By the representer theorem [31 ] , f ^ n , λ \widehat{f}_{n,\lambda} f n , λ must lie in span { k ( x 1 , ⋅ ) , … , k ( x n , ⋅ ) } \operatorname{span}\{k(x_{1},\cdot),\ldots,k(x_{n},\cdot)\} span { k ( x 1 , ⋅ ) , … , k ( x n , ⋅ )} . Define
k ^ n ( x ) = ( k ( x i , x ) n q ( x i ) ) i = 1 n , y ^ = ( f ∗ ( x i ) n q ( x i ) ) i = 1 n , K n = ( k ( x i , x j ) n q ( x i ) q ( x j ) ) i , j = 1 n . \widehat{k}_{n}(x)=\left(\frac{k(x_{i},x)}{\sqrt{nq(x_{i})}}\right)_{i=1}^{n},\qquad\widehat{y}=\left(\frac{f^{*}(x_{i})}{\sqrt{nq(x_{i})}}\right)_{i=1}^{n},\qquad K_{n}=\left(\frac{k(x_{i},x_{j})}{n\sqrt{q(x_{i})q(x_{j})}}\right)_{i,j=1}^{n}. k n ( x ) = ( n q ( x i ) k ( x i , x ) ) i = 1 n , y = ( n q ( x i ) f ∗ ( x i ) ) i = 1 n , K n = ( n q ( x i ) q ( x j ) k ( x i , x j ) ) i , j = 1 n .
Then the KRR solution can be expressed as: f ^ n , λ = k ^ n ( ⋅ ) T ( K n + λ I n ) − 1 y ^ \widehat{f}_{n,\lambda}=\widehat{k}_{n}(\cdot)^{\mathsf{T}}(K_{n}+\lambda I_{n})^{-1}\widehat{y} f n , λ = k n ( ⋅ ) T ( K n + λ I n ) − 1 y .
Degrees of Freedom
To study the convergence of learning within an RKHS, the commonly used notion of complexity is the degrees of freedom (DoFs) (see, e.g., [32 , 33 , 34 , 35 ] ). To facilitate our analysis, we specialize [14 Definition 3.1] to the case where the DoF exponent γ = 1 \gamma=1 γ = 1 .
Definition 2 For λ > 0 \lambda>0 λ > 0 and a sampling density q q q with respect to ρ \rho ρ , satisfying q > 0 q>0 q > 0 ρ \rho ρ -almost everywhere, define
N ( λ ) : = tr [ ( T + λ I ) − 1 T ] , F q ( λ ) : = ess sup x ∈ X ∥ ( T + λ I ) − 1 / 2 k ( x , ⋅ ) ∥ H 2 q ( x ) . N(\lambda):=\operatorname{tr}\left[(\mathcal{T}+\lambda I)^{-1}\mathcal{T}\right],\qquad F_{q}(\lambda):=\operatorname*{ess\,sup}_{x\in\mathcal{X}}\frac{\left\lVert(\mathcal{T}+\lambda I)^{-1/2}k(x,\cdot)\right\rVert_{\mathcal{H}}^{2}}{q(x)}. N ( λ ) := tr [ ( T + λ I ) − 1 T ] , F q ( λ ) := x ∈ X ess sup q ( x ) ( T + λ I ) − 1/2 k ( x , ⋅ ) H 2 .
where the essential supremum is taken with respect to ρ \rho ρ .
By the spectral decomposition (2) ,
N ( λ ) = ∑ j = 1 ∞ μ j μ j + λ , F q ( λ ) = e s s s u p x ∈ X 1 q ( x ) ∑ j = 1 ∞ μ j μ j + λ ∣ e j ( x ) ∣ 2 . (4) N(\lambda)=\sum_{j=1}^{\infty}\frac{\mu_j}{\mu_j+\lambda},\qquad F_q(\lambda)=\mathop{\mathrm{ess\,sup}}_{x\in\mathcal{X}}\frac{1}{q(x)}\sum_{j=1}^{\infty}\frac{\mu_j}{\mu_j+\lambda}|e_j(x)|^2. \tag*{(4)} N ( λ ) = j = 1 ∑ ∞ μ j + λ μ j , F q ( λ ) = ess sup x ∈ X q ( x ) 1 j = 1 ∑ ∞ μ j + λ μ j ∣ e j ( x ) ∣ 2 . (4)
We immediately have
F q ( λ ) ≥ ∫ X 1 q ( x ) ∑ j = 1 ∞ μ j μ j + λ ∣ e j ( x ) ∣ 2 d ρ ′ ( x ) = ∑ j = 1 ∞ μ j μ j + λ = N ( λ ) . F_q(\lambda)\ge\int_{\mathcal{X}}\frac{1}{q(x)}\sum_{j=1}^{\infty}\frac{\mu_j}{\mu_j+\lambda}|e_j(x)|^2\,\mathrm{d}\rho'(x)=\sum_{j=1}^{\infty}\frac{\mu_j}{\mu_j+\lambda}=N(\lambda). F q ( λ ) ≥ ∫ X q ( x ) 1 j = 1 ∑ ∞ μ j + λ μ j ∣ e j ( x ) ∣ 2 d ρ ′ ( x ) = j = 1 ∑ ∞ μ j + λ μ j = N ( λ ) .
One convenient choice for controlling F q ( λ ) F_q(\lambda) F q ( λ ) in terms of N ( λ ) N(\lambda) N ( λ ) is the equal-weight mixture of the reference density 1 and the leverage-score density defined in [14 Lemma 3.5] with DoF exponent γ = 1 \gamma=1 γ = 1 :
q ∗ ( k , ρ , λ ; x ) : = 1 2 + 1 2 N ( λ ) ∑ j = 1 ∞ μ j μ j + λ ∣ e j ( x ) ∣ 2 . (5) q^*(k,\rho,\lambda;x):=\frac{1}{2}+\frac{1}{2N(\lambda)}\sum_{j=1}^{\infty}\frac{\mu_j}{\mu_j+\lambda}|e_j(x)|^2. \tag*{(5)} q ∗ ( k , ρ , λ ; x ) := 2 1 + 2 N ( λ ) 1 j = 1 ∑ ∞ μ j + λ μ j ∣ e j ( x ) ∣ 2 . (5)
Then q ∗ ( k , ρ , λ ; ⋅ ) q^*(k,\rho,\lambda;\cdot) q ∗ ( k , ρ , λ ; ⋅ ) is a probability density with respect to ρ \rho ρ and satisfies q ∗ ( k , ρ , λ ; x ) ≥ 1 / 2 q^*(k,\rho,\lambda;x)\ge1/2 q ∗ ( k , ρ , λ ; x ) ≥ 1/2 for ρ \rho ρ -almost every x x x . Moreover,
F q ∗ ( k , ρ , λ ; ⋅ ) ( λ ) = 2 N ( λ ) e s s s u p x ∈ X ( 1 − 1 2 q ∗ ( k , ρ , λ ; x ) ) ≤ 2 N ( λ ) . (6) F_{q^*(k,\rho,\lambda;\cdot)}(\lambda)=2N(\lambda)\mathop{\mathrm{ess\,sup}}_{x\in\mathcal{X}}\left(1-\frac{1}{2q^*(k,\rho,\lambda;x)}\right)\le2N(\lambda). \tag*{(6)} F q ∗ ( k , ρ , λ ; ⋅ ) ( λ ) = 2 N ( λ ) ess sup x ∈ X ( 1 − 2 q ∗ ( k , ρ , λ ; x ) 1 ) ≤ 2 N ( λ ) . (6)
For further discussion of the DoFs, see [14 Section 3] .
Problem Setting
PDE Problem
Let Ω ⊂ R d \Omega\subset\mathbb{R}^d Ω ⊂ R d be a bounded domain with Lipschitz boundary. We consider the following boundary value problem:
{ L u ( x ) = f ∗ ( x ) , x ∈ Ω , B u ( x ) = g ∗ ( x ) , x ∈ ∂ Ω , (7) \begin{cases}
\mathcal{L}u(x)=f^*(x), & x\in\Omega,\\
\mathcal{B}u(x)=g^*(x), & x\in\partial\Omega, \tag*{(7)}
\end{cases} { L u ( x ) = f ∗ ( x ) , B u ( x ) = g ∗ ( x ) , x ∈ Ω , x ∈ ∂ Ω , (7)
where L \mathcal{L} L and B \mathcal{B} B are both linear differential operators and f ∗ f^* f ∗ and g ∗ g^* g ∗ are the prescribed interior and boundary data, respectively.
Let ρ I ∈ P ( Ω ) \rho_I\in\mathcal{P}(\Omega) ρ I ∈ P ( Ω ) and ρ B ∈ P ( ∂ Ω ) \rho_B\in\mathcal{P}(\partial\Omega) ρ B ∈ P ( ∂ Ω ) be reference probability measures on the interior and the boundary, respectively. Assume that d ρ I = w I d x \mathrm{d}\rho_I=w_I\,\mathrm{d}x d ρ I = w I d x and d ρ B = w B d S \mathrm{d}\rho_B=w_B\,\mathrm{d}S d ρ B = w B d S , where d x \mathrm{d}x d x denotes Lebesgue measure on Ω \Omega Ω , d S \mathrm{d}S d S denotes surface measure on ∂ Ω \partial\Omega ∂ Ω , and both densities are strictly positive almost everywhere with respect to their corresponding measures. For a prescribed penalty parameter η > 0 \eta>0 η > 0 , define the population residual functional by
R η ( u ) = ∥ L u − f ∗ ∥ L 2 ( ρ I ) 2 + η ∥ B u − g ∗ ∥ L 2 ( ρ B ) 2 . (8) R_\eta(u)=\lVert\mathcal{L}u-f^*\rVert_{L^2(\rho_I)}^2+\eta\lVert\mathcal{B}u-g^*\rVert_{L^2(\rho_B)}^2. \tag*{(8)} R η ( u ) = ∥ L u − f ∗ ∥ L 2 ( ρ I ) 2 + η ∥ B u − g ∗ ∥ L 2 ( ρ B ) 2 . (8)
We restrict attention to functions whose images under L \mathcal{L} L and B \mathcal{B} B belong to L 2 ( ρ I ) L^2(\rho_I) L 2 ( ρ I ) and L 2 ( ρ B ) L^2(\rho_B) L 2 ( ρ B ) , respectively. Define
D R : = { f ∈ D ( L ) ∩ D ( B ) : L f ∈ L 2 ( ρ I ) , B f ∈ L 2 ( ρ B ) } . (9) D_R := \left\{ f \in D(\mathcal{L}) \cap D(\mathcal{B}) : \mathcal{L}f \in L^2(\rho_I),\ \mathcal{B}f \in L^2(\rho_B) \right\}. \tag*{(9)} D R := { f ∈ D ( L ) ∩ D ( B ) : L f ∈ L 2 ( ρ I ) , B f ∈ L 2 ( ρ B ) } . (9)
where D ( L ) D(\mathcal{L}) D ( L ) and D ( B ) D(\mathcal{B}) D ( B ) denote the domains of the PDE and boundary operators, respectively. For η > 0 \eta> 0 η > 0 , define the residual seminorm on D R D_R D R by
∥ v ∥ R , η : = ( ∥ L v ∥ L 2 ( ρ I ) 2 + η ∥ B v ∥ L 2 ( ρ B ) 2 ) 1 / 2 , v ∈ D R . \|v\|_{R,\eta} := \left(\|\mathcal{L}v\|_{L^2(\rho_I)}^2 + \eta\|\mathcal{B}v\|_{L^2(\rho_B)}^2\right)^{1/2}, \qquad v \in D_R. ∥ v ∥ R , η := ( ∥ L v ∥ L 2 ( ρ I ) 2 + η ∥ B v ∥ L 2 ( ρ B ) 2 ) 1/2 , v ∈ D R .
This seminorm is a norm if the boundary value problem is well posed in D R D_R D R . Let u ∗ u^* u ∗ be the solution of (7) . Then the population residual can be written as
R η ( u ) = ∥ L ( u − u ∗ ) ∥ L 2 ( ρ I ) 2 + η ∥ B ( u − u ∗ ) ∥ L 2 ( ρ B ) 2 = ∥ u − u ∗ ∥ R , η 2 . R_\eta(u) = \|\mathcal{L}(u-u^*)\|_{L^2(\rho_I)}^2 + \eta\|\mathcal{B}(u-u^*)\|_{L^2(\rho_B)}^2 = \|u-u^*\|_{R,\eta}^2. R η ( u ) = ∥ L ( u − u ∗ ) ∥ L 2 ( ρ I ) 2 + η ∥ B ( u − u ∗ ) ∥ L 2 ( ρ B ) 2 = ∥ u − u ∗ ∥ R , η 2 .
To treat the interior and boundary components in a common space, we introduce the following definition.
Definition 3 Let ( Ω , Σ I , ρ I ) (\Omega,\Sigma_I,\rho_I) ( Ω , Σ I , ρ I ) and ( ∂ Ω , Σ B , ρ B ) (\partial\Omega,\Sigma_B,\rho_B) ( ∂ Ω , Σ B , ρ B ) be the measure spaces associated with the interior and boundary, respectively. Define the σ \sigma σ -algebra on Ω ˉ \bar{\Omega} Ω ˉ by
Σ ˉ : = { A ⊆ Ω ˉ : A ∩ Ω ∈ Σ I , A ∩ ∂ Ω ∈ Σ B } , \bar{\Sigma} := \left\{ A \subseteq\bar{\Omega} : A \cap\Omega\in\Sigma_I,\ A \cap\partial\Omega\in\Sigma_B \right\}, Σ ˉ := { A ⊆ Ω ˉ : A ∩ Ω ∈ Σ I , A ∩ ∂ Ω ∈ Σ B } ,
and define the measure ρ ˉ \bar{\rho} ρ ˉ on Σ ˉ \bar{\Sigma} Σ ˉ by
ρ ˉ ( A ) : = ρ I ( A ∩ Ω ) + ρ B ( A ∩ ∂ Ω ) , A ∈ Σ ˉ . \bar{\rho}(A) := \rho_I(A \cap\Omega) + \rho_B(A \cap\partial\Omega), \qquad A \in\bar{\Sigma}. ρ ˉ ( A ) := ρ I ( A ∩ Ω ) + ρ B ( A ∩ ∂ Ω ) , A ∈ Σ ˉ .
Set
L ˉ 2 ( Ω ˉ ) : = L 2 ( Ω ˉ , Σ ˉ , ρ ˉ ) . \bar{L}^2(\bar{\Omega}) := L^2(\bar{\Omega},\bar{\Sigma},\bar{\rho}). L ˉ 2 ( Ω ˉ ) := L 2 ( Ω ˉ , Σ ˉ , ρ ˉ ) .
Each f ∈ L ˉ 2 ( Ω ˉ ) f \in\bar{L}^2(\bar{\Omega}) f ∈ L ˉ 2 ( Ω ˉ ) is identified with a pair ( f I , f B ) ∈ L 2 ( ρ I ) ⊕ L 2 ( ρ B ) (f_I,f_B) \in L^2(\rho_I) \oplus L^2(\rho_B) ( f I , f B ) ∈ L 2 ( ρ I ) ⊕ L 2 ( ρ B ) through
f ( x ) = { f I ( x ) , x ∈ Ω , f B ( x ) , x ∈ ∂ Ω , f(x) =
\begin{cases}
f_I(x), & x \in\Omega,\\
f_B(x), & x \in\partial\Omega,
\end{cases} f ( x ) = { f I ( x ) , f B ( x ) , x ∈ Ω , x ∈ ∂ Ω ,
with equality understood almost everywhere with respect to the corresponding measures. For f = ( f I , f B ) f=(f_I,f_B) f = ( f I , f B ) and g = ( g I , g B ) g=(g_I,g_B) g = ( g I , g B ) , the inner product is
⟨ f , g ⟩ L ˉ 2 ( Ω ˉ ) : = ⟨ f I , g I ⟩ L 2 ( ρ I ) + ⟨ f B , g B ⟩ L 2 ( ρ B ) . \langle f,g\rangle_{\bar{L}^2(\bar{\Omega})} := \langle f_I,g_I\rangle_{L^2(\rho_I)} + \langle f_B,g_B\rangle_{L^2(\rho_B)}. ⟨ f , g ⟩ L ˉ 2 ( Ω ˉ ) := ⟨ f I , g I ⟩ L 2 ( ρ I ) + ⟨ f B , g B ⟩ L 2 ( ρ B ) .
The interior and boundary components are independent, with no trace compatibility imposed.
Weighted Random Feature Model
Let Θ ⊂ R d ~ \Theta\subset\mathbb{R}^{\tilde{d}} Θ ⊂ R d ~ be a bounded parameter space, let ϕ : Ω ˉ × Θ → R \phi: \bar{\Omega} \times\Theta\to\mathbb{R} ϕ : Ω ˉ × Θ → R be the feature function, and let π ∈ P ( Θ ) \pi\in\mathcal{P}(\Theta) π ∈ P ( Θ ) be a reference probability measure. We draw θ 1 , … , θ m \theta_1,\ldots,\theta_m θ 1 , … , θ m independently from a sampling distribution π ′ ∈ P ( Θ ) \pi' \in\mathcal{P}(\Theta) π ′ ∈ P ( Θ ) . We impose Assumption 2 on the pair ( ν , ν ′ ) = ( π , π ′ ) (\nu,\nu')=(\pi,\pi') ( ν , ν ′ ) = ( π , π ′ ) and write p = d π ′ / d π p = \mathrm{d}\pi'/\mathrm{d}\pi p = d π ′ / d π . The associated weighted random feature model is defined by
Σ m ϕ : = { u α : u α ( x ) = 1 m ∑ i = 1 m α i p ( θ i ) ϕ ( x , θ i ) , α ∈ R m } . \Sigma_m^\phi:= \left\{u_\alpha: u_\alpha(x) = \frac{1}{\sqrt{m}}\sum_{i=1}^{m}\frac{\alpha_i}{\sqrt{p(\theta_i)}}\phi(x,\theta_i),\ \alpha\in\mathbb{R}^m\right\}. Σ m ϕ := { u α : u α ( x ) = m 1 i = 1 ∑ m p ( θ i ) α i ϕ ( x , θ i ) , α ∈ R m } .
The weights p ( θ i ) − 1 / 2 p(\theta_i)^{-1/2} p ( θ i ) − 1/2 account for sampling from π ′ \pi' π ′ rather than π \pi π , ensuring that E [ S m , r S m , r ∗ ] = Φ r Φ r ∗ \mathbb{E}[S_{m,r}S_{m,r}^{*}]=\Phi_r\Phi_r^{*} E [ S m , r S m , r ∗ ] = Φ r Φ r ∗ , where the expectation is over the parameters; see (37) and (39) for the definitions of S m , r S_{m,r} S m , r and Φ r \Phi_r Φ r , respectively.
Collocation-Based Estimator
RFM [8 ] constructs an empirical approximation of the population residual R η \mathcal{R}_{\eta} R η defined in (8) . Let ρ I ∈ P ( Ω ) \rho_I \in\mathcal{P}(\Omega) ρ I ∈ P ( Ω ) and ρ B ∈ P ( ∂ Ω ) \rho_B \in\mathcal{P}(\partial\Omega) ρ B ∈ P ( ∂ Ω ) be the reference probability measures associated with the interior and boundary residuals, respectively. We then independently draw the collocation points according to { x i } i = 1 n I ∼ i . i . d . ρ I ′ \{x_i\}_{i=1}^{n_I} \stackrel{\mathrm{i.i.d.}}{\sim} \rho'_I { x i } i = 1 n I ∼ i.i.d. ρ I ′ and { x j + n I } j = 1 n B ∼ i . i . d . ρ B ′ \{x_{j+n_I}\}_{j=1}^{n_B} \stackrel{\mathrm{i.i.d.}}{\sim} \rho'_B { x j + n I } j = 1 n B ∼ i.i.d. ρ B ′ . Here, ρ I ′ ∈ P ( Ω ) \rho'_I \in\mathcal{P}(\Omega) ρ I ′ ∈ P ( Ω ) and ρ B ′ ∈ P ( ∂ Ω ) \rho'_B \in\mathcal{P}(\partial\Omega) ρ B ′ ∈ P ( ∂ Ω ) are the corresponding sampling measures. We apply Assumption 2 to the pairs ( ρ I , ρ I ′ ) (\rho_I,\rho'_I) ( ρ I , ρ I ′ ) on Ω \Omega Ω and ( ρ B , ρ B ′ ) (\rho_B,\rho'_B) ( ρ B , ρ B ′ ) on ∂ Ω \partial\Omega ∂ Ω , with q I : = d ρ I ′ d ρ I q_I := \frac{\mathrm{d}\rho'_I}{\mathrm{d}\rho_I} q I := d ρ I d ρ I ′ and q B : = d ρ B ′ d ρ B q_B := \frac{\mathrm{d}\rho'_B}{\mathrm{d}\rho_B} q B := d ρ B d ρ B ′ . We denote the total number of collocation points by n = n I + n B n=n_I+n_B n = n I + n B . For simplicity, we consider the global RFM without a partition of unity. We define the weighted empirical PDE residual by
R ^ n , η ( u ) = 1 n I ∑ i = 1 n I ( L u ( x i ) − f ∗ ( x i ) ) 2 q I ( x i ) + η n B ∑ j = 1 n B ( B u ( x j + n I ) − g ∗ ( x j + n I ) ) 2 q B ( x j + n I ) . (10) \widehat{\mathcal{R}}_{n,\eta}(u)=\frac{1}{n_I}\sum_{i=1}^{n_I}\frac{(\mathcal{L}u(x_i)-f^*(x_i))^2}{q_I(x_i)}+\frac{\eta}{n_B}\sum_{j=1}^{n_B}\frac{(\mathcal{B}u(x_{j+n_I})-g^*(x_{j+n_I}))^2}{q_B(x_{j+n_I})}. \tag*{(10)} R n , η ( u ) = n I 1 i = 1 ∑ n I q I ( x i ) ( L u ( x i ) − f ∗ ( x i ) ) 2 + n B η j = 1 ∑ n B q B ( x j + n I ) ( B u ( x j + n I ) − g ∗ ( x j + n I ) ) 2 . (10)
For every fixed u u u , the empirical residual is unbiased: E [ R ^ n , η ( u ) ] = R η ( u ) \mathbb{E}[\widehat{\mathcal{R}}_{n,\eta}(u)]=\mathcal{R}_{\eta}(u) E [ R n , η ( u )] = R η ( u ) , where the expectation is over the collocation points. The approximate solution obtained by RFM is
u ^ n , m , η : = argmin u α ∈ Σ m ∗ R ^ n , η ( u α ) . (11) \widehat{u}_{n,m,\eta}:=\underset{u_\alpha\in\Sigma_m^*}{\operatorname{argmin}}\ \widehat{\mathcal{R}}_{n,\eta}(u_\alpha). \tag*{(11)} u n , m , η := u α ∈ Σ m ∗ argmin R n , η ( u α ) . (11)
RKHS Characterization via the Feature Function
We formulate our analysis in the RKHS characterized by the feature function ϕ \phi ϕ . We impose the following assumption throughout the paper.
Assumption 3 The feature function ϕ \phi ϕ is jointly measurable in ( x , θ ) (x,\theta) ( x , θ ) and satisfies
ϕ ( x , ⋅ ) ∈ L 2 ( π ) \phi(x,\cdot)\in L^2(\pi) ϕ ( x , ⋅ ) ∈ L 2 ( π ) for every x ∈ Ω x\in\Omega x ∈ Ω .
Moreover, ∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ L 2 ( ρ I ) 2 d π ( θ ) < ∞ \int_{\Theta}\|\phi(\cdot,\theta)\|_{L^2(\rho_I)}^2\mathrm{d}\pi(\theta)<\infty ∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ L 2 ( ρ I ) 2 d π ( θ ) < ∞ .
For x , x ′ ∈ Ω x,x'\in\Omega x , x ′ ∈ Ω , define k ( x , x ′ ) : = ⟨ ϕ ( x , ⋅ ) , ϕ ( x ′ , ⋅ ) ⟩ L 2 ( π ) k(x,x'):=\langle\phi(x,\cdot),\phi(x',\cdot)\rangle_{L^2(\pi)} k ( x , x ′ ) := ⟨ ϕ ( x , ⋅ ) , ϕ ( x ′ , ⋅ ) ⟩ L 2 ( π ) , and denote the associated RKHS by H k \mathcal{H}_k H k . Define the feature operator Φ : L 2 ( π ) ↦ L 2 ( ρ I ) \Phi:L^2(\pi)\mapsto L^2(\rho_I) Φ : L 2 ( π ) ↦ L 2 ( ρ I ) by
Φ α = ∫ Θ α ( θ ) ϕ ( ⋅ , θ ) d π ( θ ) . (12) \Phi\alpha=\int_{\Theta}\alpha(\theta)\phi(\cdot,\theta)\,\mathrm{d}\pi(\theta). \tag*{(12)} Φ α = ∫ Θ α ( θ ) ϕ ( ⋅ , θ ) d π ( θ ) . (12)
The feature-space characterization of an RKHS [30 Theorem 4.21] yields
H k = Range ( Φ ) = { Φ α : α ∈ L 2 ( π ) } , ∥ f ∥ H k = min α ∈ L 2 ( π ) Φ α = f ∥ α ∥ L 2 ( π ) . (13) \mathcal{H}_k=\operatorname{Range}(\Phi)=\{\Phi\alpha:\alpha\in L^2(\pi)\},\qquad\|f\|_{\mathcal{H}_k}=\min_{\substack{\alpha\in L^2(\pi)\\\Phi\alpha=f}}\|\alpha\|_{L^2(\pi)}. \tag*{(13)} H k = Range ( Φ ) = { Φ α : α ∈ L 2 ( π )} , ∥ f ∥ H k = α ∈ L 2 ( π ) Φ α = f min ∥ α ∥ L 2 ( π ) . (13)
Although a function may admit multiple representing coefficients, its minimum-norm coefficient is unique and belongs to ( ker Φ ) ⊥ (\ker\Phi)^\perp ( ker Φ ) ⊥ . Throughout this paper, we assume that the exact solution u ∗ u^* u ∗ to (7) belongs to H k \mathcal{H}_k H k . We denote its unique minimum-norm representing coefficient by b ∗ b^* b ∗ . Thus,
u ∗ = Φ b ∗ , b ∗ ∈ ( ker Φ ) ⊥ , ∥ b ∗ ∥ L 2 ( π ) = ∥ u ∗ ∥ H k . u^* = \Phi b^*, \qquad b^* \in(\ker\Phi)^\perp, \qquad\|b^*\|_{L^2(\pi)} = \|u^*\|_{\mathcal{H}_k}. u ∗ = Φ b ∗ , b ∗ ∈ ( ker Φ ) ⊥ , ∥ b ∗ ∥ L 2 ( π ) = ∥ u ∗ ∥ H k .
Additional Feature Assumptions
In addition to Assumption 3, we introduce two conditions for Sobolev approximation and PDE analysis, respectively.
Assumption 4 (Sobolev feature regularity) For the prescribed integer r ≥ 0 r \ge0 r ≥ 0 , the map θ ↦ ϕ ( ⋅ , θ ) \theta\mapsto\phi(\cdot,\theta) θ ↦ ϕ ( ⋅ , θ ) is strongly measurable with values in H r ( Ω ) H^r(\Omega) H r ( Ω ) and satisfies
∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ H r ( Ω ) 2 d π ( θ ) < ∞ . \int_\Theta\|\phi(\cdot,\theta)\|_{H^r(\Omega)}^2 \,\mathrm{d}\pi(\theta) < \infty. ∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ H r ( Ω ) 2 d π ( θ ) < ∞.
Under Assumption 4, for every a ∈ L 2 ( π ) a \in L^2(\pi) a ∈ L 2 ( π ) , the integral defining Φ a \Phi a Φ a in (12) is well defined as an H r ( Ω ) H^r(\Omega) H r ( Ω ) -valued Bochner integral. Moreover, Φ \Phi Φ , viewed as an operator from L 2 ( π ) L^2(\pi) L 2 ( π ) to H r ( Ω ) H^r(\Omega) H r ( Ω ) , is Hilbert–Schmidt. It is used for the Sobolev approximation results with the specified order r r r .
For the PDE analysis, we impose the following compatibility condition to justify interchanging L \mathcal{L} L and B \mathcal{B} B with the feature integral. Specifically, for every b ∈ L 2 ( π ) b \in L^2(\pi) b ∈ L 2 ( π ) , it ensures that
L ( Φ b ) = ∫ Θ L ϕ ( ⋅ , θ ) b ( θ ) d π ( θ ) in L 2 ( ρ I ) , \mathcal{L}(\Phi b) = \int_\Theta\mathcal{L}\phi(\cdot,\theta)b(\theta)\,\mathrm{d}\pi(\theta) \quad\text{in } L^2(\rho_I), L ( Φ b ) = ∫ Θ L ϕ ( ⋅ , θ ) b ( θ ) d π ( θ ) in L 2 ( ρ I ) ,
B ( Φ b ) = ∫ Θ B ϕ ( ⋅ , θ ) b ( θ ) d π ( θ ) in L 2 ( ρ B ) . \mathcal{B}(\Phi b) = \int_\Theta\mathcal{B}\phi(\cdot,\theta)b(\theta)\,\mathrm{d}\pi(\theta) \quad\text{in } L^2(\rho_B). B ( Φ b ) = ∫ Θ B ϕ ( ⋅ , θ ) b ( θ ) d π ( θ ) in L 2 ( ρ B ) .
Assumption 5 (Compatibility with the PDE operators) There exists a Banach function space V ⊂ D R V \subset D_R V ⊂ D R continuously embedded in C ( Ω ‾ ) C(\overline{\Omega}) C ( Ω ) such that the map θ ↦ ϕ ( ⋅ , θ ) \theta\mapsto\phi(\cdot,\theta) θ ↦ ϕ ( ⋅ , θ ) is strongly measurable with values in V V V and satisfies
∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ V 2 d π ( θ ) < ∞ . \int_\Theta\|\phi(\cdot,\theta)\|_V^2 \,\mathrm{d}\pi(\theta) < \infty. ∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ V 2 d π ( θ ) < ∞. Furthermore,
L : V → L 2 ( ρ I ) , B : V → L 2 ( ρ B ) \mathcal{L}: V \to L^2(\rho_I), \qquad\mathcal{B}: V \to L^2(\rho_B) L : V → L 2 ( ρ I ) , B : V → L 2 ( ρ B ) are bounded linear operators.
Assumption 5 also implies that H k ⊂ V \mathcal{H}_k \subset V H k ⊂ V and Σ m ϕ ⊂ V \Sigma^\phi_m \subset V Σ m ϕ ⊂ V almost surely when the parameters are sampled from a distribution absolutely continuous with respect to π \pi π . It is used for residual approximation and collocation estimation. For the collocation analysis, we impose an additional condition on the space V V V .
Assumption 6 Let V V V be the space in Assumption 5. For each v ∈ V v \in V v ∈ V , fix pointwise representatives of L v \mathcal{L}v L v on Ω \Omega Ω and B v \mathcal{B}v B v on ∂ Ω \partial\Omega ∂ Ω . For every x ∈ Ω x \in\Omega x ∈ Ω and y ∈ ∂ Ω y \in\partial\Omega y ∈ ∂ Ω , the maps
v ↦ ( L v ) ( x ) , v ↦ ( B v ) ( y ) v \mapsto(\mathcal{L}v)(x), \qquad v \mapsto(\mathcal{B}v)(y) v ↦ ( L v ) ( x ) , v ↦ ( B v ) ( y )
are bounded linear functionals on V V V .
Main Results
Our analysis separates two sources of error: approximation error due to finitely many random features and estimation error due to finitely many collocation observations. We first establish upper and lower bounds for these errors in terms of the eigenvalues of the corresponding operators. We then establish a lower bound on the error under simultaneous restrictions on the numbers of random features and collocation observations and investigate how activation regularity affects these eigenvalues.
For a measurable positive semidefinite kernel k ~ \widetilde{k} k on Θ \Theta Θ , let T ~ : L 2 ( π ) → L 2 ( π ) \widetilde{T}: L^2(\pi) \to L^2(\pi) T : L 2 ( π ) → L 2 ( π ) denote the associated integral operator,
( T ~ a ) ( θ ) : = ∫ Θ k ~ ( θ , θ ′ ) a ( θ ′ ) d π ( θ ′ ) . (\widetilde{T}a)(\theta) := \int_{\Theta} \widetilde{k}(\theta,\theta')a(\theta')\,\mathrm{d}\pi(\theta'). ( T a ) ( θ ) := ∫ Θ k ( θ , θ ′ ) a ( θ ′ ) d π ( θ ′ ) .
We use matching subscripts to distinguish the kernels and their associated integral operators.
Approximation Error
We first study the best approximation of functions in the unit ball of H k \mathcal{H}_k H k by m m m random features. We establish upper and lower bounds in Sobolev norms and in the residual seminorm.
Approximation error in Sobolev norm.
For the prescribed order r r r , we impose Assumption 4. Define the kernel on Θ \Theta Θ by
k ~ r ( θ , θ ′ ) : = ⟨ ϕ ( ⋅ , θ ) , ϕ ( ⋅ , θ ′ ) ⟩ H r ( Ω ) . \widetilde{k}_r(\theta,\theta') := \langle\phi(\cdot,\theta),\phi(\cdot,\theta')\rangle_{H^r(\Omega)}. k r ( θ , θ ′ ) := ⟨ ϕ ( ⋅ , θ ) , ϕ ( ⋅ , θ ′ ) ⟩ H r ( Ω ) .
Let { μ ~ j ( r ) } j ≥ 1 \{\widetilde{\mu}^{(r)}_j\}_{j\geq1} { μ j ( r ) } j ≥ 1 denote the positive eigenvalues of the associated integral operator T ~ r \widetilde{T}_r T r , arranged in nonincreasing order. We first establish an upper bound for the worst-case approximation error in H r ( Ω ) H^r(\Omega) H r ( Ω ) .
Theorem 1 Suppose that { μ ~ j ( r ) } j ≥ 1 \{\widetilde{\mu}^{(r)}_j\}_{j\geq1} { μ j ( r ) } j ≥ 1 satisfies Assumption 1. There exist constants c 0 , m 0 > 0 c_0,m_0>0 c 0 , m 0 > 0 , depending only on { μ ~ j ( r ) } j ≥ 1 \{\widetilde{\mu}^{(r)}_j\}_{j\geq1} { μ j ( r ) } j ≥ 1 and satisfying c 0 m 0 ≥ 1 c_0m_0\geq1 c 0 m 0 ≥ 1 , such that the following holds. For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , let m m m satisfy
m log ( 14 m / δ ) ≥ m 0 . \frac{m}{\log(14m/\delta)} \geq m_0. log ( 14 m / δ ) m ≥ m 0 . Denote
l m : = ⌊ c 0 m log ( 14 m / δ ) ⌋ l_m := \left\lfloor\frac{c_0m}{\log(14m/\delta)} \right\rfloor l m := ⌊ log ( 14 m / δ ) c 0 m ⌋ and p m ( θ ) = q ∗ ( k ~ r , π , μ ~ l m ( r ) ; θ ) p_m(\theta)=q^*(\widetilde{k}_r,\pi,\widetilde{\mu}^{(r)}_{l_m};\theta) p m ( θ ) = q ∗ ( k r , π , μ l m ( r ) ; θ ) . If θ 1 , … , θ m \theta_1,\ldots,\theta_m θ 1 , … , θ m are sampled independently from p m d π p_m\,\mathrm{d}\pi p m d π , then, with probability at least 1 − δ 1-\delta 1 − δ ,
sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ Σ m ϕ ∥ u − u ∗ ∥ H r ( Ω ) 2 ≤ 16 μ ~ l m ( r ) . \sup_{\|u^*\|_{\mathcal{H}_k}\leq1}\inf_{u\in\Sigma_m^\phi}\|u-u^*\|_{H^r(\Omega)}^2 \leq16\widetilde{\mu}^{(r)}_{l_m}. ∥ u ∗ ∥ H k ≤ 1 sup u ∈ Σ m ϕ inf ∥ u − u ∗ ∥ H r ( Ω ) 2 ≤ 16 μ l m ( r ) .
We next establish a lower bound that holds for every fixed choice of feature parameters.
Theorem 2 For every fixed collection of sampled parameters θ 1 , … , θ m \theta_1,\ldots,\theta_m θ 1 , … , θ m in the parameter space Θ \Theta Θ ,
sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ Σ m ∥ u − u ∗ ∥ H r ( Ω ) 2 ≥ μ ~ m + 1 ( r ) . (14) \sup_{\|u^*\|_{\mathcal{H}_k}\le1}\inf_{u\in\Sigma_m}\|u-u^*\|_{\mathcal{H}_r(\Omega)}^2 \ge\tilde{\mu}_{m+1}^{(r)}. \tag*{(14)} ∥ u ∗ ∥ H k ≤ 1 sup u ∈ Σ m inf ∥ u − u ∗ ∥ H r ( Ω ) 2 ≥ μ ~ m + 1 ( r ) . (14)
Approximation error in residual seminorm.
We next consider approximation error measured by the residual seminorm. We impose Assumption 5 and define the kernel
k ~ R η ( θ , θ ′ ) : = ⟨ L ϕ ( ⋅ , θ ) , L ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ I ) + η ⟨ B ϕ ( ⋅ , θ ) , B ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ B ) . \tilde{k}_{R_\eta}(\theta,\theta') := \langle L\phi(\cdot,\theta),L\phi(\cdot,\theta')\rangle_{L^2(\rho_I)}+\eta\langle B\phi(\cdot,\theta),B\phi(\cdot,\theta')\rangle_{L^2(\rho_B)}. k ~ R η ( θ , θ ′ ) := ⟨ L ϕ ( ⋅ , θ ) , L ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ I ) + η ⟨ B ϕ ( ⋅ , θ ) , B ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ B ) .
Let { μ ~ j ( R η ) } j ≥ 1 \{\tilde{\mu}_j^{(R_\eta)}\}_{j\ge1} { μ ~ j ( R η ) } j ≥ 1 denote the positive eigenvalues of the associated integral operator T ~ R η \widetilde{\mathcal{T}}_{R_\eta} T R η , arranged in nonincreasing order. We establish the following upper bound and lower bound for the approximation error in the residual seminorm.
Theorem 3 Suppose that { μ ~ j ( R η ) } j ≥ 1 \{\tilde{\mu}_j^{(R_\eta)}\}_{j\ge1} { μ ~ j ( R η ) } j ≥ 1 satisfies Assumption 1. There exist constants c 0 , m 0 > 0 c_0,m_0>0 c 0 , m 0 > 0 , depending only on { μ ~ j ( R η ) } j ≥ 1 \{\tilde{\mu}_j^{(R_\eta)}\}_{j\ge1} { μ ~ j ( R η ) } j ≥ 1 and satisfying c 0 m 0 ≥ 1 c_0m_0\ge1 c 0 m 0 ≥ 1 , such that the following holds. For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , let m m m satisfy
m log ( 14 m / δ ) ≥ m 0 . \frac{m}{\log(14m/\delta)}\ge m_0. log ( 14 m / δ ) m ≥ m 0 . Denote
l m : = ⌊ c 0 m log ( 14 m / δ ) ⌋ l_m:=\left\lfloor\frac{c_0m}{\log(14m/\delta)}\right\rfloor l m := ⌊ log ( 14 m / δ ) c 0 m ⌋ and
p m ( θ ) = q ∗ ( k ~ R η , π , μ ~ l m ( R η ) ; θ ) . p_m(\theta)=q^*(\tilde{k}_{R_\eta},\pi,\tilde{\mu}_{l_m}^{(R_\eta)};\theta). p m ( θ ) = q ∗ ( k ~ R η , π , μ ~ l m ( R η ) ; θ ) . If θ 1 , … , θ m \theta_1,\ldots,\theta_m θ 1 , … , θ m are sampled independently from p m d π p_m\,\mathrm{d}\pi p m d π , then, with probability at least 1 − δ 1-\delta 1 − δ ,
sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ Σ m ∥ u − u ∗ ∥ R η 2 ≤ 16 μ ~ l m ( R η ) . (15) \sup_{\|u^*\|_{\mathcal{H}_k}\le1}\inf_{u\in\Sigma_m}\|u-u^*\|_{R_\eta}^2\le16\tilde{\mu}_{l_m}^{(R_\eta)}. \tag*{(15)} ∥ u ∗ ∥ H k ≤ 1 sup u ∈ Σ m inf ∥ u − u ∗ ∥ R η 2 ≤ 16 μ ~ l m ( R η ) . (15)
Theorem 4 For every fixed collection of sampled parameters θ 1 , … , θ m \theta_1,\ldots,\theta_m θ 1 , … , θ m in the parameter space Θ \Theta Θ ,
sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ Σ m ∥ u − u ∗ ∥ R η 2 ≥ μ ~ m + 1 ( R η ) . (16) \sup_{\|u^*\|_{\mathcal{H}_k}\le1}\inf_{u\in\Sigma_m}\|u-u^*\|_{R_\eta}^2\ge\tilde{\mu}_{m+1}^{(R_\eta)}. \tag*{(16)} ∥ u ∗ ∥ H k ≤ 1 sup u ∈ Σ m inf ∥ u − u ∗ ∥ R η 2 ≥ μ ~ m + 1 ( R η ) . (16)
The approximation error analysis is deferred to § 5.2 .
Estimation Error
We now study the estimation error associated with finitely many collocation observations. Throughout this subsection, we impose Assumption 5. Let
X n : = ( x 1 , … , x n ) ∈ Ω n I × ( ∂ Ω ) n B , n = n I + n B , X_n:=(x_1,\ldots,x_n)\in\Omega^{n_I}\times(\partial\Omega)^{n_B},\qquad n=n_I+n_B, X n := ( x 1 , … , x n ) ∈ Ω n I × ( ∂ Ω ) n B , n = n I + n B ,
be a collection of collocation points. Define the observation vector by
z n ( X n , u ∗ ) : = ( ( L u ∗ ( x i ) ) i = 1 n I , ( B u ∗ ( x i ) ) i = n I + 1 n ) ⊤ ∈ R n . z_n(X_n,u^*):=\left(\left(Lu^*(x_i)\right)_{i=1}^{n_I},\left(Bu^*(x_i)\right)_{i=n_I+1}^{n}\right)^\top\in\mathbb{R}^n. z n ( X n , u ∗ ) := ( ( L u ∗ ( x i ) ) i = 1 n I , ( B u ∗ ( x i ) ) i = n I + 1 n ) ⊤ ∈ R n .
Our objective is to recover u ∗ u^* u ∗ from the data S n ( u ∗ ) : = ( X n , z n ( X n , u ∗ ) ) S_n(u^*) := (X_n, z_n(X_n,u^*)) S n ( u ∗ ) := ( X n , z n ( X n , u ∗ )) . For λ > 0 \lambda> 0 λ > 0 , consider the regularized estimator based on the data:
u ^ n , η , λ : = argmin u ∈ H k { R ^ n , η ( u ) + λ ∥ u ∥ H k 2 } . (17) \widehat{u}_{n,\eta,\lambda} := \underset{u\in\mathcal{H}_k}{\operatorname{argmin}}\left\{\widehat{R}_{n,\eta}(u) + \lambda\lVert u\rVert_{\mathcal{H}_k}^2\right\}. \tag*{(17)} u n , η , λ := u ∈ H k argmin { R n , η ( u ) + λ ∥ u ∥ H k 2 } . (17)
where R ^ n , η \widehat{R}_{n,\eta} R n , η is the weighted empirical PDE residual defined in (10) . Here η \eta η weights the boundary term in the empirical residual R ^ n , η \widehat{R}_{n,\eta} R n , η , whereas λ \lambda λ weights the squared RKHS norm ∥ u ∥ H k 2 \lVert u\rVert_{\mathcal{H}_k}^2 ∥ u ∥ H k 2 . Let n = n I + n B n=n_I+n_B n = n I + n B and set ν I : = n / n I \nu_I:=n/n_I ν I := n / n I , ν B : = n / n B \nu_B:=n/n_B ν B := n / n B , and ν + : = max { ν I , ν B } \nu_+:=\max\{\nu_I,\nu_B\} ν + := max { ν I , ν B } . Define the kernel on Θ \Theta Θ by
k ~ P D E ( θ , θ ′ ) : = ν I ⟨ L ϕ ( ⋅ , θ ) , L ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ I ) + η ν B ⟨ B ϕ ( ⋅ , θ ) , B ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ B ) . \widetilde{k}_{\mathrm{PDE}}(\theta,\theta') := \nu_I\langle\mathcal{L}\phi(\cdot,\theta),\mathcal{L}\phi(\cdot,\theta')\rangle_{L^2(\rho_I)}+\eta\nu_B\langle\mathcal{B}\phi(\cdot,\theta),\mathcal{B}\phi(\cdot,\theta')\rangle_{L^2(\rho_B)}. k PDE ( θ , θ ′ ) := ν I ⟨ L ϕ ( ⋅ , θ ) , L ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ I ) + η ν B ⟨ B ϕ ( ⋅ , θ ) , B ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ B ) .
Let { μ j P D E } j ≥ 1 \{\mu_j^{\mathrm{PDE}}\}_{j\geq1} { μ j PDE } j ≥ 1 denote the positive eigenvalues of the associated integral operator T ~ P D E \widetilde{T}_{\mathrm{PDE}} T PDE , arranged in nonincreasing order. To specify the interior and boundary sampling densities q I q_I q I and q B q_B q B in the following theorem, define
k L ( x , x ′ ) : = ∫ Θ L ϕ ( x , θ ) L ϕ ( x ′ , θ ) d π ( θ ) , x , x ′ ∈ Ω . (18) k_{\mathcal{L}}(x,x'):=\int_{\Theta}\mathcal{L}\phi(x,\theta)\mathcal{L}\phi(x',\theta)\,\mathrm{d}\pi(\theta),\qquad x,x'\in\Omega. \tag*{(18)} k L ( x , x ′ ) := ∫ Θ L ϕ ( x , θ ) L ϕ ( x ′ , θ ) d π ( θ ) , x , x ′ ∈ Ω. (18)
and
k B ( x , x ′ ) : = ∫ Θ B ϕ ( x , θ ) B ϕ ( x ′ , θ ) d π ( θ ) , x , x ′ ∈ ∂ Ω . (19) k_{\mathcal{B}}(x,x'):=\int_{\Theta}\mathcal{B}\phi(x,\theta)\mathcal{B}\phi(x',\theta)\,\mathrm{d}\pi(\theta),\qquad x,x'\in\partial\Omega. \tag*{(19)} k B ( x , x ′ ) := ∫ Θ B ϕ ( x , θ ) B ϕ ( x ′ , θ ) d π ( θ ) , x , x ′ ∈ ∂ Ω. (19)
We establish an upper bound for the worst-case estimation error in the residual seminorm.
Theorem 5 Fix η > 0 \eta>0 η > 0 and the constants ν I \nu_I ν I and ν B \nu_B ν B . Suppose that the eigenvalues { μ j P D E } j ≥ 1 \{\mu_j^{\mathrm{PDE}}\}_{j\geq1} { μ j PDE } j ≥ 1 satisfy one of the eigenvalue decay conditions in Lemma 20. There exist constants c 0 , n 0 > 0 c_0,n_0>0 c 0 , n 0 > 0 , depending only on { μ j P D E } j ≥ 1 \{\mu_j^{\mathrm{PDE}}\}_{j\geq1} { μ j PDE } j ≥ 1 and the fixed sampling ratios ν I , ν B \nu_I,\nu_B ν I , ν B , and satisfying c 0 n 0 ≥ 1 c_0n_0\geq1 c 0 n 0 ≥ 1 , such that the following holds.
For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , let n n n satisfy n log ( 28 n / δ ) ≥ n 0 \frac{n}{\log(28n/\delta)}\geq n_0 l o g ( 28 n / δ ) n ≥ n 0 . Denote l n : = ⌈ c 0 n log ( 28 n / δ ) ⌉ l_n:=\left\lceil\frac{c_0n}{\log(28n/\delta)}\right\rceil l n := ⌈ l o g ( 28 n / δ ) c 0 n ⌉ , q I = q ∗ ( k L , ρ I , μ l n P D E ; x ) q_I=q^*(k_{\mathcal{L}},\rho_I,\mu_{l_n}^{\mathrm{PDE}};x) q I = q ∗ ( k L , ρ I , μ l n PDE ; x ) and q B = q ∗ ( k B , ρ B , μ l n P D E / η ; x ) q_B=q^*(k_{\mathcal{B}},\rho_B,\mu_{l_n}^{\mathrm{PDE}}/\eta;x) q B = q ∗ ( k B , ρ B , μ l n PDE / η ; x ) . If x 1 , … , x n I x_1,\ldots,x_{n_I} x 1 , … , x n I are sampled independently from q I d ρ I q_I\,\mathrm{d}\rho_I q I d ρ I and x n I + 1 , … , x n x_{n_I+1},\ldots,x_n x n I + 1 , … , x n are sampled independently from q B d ρ B q_B\,\mathrm{d}\rho_B q B d ρ B , with the two collections independent, then, with probability at least 1 − δ 1-\delta 1 − δ , the estimator u ^ n , η , λ \widehat{u}_{n,\eta,\lambda} u n , η , λ defined in (17) satisfies the following bound simultaneously for every 0 < λ ≤ μ l n P D E 0<\lambda\leq\mu_{l_n}^{\mathrm{PDE}} 0 < λ ≤ μ l n PDE .
sup ∥ u ∗ ∥ H k ≤ 1 ∥ u ^ n , η , λ − u ∗ ∥ R η 2 ≤ 4 μ l n P D E . \sup_{\lVert u^*\rVert_{\mathcal{H}_k}\leq1}\lVert\widehat{u}_{n,\eta,\lambda}-u^*\rVert_{\mathcal{R}_\eta}^2\leq4\mu_{l_n}^{\mathrm{PDE}}. ∥ u ∗ ∥ H k ≤ 1 sup ∥ u n , η , λ − u ∗ ∥ R η 2 ≤ 4 μ l n PDE .
We next establish a lower bound for every fixed collection of collocation points. Define the class of all estimators based on the collocation points and the corresponding PDE observations by
M n P D E : = { G : ( Ω n I × ( ∂ Ω ) n B ) × R n → D R } . (20) \mathcal{M}_n^{\mathrm{PDE}}:=\left\{G:(\Omega^{n_I}\times(\partial\Omega)^{n_B})\times\mathbb{R}^n\to D_{\mathbb{R}}\right\}. \tag*{(20)} M n PDE := { G : ( Ω n I × ( ∂ Ω ) n B ) × R n → D R } . (20)
Theorem 6 For every fixed collection X n X_n X n of collocation points, let S n ( u ∗ ) = ( X n , z n ( X n , u ∗ ) ) S_n(u^*)=(X_n,z_n(X_n,u^*)) S n ( u ∗ ) = ( X n , z n ( X n , u ∗ )) . Then we have
inf G ∈ M n P D E sup ∥ u ∗ ∥ H k ≤ 1 ∥ G ( S n ( u ∗ ) ) − u ∗ ∥ R n 2 ≥ μ n + 1 P D E ν + . \inf_{G\in\mathcal{M}^{\mathrm{PDE}}_n}\sup_{\lVert u^*\rVert_{H_k}\leq1}\lVert G(S_n(u^*))-u^*\rVert_{\mathcal{R}_n}^2\geq\frac{\mu_{n+1}^{\mathrm{PDE}}}{\nu_+}. G ∈ M n PDE inf ∥ u ∗ ∥ H k ≤ 1 sup ∥ G ( S n ( u ∗ )) − u ∗ ∥ R n 2 ≥ ν + μ n + 1 PDE .
We now consider the combined restrictions imposed by finitely many random features and collocation observations. Consider the class of estimators in which the outputs are restricted to the random feature model
M m , n : = { G : ( Ω n × ( ∂ Ω ) n B ) × R n → Σ m ϕ } . (21) \mathcal{M}_{m,n}:=\{G:(\Omega^n\times(\partial\Omega)^{n_B})\times\mathbb{R}^n\to\Sigma_m^\phi\}. \tag*{(21)} M m , n := { G : ( Ω n × ( ∂ Ω ) n B ) × R n → Σ m ϕ } . (21)
Define the optimal worst-case squared residual error by
E m , n : = inf G ∈ M m , n sup ∥ u ∗ ∥ H k ≤ 1 ∥ G ( S n ( u ∗ ) ) − u ∗ ∥ R n 2 . (22) E_{m,n}:=\inf_{G\in\mathcal{M}_{m,n}}\sup_{\lVert u^*\rVert_{H_k}\leq1}\lVert G(S_n(u^*))-u^*\rVert_{\mathcal{R}_n}^2. \tag*{(22)} E m , n := G ∈ M m , n inf ∥ u ∗ ∥ H k ≤ 1 sup ∥ G ( S n ( u ∗ )) − u ∗ ∥ R n 2 . (22)
Corollary 7 For every fixed admissible choice of feature parameters { θ i } i = 1 m \{\theta_i\}_{i=1}^m { θ i } i = 1 m and collocation points X n X_n X n , we have
E m , n ≥ 1 ν + max { μ m + 1 P D E , μ n + 1 P D E } . (23) E_{m,n}\geq\frac{1}{\nu_+}\max\{\mu_{m+1}^{\mathrm{PDE}},\mu_{n+1}^{\mathrm{PDE}}\}. \tag*{(23)} E m , n ≥ ν + 1 max { μ m + 1 PDE , μ n + 1 PDE } . (23)
The RFM solution in (11) , viewed as a function of S n ( u ∗ ) S_n(u^*) S n ( u ∗ ) , defines an estimator G ∈ M m , n G\in\mathcal{M}_{m,n} G ∈ M m , n , with a fixed selection rule based only on these data if the minimizer is not unique. Therefore, Corollary 7 yields
sup ∥ u ∗ ∥ H k ≤ 1 ∥ u ^ n , m , η − u ∗ ∥ R n 2 ≥ E m , n ≥ 1 ν + max { μ m + 1 P D E , μ n + 1 P D E } . \sup_{\lVert u^*\rVert_{H_k}\leq1}\lVert\widehat{u}_{n,m,\eta}-u^*\rVert_{\mathcal{R}_n}^2\geq E_{m,n}\geq\frac{1}{\nu_+}\max\{\mu_{m+1}^{\mathrm{PDE}},\mu_{n+1}^{\mathrm{PDE}}\}. ∥ u ∗ ∥ H k ≤ 1 sup ∥ u n , m , η − u ∗ ∥ R n 2 ≥ E m , n ≥ ν + 1 max { μ m + 1 PDE , μ n + 1 PDE } .
The estimation error analysis is deferred to § 5.3 .
Remark 1 The estimation error analyzed in this section is measured in the residual seminorm. Upper bounds in other norms can be obtained from stability estimates for the underlying boundary value problem. For example, let Ω ⊂ R d \Omega\subset\mathbb{R}^d Ω ⊂ R d be a bounded domain with C ∞ C^\infty C ∞ boundary, and consider
L u = − ∑ i , j = 1 d a i j ( x ) ∂ i j u + ∑ i = 1 d b i ( x ) ∂ i u + c ( x ) u , B u = Tr u , (24) \mathcal{L}u=-\sum_{i,j=1}^{d}a^{ij}(x)\partial_{ij}u+\sum_{i=1}^{d}b^i(x)\partial_i u+c(x)u,\qquad Bu=\operatorname{Tr}u, \tag*{(24)} L u = − i , j = 1 ∑ d a ij ( x ) ∂ ij u + i = 1 ∑ d b i ( x ) ∂ i u + c ( x ) u , B u = Tr u , (24)
where the coefficients a i j , b i , c ∈ C ∞ ( Ω ‾ ) a^{ij},b^i,c\in C^\infty(\overline{\Omega}) a ij , b i , c ∈ C ∞ ( Ω ) and satisfy
∑ i , j = 1 d a i j ( x ) ξ i ξ j ≥ α ∣ ξ ∣ 2 , x ∈ Ω ‾ , ξ ∈ R d , \sum_{i,j=1}^{d}a^{ij}(x)\xi_i\xi_j\geq\alpha\lvert\xi\rvert^2,\qquad x\in\overline{\Omega},\quad\xi\in\mathbb{R}^d, i , j = 1 ∑ d a ij ( x ) ξ i ξ j ≥ α ∣ ξ ∣ 2 , x ∈ Ω , ξ ∈ R d ,
for some α > 0 \alpha>0 α > 0 . Assume that the homogeneous Dirichlet problem
L v = 0 in Ω , B v = 0 on ∂ Ω \mathcal{L}v=0\quad\text{in }\Omega,\qquad Bv=0\quad\text{on }\partial\Omega L v = 0 in Ω , B v = 0 on ∂ Ω
has only the trivial solution. The stability estimate in [36 Appendix B.1, Eq. (B2)] gives
∥ v ∥ H 1 / 2 ( Ω ) 2 ≲ ∥ L v ∥ L 2 ( Ω ) 2 + ∥ B v ∥ L 2 ( ∂ Ω ) 2 , v ∈ H 2 ( Ω ) . \lVert v\rVert_{H^{1/2}(\Omega)}^2 \lesssim\lVert\mathcal{L}v\rVert_{L^2(\Omega)}^2 + \lVert\mathcal{B}v\rVert_{L^2(\partial\Omega)}^2,\qquad v \in H^2(\Omega). ∥ v ∥ H 1/2 ( Ω ) 2 ≲ ∥ L v ∥ L 2 ( Ω ) 2 + ∥ B v ∥ L 2 ( ∂ Ω ) 2 , v ∈ H 2 ( Ω ) .
Suppose that ρ I \rho_I ρ I and ρ B \rho_B ρ B have densities bounded below by positive constants with respect to Lebesgue measure on Ω \Omega Ω and surface measure on ∂ Ω \partial\Omega ∂ Ω , respectively. For an exact solution u ∗ ∈ H 2 ( Ω ) u^* \in H^2(\Omega) u ∗ ∈ H 2 ( Ω ) satisfying L u ∗ = f ∗ \mathcal{L}u^* = f^* L u ∗ = f ∗ and B u ∗ = g ∗ \mathcal{B}u^* = g^* B u ∗ = g ∗ , every u ∈ H 2 ( Ω ) u \in H^2(\Omega) u ∈ H 2 ( Ω ) therefore satisfies
∥ u − u ∗ ∥ H 1 / 2 ( Ω ) 2 ≲ ∥ L u − f ∗ ∥ L 2 ( ρ I ) 2 + ∥ B u − g ∗ ∥ L 2 ( ρ B ) 2 . \lVert u-u^*\rVert_{H^{1/2}(\Omega)}^2 \lesssim\lVert\mathcal{L}u-f^*\rVert_{L^2(\rho_I)}^2 + \lVert\mathcal{B}u-g^*\rVert_{L^2(\rho_B)}^2. ∥ u − u ∗ ∥ H 1/2 ( Ω ) 2 ≲ ∥ L u − f ∗ ∥ L 2 ( ρ I ) 2 + ∥ B u − g ∗ ∥ L 2 ( ρ B ) 2 .
Consequently, upper bounds for the residual seminorm yield upper bounds of the same order for the squared H 1 / 2 ( Ω ) H^{1/2}(\Omega) H 1/2 ( Ω ) error, and hence for the squared L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) error.
Eigenvalue Decay Induced by Activation Regularity
To obtain explicit convergence rates, we estimate the eigenvalue decay of the integral operator associated with the chosen feature function. We consider the commonly used features
ϕ ( x , θ ) = σ ( k ⊤ x + b ) , (25) \phi(x,\theta)=\sigma(k^\top x+b), \tag*{(25)} ϕ ( x , θ ) = σ ( k ⊤ x + b ) , (25)
where σ : R → R \sigma:\mathbb{R}\to\mathbb{R} σ : R → R is the activation function and θ = ( k , b ) \theta=(k,b) θ = ( k , b ) belongs to a compact parameter set Θ ⊂ R d × R \Theta\subset\mathbb{R}^d\times\mathbb{R} Θ ⊂ R d × R . The residual approximation and collocation estimation bounds in Theorems 3–6 apply to general linear differential and boundary operators under their respective stated assumptions. For fixed proportions of interior and boundary observations, (58) implies that μ ~ j ( R η ) ≳ μ j P D E \widetilde{\mu}^{(R_\eta)}_j \gtrsim\mu^{\mathrm{PDE}}_j μ j ( R η ) ≳ μ j PDE uniformly in j j j . It therefore suffices to estimate the PDE eigenvalues. To estimate the PDE eigenvalues, we specialize to the elliptic boundary value problem in (24) . The same approach applies to other linear equations. The following theorem bounds the Sobolev and PDE eigenvalues in terms of the regularity of σ \sigma σ .
Theorem 8 Assume that ρ I \rho_I ρ I is a probability measure on Ω \Omega Ω with density w I ∈ L ∞ ( Ω ) w_I\in L^\infty(\Omega) w I ∈ L ∞ ( Ω ) with respect to Lebesgue measure. Let ρ B \rho_B ρ B be a probability measure on ∂ Ω \partial\Omega ∂ Ω . Let L \mathcal{L} L be the elliptic operator in (24) , with a i j , b i , c ∈ L ∞ ( Ω ) a^{ij}, b^i, c\in L^\infty(\Omega) a ij , b i , c ∈ L ∞ ( Ω ) , and let B u = u ∣ ∂ Ω \mathcal{B}u=u|_{\partial\Omega} B u = u ∣ ∂ Ω . Fix η > 0 \eta>0 η > 0 and the constants ν I \nu_I ν I and ν B \nu_B ν B . For θ = ( k , b ) ∈ Θ \theta=(k,b)\in\Theta θ = ( k , b ) ∈ Θ , set ϕ ( x , θ ) = σ ( k ⊤ x + b ) \phi(x,\theta)=\sigma(k^\top x+b) ϕ ( x , θ ) = σ ( k ⊤ x + b ) , and choose M > max x ∈ Ω , θ ∈ Θ ∣ k ⊤ x + b ∣ M>\max_{x\in\Omega,\ \theta\in\Theta}|k^\top x+b| M > max x ∈ Ω , θ ∈ Θ ∣ k ⊤ x + b ∣ .
(i) If σ ∈ H p ( [ − M , M ] ) ∩ C p − 1 ( [ − M , M ] ) \sigma\in H^p([-M,M])\cap C^{p-1}([-M,M]) σ ∈ H p ([ − M , M ]) ∩ C p − 1 ([ − M , M ]) for an integer p ≥ max { 2 , r } p\ge\max\{2,r\} p ≥ max { 2 , r } , then
μ ~ j ( r ) = o ( j − 2 ( p − r ) d − 1 ) , μ j P D E = o ( j − 2 ( p − 2 ) d − 1 ) . \widetilde{\mu}^{(r)}_j=o\left(j^{-\frac{2(p-r)}{d}-1}\right),\qquad\mu^{\mathrm{PDE}}_j=o\left(j^{-\frac{2(p-2)}{d}-1}\right). μ j ( r ) = o ( j − d 2 ( p − r ) − 1 ) , μ j PDE = o ( j − d 2 ( p − 2 ) − 1 ) . (ii) If σ \sigma σ is real analytic on an open interval containing [ − M , M ] [-M,M] [ − M , M ] , then there exist constants C > 0 C>0 C > 0 and τ ∈ ( 0 , 1 ) \tau\in(0,1) τ ∈ ( 0 , 1 ) such that
μ ~ j ( r ) ≤ C τ 2 j 1 / d , μ j P D E ≤ C τ 2 j 1 / d . \widetilde{\mu}^{(r)}_j\le C\tau^{2j^{1/d}},\qquad\mu^{\mathrm{PDE}}_j\le C\tau^{2j^{1/d}}. μ j ( r ) ≤ C τ 2 j 1/ d , μ j PDE ≤ C τ 2 j 1/ d .
Remark 2 Under the assumptions of Theorem 8, if σ ∈ C ∞ ( [ − M , M ] ) \sigma\in C^\infty([-M,M]) σ ∈ C ∞ ([ − M , M ]) , then, for every fixed integer r ≥ 0 r\ge0 r ≥ 0 and every t > 0 t>0 t > 0 ,
μ ~ m ( r ) = o ( m − t ) , μ n P D E = o ( n − t ) . \widetilde{\mu}^{(r)}_m=o(m^{-t}),\qquad\mu^{\mathrm{PDE}}_n=o(n^{-t}). μ m ( r ) = o ( m − t ) , μ n PDE = o ( n − t ) .
Thus, both eigenvalue sequences decay super-algebraically. Smoothness alone does not imply the stretched-exponential bounds in the analytic case.
For the rectified power activation ReLU k ( t ) : = ( max { t , 0 } ) k \operatorname{ReLU}^{k}(t) := (\max\{t,0\})^{k} ReLU k ( t ) := ( max { t , 0 } ) k , where k ≥ 1 k \ge1 k ≥ 1 is an integer, we obtain the following sharper upper bounds.
Theorem 9 Let σ = ReLU k \sigma= \operatorname{ReLU}^{k} σ = ReLU k for an integer k ≥ 1 k \ge1 k ≥ 1 . For each integer 0 ≤ r ≤ k 0 \le r \le k 0 ≤ r ≤ k , the eigenvalues { μ ~ j ( r ) } j ≥ 1 \{\tilde{\mu}_{j}^{(r)}\}_{j\ge1} { μ ~ j ( r ) } j ≥ 1 satisfy
μ ~ j ( r ) ≤ C j − 1 − 2 ( k − r ) + 1 d , j ≥ 1. (26) \tilde{\mu}_{j}^{(r)} \le Cj^{-1-\frac{2(k-r)+1}{d}}, \qquad j \ge1. \tag*{(26)} μ ~ j ( r ) ≤ C j − 1 − d 2 ( k − r ) + 1 , j ≥ 1. (26) For k ≥ 2 k \ge2 k ≥ 2 , consider the second-order elliptic operator L \mathcal{L} L and the Dirichlet boundary operator B u = u ∣ ∂ Ω Bu = u|_{\partial\Omega} B u = u ∣ ∂ Ω specified in Theorem 8 . The eigenvalues { μ j P D E } j ≥ 1 \{\mu_{j}^{\mathrm{PDE}}\}_{j\ge1} { μ j PDE } j ≥ 1 satisfy
μ j P D E ≤ C j − 1 − 2 k − 3 d , j ≥ 1. \mu_{j}^{\mathrm{PDE}} \le Cj^{-1-\frac{2k-3}{d}}, \qquad j \ge1. μ j PDE ≤ C j − 1 − d 2 k − 3 , j ≥ 1.
The eigenvalue bound in Theorem 9 , together with Theorem 1, yields the following approximation rate. For each integer 0 ≤ r ≤ k 0 \le r \le k 0 ≤ r ≤ k and fixed δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , a sampling density can be chosen such that, for all sufficiently large m m m , with probability at least 1 − δ 1-\delta 1 − δ ,
sup ∥ u ∥ H k ≤ 1 u ∈ H m ∥ u − u ∗ ∥ H r ( Ω ) ≤ C δ ( m log m ) − 1 2 − 2 ( k − r ) + 1 2 d . \sup_{\substack{\|u\|_{\mathcal{H}^{k}}\le1\\u\in\mathcal{H}_{m}}}\|u-u^{*}\|_{H^{r}(\Omega)} \le C_{\delta}\left(\frac{m}{\log m}\right)^{-\frac{1}{2}-\frac{2(k-r)+1}{2d}}. ∥ u ∥ H k ≤ 1 u ∈ H m sup ∥ u − u ∗ ∥ H r ( Ω ) ≤ C δ ( log m m ) − 2 1 − 2 d 2 ( k − r ) + 1 .
Unlike [18 ] , our analysis allows general compact parameter sets rather than requiring the feature parameters to lie on the unit sphere. When Θ = S d \Theta= \mathbb{S}^{d} Θ = S d and π \pi π is uniform, the RKHS H k \mathcal{H}_{k} H k coincides with H ( d + 2 k + 1 ) / 2 ( Ω ) H^{(d+2k+1)/2}(\Omega) H ( d + 2 k + 1 ) /2 ( Ω ) up to equivalent norms; see [18 ] , Theorem 2.3. Our bound therefore recovers the approximation rate in [18 ] , Theorem 6.2, possibly under a different sampling density, and extends this rate to more general compact parameter sets. The proofs of Theorems 8 and 9 are deferred to Appendices B.§ 1 and B.§ 2 , respectively.
Approximation and Estimation Analysis
This section develops the analysis underlying the approximation and estimation bounds in § 4 . We first develop a duality framework in Hilbert spaces that establishes an isometric correspondence between function spaces on the computational and parameter domains, yielding an identity between the corresponding approximation errors. We then use this framework to derive approximation bounds in the Sobolev norm and residual seminorm through kernel ridge regression on the parameter domain. Finally, we analyze estimation errors and combine the approximation and estimation lower bounds to quantify the joint limitations imposed by finitely many random features and collocation observations.
A Duality Framework in Hilbert Spaces
To analyze finite-feature approximation in different error norms, we develop a duality framework in Hilbert spaces. The standard operator-theoretic framework for kernel methods is typically formulated in the L 2 L^2 L 2 setting, as in [14 Appendix B.2] . To accommodate the norms arising in PDEs, we extend this framework to general Hilbert spaces, which provides the foundation for our subsequent analysis. Let X \mathcal{X} X be a nonempty set, and let the ambient space B B B be a separable Hilbert space of real-valued functions on X \mathcal{X} X . The choice of B B B will depend on the error metric under consideration. Let Θ \Theta Θ be the parameter space equipped with the reference distribution π \pi π . To ensure that the associated feature operator is Hilbert–Schmidt, we impose the following condition.
Assumption 7 The map θ ↦ ϕ ( ⋅ , θ ) \theta\mapsto\phi(\cdot,\theta) θ ↦ ϕ ( ⋅ , θ ) is strongly measurable as a B B B -valued map and satisfies
∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ B 2 d π ( θ ) < ∞ . \int_{\Theta} \|\phi(\cdot,\theta)\|_{B}^{2}\,\mathrm{d}\pi(\theta) < \infty. ∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ B 2 d π ( θ ) < ∞. Equivalently, ϕ ∈ L 2 ( π ; B ) \phi\in L^{2}(\pi;B) ϕ ∈ L 2 ( π ; B ) in the Bochner sense.
Define the feature operator Φ : L 2 ( π ) → B \Phi: L^{2}(\pi) \to B Φ : L 2 ( π ) → B by
( Φ α ) ( x ) = ⟨ ϕ ( x , ⋅ ) , α ⟩ L 2 ( π ) , α ∈ L 2 ( π ) . (27) (\Phi\alpha)(x) = \langle\phi(x,\cdot),\alpha\rangle_{L^{2}(\pi)}, \qquad\alpha\in L^{2}(\pi). \tag*{(27)} ( Φ α ) ( x ) = ⟨ ϕ ( x , ⋅ ) , α ⟩ L 2 ( π ) , α ∈ L 2 ( π ) . (27)
Its adjoint Φ ∗ : B → L 2 ( π ) \Phi^{*} : B \to L^{2}(\pi) Φ ∗ : B → L 2 ( π ) is given by
( Φ ∗ f ) ( θ ) = ⟨ ϕ ( ⋅ , θ ) , f ⟩ B , f ∈ B . (28) (\Phi^{*}f)(\theta) = \langle\phi(\cdot,\theta),f\rangle_{B}, \qquad f \in B. \tag*{(28)} ( Φ ∗ f ) ( θ ) = ⟨ ϕ ( ⋅ , θ ) , f ⟩ B , f ∈ B . (28)
Assumption 7 implies that Φ ∗ \Phi^{*} Φ ∗ is Hilbert–Schmidt. Indeed, for any orthonormal basis { e j } j = 1 ∞ \{e_j\}_{j=1}^{\infty} { e j } j = 1 ∞ of B B B , Tonelli’s theorem and Parseval’s identity yield
∥ Φ ∗ ∥ H S 2 = ∑ j = 1 ∞ ∥ Φ ∗ e j ∥ L 2 ( π ) 2 = ∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ B 2 d π ( θ ) < ∞ . \|\Phi^{*}\|_{\mathrm{HS}}^{2} = \sum_{j=1}^{\infty}\|\Phi^{*}e_j\|_{L^{2}(\pi)}^{2} = \int_{\Theta}\|\phi(\cdot,\theta)\|_{B}^{2}\,\mathrm{d}\pi(\theta) < \infty. ∥ Φ ∗ ∥ HS 2 = j = 1 ∑ ∞ ∥ Φ ∗ e j ∥ L 2 ( π ) 2 = ∫ Θ ∥ ϕ ( ⋅ , θ ) ∥ B 2 d π ( θ ) < ∞.
Hence, Φ ∗ \Phi^{*} Φ ∗ is compact and admits a singular value decomposition:
Φ ∗ = ∑ j = 1 ∞ μ j 1 / 2 e ~ j ⊗ e j (29) \Phi^{*} = \sum_{j=1}^{\infty}\mu_j^{1/2}\widetilde{e}_j \otimes e_j \tag*{(29)} Φ ∗ = j = 1 ∑ ∞ μ j 1/2 e j ⊗ e j (29)
where { e ~ j } j = 1 ∞ \{\widetilde{e}_j\}_{j=1}^{\infty} { e j } j = 1 ∞ and { e j } j = 1 ∞ \{e_j\}_{j=1}^{\infty} { e j } j = 1 ∞ are the corresponding orthonormal bases of L 2 ( π ) L^{2}(\pi) L 2 ( π ) and B B B , respectively. Consequently, the operators T : = Φ Φ ∗ : B → B \mathcal{T} := \Phi\Phi^{*} : B \to B T := Φ Φ ∗ : B → B and T ~ : = Φ ∗ Φ : L 2 ( π ) → L 2 ( π ) \widetilde{\mathcal{T}} := \Phi^{*}\Phi: L^{2}(\pi) \to L^{2}(\pi) T := Φ ∗ Φ : L 2 ( π ) → L 2 ( π ) are self-adjoint, positive, and compact. Their spectral decompositions are given by
T = ∑ j = 1 ∞ μ j e j ⊗ e j , T ~ = ∑ j = 1 ∞ μ j e ~ j ⊗ e ~ j . \mathcal{T} = \sum_{j=1}^{\infty}\mu_j e_j \otimes e_j, \qquad\widetilde{\mathcal{T}} = \sum_{j=1}^{\infty}\mu_j \widetilde{e}_j \otimes\widetilde{e}_j. T = j = 1 ∑ ∞ μ j e j ⊗ e j , T = j = 1 ∑ ∞ μ j e j ⊗ e j .
We can therefore define the interpolation spaces on X \mathcal{X} X and Θ \Theta Θ , analogously to (3) , as follows:
H s = { ∑ j = 1 ∞ a j μ j s / 2 e j : ∑ j = 1 ∞ a j 2 < ∞ } , ∥ ∑ j = 1 ∞ a j μ j s / 2 e j ∥ H s 2 = ∑ j = 1 ∞ a j 2 . (30) \mathcal{H}^{s}=\left\{\sum_{j=1}^{\infty}a_j\mu_j^{s/2}e_j:\sum_{j=1}^{\infty}a_j^2<\infty\right\},\qquad\left\|\sum_{j=1}^{\infty}a_j\mu_j^{s/2}e_j\right\|_{\mathcal{H}^{s}}^2=\sum_{j=1}^{\infty}a_j^2. \tag*{(30)} H s = { j = 1 ∑ ∞ a j μ j s /2 e j : j = 1 ∑ ∞ a j 2 < ∞ } , j = 1 ∑ ∞ a j μ j s /2 e j H s 2 = j = 1 ∑ ∞ a j 2 . (30)
and
H ~ s = { ∑ j = 1 ∞ a j μ j s / 2 e ~ j : ∑ j = 1 ∞ a j 2 < ∞ } , ∥ ∑ j = 1 ∞ a j μ j s / 2 e ~ j ∥ H ~ s 2 = ∑ j = 1 ∞ a j 2 . (31) \widetilde{\mathcal{H}}^{s}=\left\{\sum_{j=1}^{\infty}a_j\mu_j^{s/2}\widetilde{e}_j:\sum_{j=1}^{\infty}a_j^2<\infty\right\},\qquad\left\|\sum_{j=1}^{\infty}a_j\mu_j^{s/2}\widetilde{e}_j\right\|_{\widetilde{\mathcal{H}}^{s}}^2=\sum_{j=1}^{\infty}a_j^2. \tag*{(31)} H s = { j = 1 ∑ ∞ a j μ j s /2 e j : j = 1 ∑ ∞ a j 2 < ∞ } , j = 1 ∑ ∞ a j μ j s /2 e j H s 2 = j = 1 ∑ ∞ a j 2 . (31)
By definition, H 0 = B \mathcal{H}^{0}=B H 0 = B and H ~ 0 = L 2 ( π ) \widetilde{\mathcal{H}}^{0}=L^{2}(\pi) H 0 = L 2 ( π ) . The following lemma shows that Φ ∗ \Phi^{*} Φ ∗ induces an isometric isomorphism between H s − 1 \mathcal{H}^{s-1} H s − 1 and H ~ s \widetilde{\mathcal{H}}^{s} H s .
Lemma 10 Let s ≥ 1 s\ge1 s ≥ 1 . For any g ∈ H ~ s g\in\widetilde{\mathcal{H}}^{s} g ∈ H s , there exists a unique u ∈ H s − 1 u\in\mathcal{H}^{s-1} u ∈ H s − 1 such that Φ ∗ u = g \Phi^{*}u=g Φ ∗ u = g and ∥ g ∥ H ~ s = ∥ u ∥ H s − 1 \|g\|_{\widetilde{\mathcal{H}}^{s}}=\|u\|_{\mathcal{H}^{s-1}} ∥ g ∥ H s = ∥ u ∥ H s − 1 .
Proof Following the argument in the proof of [14 Proposition B.2] , let
u g = ∑ j = 1 ∞ μ j − 1 / 2 ⟨ g , e ~ j ⟩ L 2 ( π ) e j . u_g=\sum_{j=1}^{\infty}\mu_j^{-1/2}\langle g,\widetilde{e}_j\rangle_{L^2(\pi)}e_j. u g = j = 1 ∑ ∞ μ j − 1/2 ⟨ g , e j ⟩ L 2 ( π ) e j .
It follows from the singular value decomposition of Φ ∗ \Phi^{*} Φ ∗ as in (29) that
Φ ∗ u g = ∑ j = 1 ∞ ⟨ g , e ~ j ⟩ L 2 ( π ) e ~ j = g , \Phi^{*}u_g=\sum_{j=1}^{\infty}\langle g,\widetilde{e}_j\rangle_{L^2(\pi)}\widetilde{e}_j=g, Φ ∗ u g = j = 1 ∑ ∞ ⟨ g , e j ⟩ L 2 ( π ) e j = g ,
and
∥ u g ∥ H s − 1 2 = ∑ j = 1 ∞ μ j − ( s − 1 ) μ j − 1 ⟨ g , e ~ j ⟩ L 2 ( π ) 2 = ∥ g ∥ H ~ s 2 . \|u_g\|_{\mathcal{H}^{s-1}}^2=\sum_{j=1}^{\infty}\mu_j^{-(s-1)}\mu_j^{-1}\langle g,\widetilde{e}_j\rangle_{L^2(\pi)}^2=\|g\|_{\widetilde{\mathcal{H}}^{s}}^2. ∥ u g ∥ H s − 1 2 = j = 1 ∑ ∞ μ j − ( s − 1 ) μ j − 1 ⟨ g , e j ⟩ L 2 ( π ) 2 = ∥ g ∥ H s 2 .
It suffices to justify the uniqueness. Suppose u = ∑ j = 1 ∞ u j e j ∈ H s − 1 u=\sum_{j=1}^{\infty}u_je_j\in\mathcal{H}^{s-1} u = ∑ j = 1 ∞ u j e j ∈ H s − 1 such that Φ ∗ u = ∑ j = 1 ∞ μ j 1 / 2 u j e ~ j = g \Phi^{*}u=\sum_{j=1}^{\infty}\mu_j^{1/2}u_j\widetilde{e}_j=g Φ ∗ u = ∑ j = 1 ∞ μ j 1/2 u j e j = g . Since μ j > 0 \mu_j>0 μ j > 0 for all j ≥ 1 j\ge1 j ≥ 1 , we must have u j = μ j − 1 / 2 ⟨ g , e ~ j ⟩ L 2 ( π ) u_j=\mu_j^{-1/2}\langle g,\widetilde{e}_j\rangle_{L^2(\pi)} u j = μ j − 1/2 ⟨ g , e j ⟩ L 2 ( π ) for all j j j . Consequently, u = u g u=u_g u = u g . □
The following lemma relates the approximation errors in the two spaces, provided that both error norms are finite.
Lemma 11 Let s ≥ 1 s\ge1 s ≥ 1 and u ∗ ∈ H s − 1 u^*\in\mathcal{H}^{s-1} u ∗ ∈ H s − 1 , and set g ∗ = Φ ∗ u ∗ ∈ H ~ s g^*=\Phi^{*}u^*\in\widetilde{\mathcal{H}}^{s} g ∗ = Φ ∗ u ∗ ∈ H s . For any u ^ ∈ B \widehat{u}\in B u ∈ B , define g ^ = Φ ∗ u ^ \widehat{g}=\Phi^{*}\widehat{u} g = Φ ∗ u . Then, for every p ∈ [ 0 , s ] p\in[0,s] p ∈ [ 0 , s ] , we have
∥ g ^ − g ∗ ∥ H ~ p 2 = ∥ u ^ − u ∗ ∥ H p − 1 2 (32) \|\widehat{g}-g^*\|_{\widetilde{\mathcal{H}}^{p}}^2=\|\widehat{u}-u^*\|_{\mathcal{H}^{p-1}}^2 \tag*{(32)} ∥ g − g ∗ ∥ H p 2 = ∥ u − u ∗ ∥ H p − 1 2 (32) whenever u ^ − u ∗ ∈ H p − 1 \widehat{u}-u^*\in\mathcal{H}^{p-1} u − u ∗ ∈ H p − 1 and g ^ − g ∗ ∈ H ~ p \widehat{g}-g^*\in\widetilde{\mathcal{H}}^{p} g − g ∗ ∈ H p .
Proof. The inclusion g ∗ ∈ H ~ s g^* \in\widetilde{\mathcal{H}}^s g ∗ ∈ H s follows from Lemma 10 . Set
b : = u ^ − u ∗ , h : = g ^ − g ∗ = Φ ∗ b . b := \widehat{u}-u^*, \qquad h := \widehat{g}-g^*=\Phi^*b. b := u − u ∗ , h := g − g ∗ = Φ ∗ b .
Write b = ∑ j = 1 ∞ b j e j b=\sum_{j=1}^{\infty}b_je_j b = ∑ j = 1 ∞ b j e j , with convergence in H p − 1 \mathcal{H}^{p-1} H p − 1 , and define
b N : = ∑ j = 1 N b j e j , h N : = Φ ∗ b N = ∑ j = 1 N μ j 1 / 2 b j e ~ j . b_N:=\sum_{j=1}^{N}b_je_j,\qquad h_N:=\Phi^*b_N=\sum_{j=1}^{N}\mu_j^{1/2}b_j\widetilde e_j. b N := j = 1 ∑ N b j e j , h N := Φ ∗ b N = j = 1 ∑ N μ j 1/2 b j e j .
By the spectral representation of Φ ∗ \Phi^* Φ ∗ , h N h_N h N is the spectral truncation of h h h . It holds that
b N → b in H p − 1 , h N → h in H ~ p . b_N\to b\ \text{ in }\mathcal{H}^{p-1},\qquad h_N\to h\ \text{ in }\widetilde{\mathcal{H}}^p. b N → b in H p − 1 , h N → h in H p .
For each N N N ,
∥ h N ∥ H ~ p 2 = ∥ T − p / 2 Φ ∗ b N ∥ L 2 ( π ) 2 = ⟨ b N , Φ T ~ − p Φ ∗ b N ⟩ B \lVert h_N\rVert_{\widetilde{\mathcal{H}}^p}^{2}
=\lVert\mathcal{T}^{-p/2}\Phi^*b_N\rVert_{L^2(\pi)}^{2}
=\langle b_N,\Phi\widetilde{\mathcal{T}}^{-p}\Phi^*b_N\rangle_B ∥ h N ∥ H p 2 = ∥ T − p /2 Φ ∗ b N ∥ L 2 ( π ) 2 = ⟨ b N , Φ T − p Φ ∗ b N ⟩ B
= ( i ) ⟨ b N , T 1 − p b N ⟩ B = ∥ T − ( p − 1 ) / 2 b N ∥ B 2 = ∥ b N ∥ H p − 1 2 , \overset{(i)}{=}\langle b_N,\mathcal{T}^{1-p}b_N\rangle_B
=\lVert\mathcal{T}^{-(p-1)/2}b_N\rVert_B^{2}
=\lVert b_N\rVert_{\mathcal{H}^{p-1}}^{2}, = ( i ) ⟨ b N , T 1 − p b N ⟩ B = ∥ T − ( p − 1 ) /2 b N ∥ B 2 = ∥ b N ∥ H p − 1 2 ,
where ( i ) (i) ( i ) follows from the identity
Φ T ~ − p Φ ∗ e j = μ j 1 / 2 Φ T ~ − p e ~ j = μ j 1 / 2 μ j − p Φ e ~ j = μ j 1 − p e j = T 1 − p e j , \Phi\widetilde{\mathcal{T}}^{-p}\Phi^*e_j
=\mu_j^{1/2}\Phi\widetilde{\mathcal{T}}^{-p}\widetilde e_j
=\mu_j^{1/2}\mu_j^{-p}\Phi\widetilde e_j
=\mu_j^{1-p}e_j
=\mathcal{T}^{1-p}e_j, Φ T − p Φ ∗ e j = μ j 1/2 Φ T − p e j = μ j 1/2 μ j − p Φ e j = μ j 1 − p e j = T 1 − p e j ,
which holds for each j j j . Passing to the limit as N → ∞ N\to\infty N → ∞ proves (32) . □
Remark 3 Although Lemma 11 is stated for general s s s and p p p , the subsequent analyses of the approximation error in both the Sobolev and residual norms use only the case s = 2 s=2 s = 2 and p = 1 p=1 p = 1 .
The following lemma shows that the interpolation space H 1 \mathcal{H}^1 H 1 does not depend on the choice of the ambient space B B B .
Lemma 12 Let B 1 B_1 B 1 and B 2 B_2 B 2 be real Hilbert spaces continuously embedded in a common Hilbert space G G G , and identify them with their images in G G G . Suppose that the same feature map θ ↦ ϕ ( ⋅ , θ ) \theta\mapsto\phi(\cdot,\theta) θ ↦ ϕ ( ⋅ , θ ) belongs to L 2 ( π ; B i ) L^2(\pi;B_i) L 2 ( π ; B i ) for i = 1 , 2 i=1,2 i = 1 , 2 . Define Φ i ∗ : B i → L 2 ( π ) \Phi_i^*:B_i\to L^2(\pi) Φ i ∗ : B i → L 2 ( π ) by
( Φ i ∗ f ) ( θ ) = ⟨ ϕ ( ⋅ , θ ) , f ⟩ B i , (\Phi_i^*f)(\theta)=\langle\phi(\cdot,\theta),f\rangle_{B_i}, ( Φ i ∗ f ) ( θ ) = ⟨ ϕ ( ⋅ , θ ) , f ⟩ B i , and let Φ i : L 2 ( π ) → B i \Phi_i:L^2(\pi)\to B_i Φ i : L 2 ( π ) → B i be its adjoint. Let H i 1 \mathcal{H}_i^1 H i 1 be the interpolation space of order 1 1 1 associated with Φ i Φ i ∗ \Phi_i\Phi_i^* Φ i Φ i ∗ , as defined in (30) . Then
H i 1 = Range ( Φ i ) , ∥ f ∥ H i 1 = min α ∈ L 2 ( π ) Φ i α = f ∥ α ∥ L 2 ( π ) . \mathcal{H}_i^1=\operatorname{Range}(\Phi_i),\qquad
\lVert f\rVert_{\mathcal{H}_i^1}
=\min_{\substack{\alpha\in L^2(\pi)\\\Phi_i\alpha=f}}\lVert\alpha\rVert_{L^2(\pi)}. H i 1 = Range ( Φ i ) , ∥ f ∥ H i 1 = α ∈ L 2 ( π ) Φ i α = f min ∥ α ∥ L 2 ( π ) . Moreover, as subspaces of G G G ,
H 1 1 = H 2 1 , ∥ f ∥ H 1 1 = ∥ f ∥ H 2 1 . \mathcal{H}_1^1=\mathcal{H}_2^1,\qquad
\lVert f\rVert_{\mathcal{H}_1^1}=\lVert f\rVert_{\mathcal{H}_2^1}. H 1 1 = H 2 1 , ∥ f ∥ H 1 1 = ∥ f ∥ H 2 1 .
Proof. By the singular value decomposition of Φ i \Phi_i Φ i ,
H i 1 = Range ( Φ i ) , ∥ f ∥ H i 1 = min α ∈ L 2 ( π ) Φ i α = f ∥ α ∥ L 2 ( π ) , i = 1 , 2. (33) \mathcal{H}_i^1=\operatorname{Range}(\Phi_i),\qquad
\lVert f\rVert_{\mathcal{H}_i^1}
=\min_{\substack{\alpha\in L^2(\pi)\\\Phi_i\alpha=f}}\lVert\alpha\rVert_{L^2(\pi)},\qquad i=1,2. \tag*{(33)} H i 1 = Range ( Φ i ) , ∥ f ∥ H i 1 = α ∈ L 2 ( π ) Φ i α = f min ∥ α ∥ L 2 ( π ) , i = 1 , 2. (33)
Since the embeddings B i ↪ G B_i \hookrightarrow G B i ↪ G are continuous, the Bochner integral representations give, in G G G ,
Φ 1 α = ∫ Θ ϕ ( ⋅ , θ ) α ( θ ) d π ( θ ) = Φ 2 α for every α ∈ L 2 ( π ) . \Phi_1\alpha= \int_{\Theta} \phi(\cdot,\theta)\alpha(\theta)\,\mathrm{d}\pi(\theta) = \Phi_2\alpha\qquad\text{for every } \alpha\in L^2(\pi). Φ 1 α = ∫ Θ ϕ ( ⋅ , θ ) α ( θ ) d π ( θ ) = Φ 2 α for every α ∈ L 2 ( π ) .
Consequently,
H 1 1 = Range ( Φ 1 ) = Range ( Φ 2 ) = H 2 1 . \mathcal{H}_1^1 = \operatorname{Range}(\Phi_1) = \operatorname{Range}(\Phi_2) = \mathcal{H}_2^1. H 1 1 = Range ( Φ 1 ) = Range ( Φ 2 ) = H 2 1 .
For any f ∈ H 1 1 = H 2 1 f \in\mathcal{H}_1^1 = \mathcal{H}_2^1 f ∈ H 1 1 = H 2 1 ,
∥ f ∥ H 1 1 = min α ∈ L 2 ( π ) Φ 1 α = f ∥ α ∥ L 2 ( π ) = min α ∈ L 2 ( π ) Φ 2 α = f ∥ α ∥ L 2 ( π ) = ∥ f ∥ H 2 1 . \|f\|_{\mathcal{H}_1^1} = \min_{\substack{\alpha\in L^2(\pi)\\\Phi_1\alpha=f}}\|\alpha\|_{L^2(\pi)} = \min_{\substack{\alpha\in L^2(\pi)\\\Phi_2\alpha=f}}\|\alpha\|_{L^2(\pi)} = \|f\|_{\mathcal{H}_2^1}. ∥ f ∥ H 1 1 = α ∈ L 2 ( π ) Φ 1 α = f min ∥ α ∥ L 2 ( π ) = α ∈ L 2 ( π ) Φ 2 α = f min ∥ α ∥ L 2 ( π ) = ∥ f ∥ H 2 1 .
Remark 4 (Connection with RKHS) In the setting of Lemma 12 , suppose additionally that B 1 B_1 B 1 and B 2 B_2 B 2 consist of real-valued functions on X \mathcal{X} X . The operators Φ i \Phi_i Φ i , i = 1 , 2 i=1,2 i = 1 , 2 , are defined in the same way as Φ \Phi Φ in (12) . Moreover, the characterization of the interpolation space of order 1 1 1 in (33) coincides with that of the RKHS H k \mathcal{H}_k H k in (13) . Consequently,
H 1 1 = H k = H 2 1 , ∥ f ∥ H 1 1 = ∥ f ∥ H k = ∥ f ∥ H 2 1 . \mathcal{H}_1^1 = \mathcal{H}_k = \mathcal{H}_2^1,\qquad\|f\|_{\mathcal{H}_1^1} = \|f\|_{\mathcal{H}_k} = \|f\|_{\mathcal{H}_2^1}. H 1 1 = H k = H 2 1 , ∥ f ∥ H 1 1 = ∥ f ∥ H k = ∥ f ∥ H 2 1 .
Approximation Error Analysis
In this subsection, we analyze finite-feature approximation in Sobolev norms and the residual seminorm using the duality framework developed in Section 5.1. The key idea is to interpret feature selection as sampling on the parameter space: the adjoint feature operator maps features to kernel sections, while their inner products with the target become point evaluations of a transformed target. This correspondence yields a KRR representation of the regularized approximator and an exact identity between the corresponding errors. We first study the approximation error of the random feature model in Sobolev norms. For r ∈ N r \in\mathbb{N} r ∈ N and u ∗ ∈ H k u^* \in\mathcal{H}_k u ∗ ∈ H k , we consider the following linear least-squares problem
min u α ∈ Σ m ϕ ∥ u α − u ∗ ∥ H r ( Ω ) 2 . \min_{u_\alpha\in\Sigma_m^\phi}\|u_\alpha-u^*\|_{H^r(\Omega)}^2. u α ∈ Σ m ϕ min ∥ u α − u ∗ ∥ H r ( Ω ) 2 .
The above optimization problem is equivalent to the following finite-dimensional optimization problem over the coefficient vector α ∈ R m \alpha\in\mathbb{R}^m α ∈ R m :
min α ∈ R m ( α ⊤ K m α − 2 y ⊤ α + c ) , \min_{\alpha\in\mathbb{R}^m}\left(\alpha^\top K_m\alpha-2y^\top\alpha+c\right), α ∈ R m min ( α ⊤ K m α − 2 y ⊤ α + c ) ,
where c = ∥ u ∗ ∥ H r ( Ω ) 2 c=\|u^*\|_{H^r(\Omega)}^2 c = ∥ u ∗ ∥ H r ( Ω ) 2 and
K m = ( ⟨ ϕ ( ⋅ , θ i ) , ϕ ( ⋅ , θ j ) ⟩ H r ( Ω ) m p ( θ i ) p ( θ j ) ) i , j = 1 m , y = ( ⟨ ϕ ( ⋅ , θ i ) , u ∗ ( ⋅ ) ⟩ H r ( Ω ) m p ( θ i ) ) i = 1 m . (34) K_m=\left(\frac{\langle\phi(\cdot,\theta_i),\phi(\cdot,\theta_j)\rangle_{H^r(\Omega)}}{m\sqrt{p(\theta_i)p(\theta_j)}}\right)_{i,j=1}^m,\qquad
y=\left(\frac{\langle\phi(\cdot,\theta_i),u^*(\cdot)\rangle_{H^r(\Omega)}}{\sqrt{mp(\theta_i)}}\right)_{i=1}^m. \tag*{(34)} K m = ( m p ( θ i ) p ( θ j ) ⟨ ϕ ( ⋅ , θ i ) , ϕ ( ⋅ , θ j ) ⟩ H r ( Ω ) ) i , j = 1 m , y = ( m p ( θ i ) ⟨ ϕ ( ⋅ , θ i ) , u ∗ ( ⋅ ) ⟩ H r ( Ω ) ) i = 1 m . (34)
Since the Gram matrix K m K_m K m may be singular, the coefficient vector representing the best approximation may be non-unique. To obtain a unique coefficient representation and facilitate the subsequent error analysis, we introduce a regularization parameter λ > 0 \lambda> 0 λ > 0 and consider
u ^ m , λ ( r ) = argmin u α ∈ Σ m Φ ( ∥ u α − u ∗ ∥ H r ( Ω ) 2 + λ ∥ α ∥ 2 2 ) . (35) \widehat{u}^{(r)}_{m,\lambda} = \underset{u_\alpha\in\Sigma_m^\Phi}{\operatorname{argmin}}\left(\left\|u_\alpha-u^*\right\|_{H^r(\Omega)}^2+\lambda\|\alpha\|_2^2\right). \tag*{(35)} u m , λ ( r ) = u α ∈ Σ m Φ argmin ( ∥ u α − u ∗ ∥ H r ( Ω ) 2 + λ ∥ α ∥ 2 2 ) . (35)
Define ϕ ^ m : Ω → R m \widehat{\phi}_m:\Omega\to\mathbb{R}^m ϕ m : Ω → R m with ( ϕ ^ m ( ⋅ ) ) i = ϕ ( ⋅ , θ i ) m p ( θ i ) (\widehat{\phi}_m(\cdot))_i=\frac{\phi(\cdot,\theta_i)}{\sqrt{mp(\theta_i)}} ( ϕ m ( ⋅ ) ) i = m p ( θ i ) ϕ ( ⋅ , θ i ) . Then u ^ m , λ ( r ) = ϕ ^ m ( ⋅ ) T α ^ m , λ ( r ) \widehat{u}^{(r)}_{m,\lambda}=\widehat{\phi}_m(\cdot)^\mathsf{T}\widehat{\alpha}^{(r)}_{m,\lambda} u m , λ ( r ) = ϕ m ( ⋅ ) T α m , λ ( r ) , where
α ^ m , λ ( r ) = argmin α ∈ R m ( α T ( K m + λ I m ) α − 2 y T α + c ) = ( K m + λ I m ) − 1 y . (36) \widehat{\alpha}^{(r)}_{m,\lambda}=\underset{\alpha\in\mathbb{R}^m}{\operatorname{argmin}}\left(\alpha^\mathsf{T}(K_m+\lambda I_m)\alpha-2y^\mathsf{T}\alpha+c\right)=(K_m+\lambda I_m)^{-1}y. \tag*{(36)} α m , λ ( r ) = α ∈ R m argmin ( α T ( K m + λ I m ) α − 2 y T α + c ) = ( K m + λ I m ) − 1 y . (36)
We next express the approximant in operator form. Let S m , r : R m → H r ( Ω ) S_{m,r}:\mathbb{R}^m\to H^r(\Omega) S m , r : R m → H r ( Ω ) be the discrete operator and let S m , r ∗ : H r ( Ω ) → R m S_{m,r}^*:H^r(\Omega)\to\mathbb{R}^m S m , r ∗ : H r ( Ω ) → R m denote its adjoint. For a ∈ R m a\in\mathbb{R}^m a ∈ R m and f ∈ H r ( Ω ) f\in H^r(\Omega) f ∈ H r ( Ω ) , these operators are given by
S m , r a = 1 m ∑ i = 1 m a i p ( θ i ) ϕ ( ⋅ , θ i ) , ( S m , r ∗ f ) j = ⟨ ϕ ( ⋅ , θ j ) , f ( ⋅ ) ⟩ H r ( Ω ) m p ( θ j ) , j = 1 , … , m . (37) S_{m,r}a=\frac{1}{\sqrt{m}}\sum_{i=1}^{m}\frac{a_i}{\sqrt{p(\theta_i)}}\phi(\cdot,\theta_i),\qquad(S_{m,r}^*f)_j=\frac{\langle\phi(\cdot,\theta_j),f(\cdot)\rangle_{H^r(\Omega)}}{\sqrt{mp(\theta_j)}},\quad j=1,\ldots,m. \tag*{(37)} S m , r a = m 1 i = 1 ∑ m p ( θ i ) a i ϕ ( ⋅ , θ i ) , ( S m , r ∗ f ) j = m p ( θ j ) ⟨ ϕ ( ⋅ , θ j ) , f ( ⋅ ) ⟩ H r ( Ω ) , j = 1 , … , m . (37)
One can easily verify that
u ^ m , λ ( r ) = S m , r ( S m , r ∗ S m , r + λ I m ) − 1 S m , r ∗ u ∗ . (38) \widehat{u}^{(r)}_{m,\lambda}=S_{m,r}(S_{m,r}^*S_{m,r}+\lambda I_m)^{-1}S_{m,r}^*u^*. \tag*{(38)} u m , λ ( r ) = S m , r ( S m , r ∗ S m , r + λ I m ) − 1 S m , r ∗ u ∗ . (38)
The corresponding continuous operator Φ r : L 2 ( π ) → H r ( Ω ) \Phi_r:L^2(\pi)\to H^r(\Omega) Φ r : L 2 ( π ) → H r ( Ω ) and its adjoint Φ r ∗ \Phi_r^* Φ r ∗ are obtained by specializing (27) and (28) , respectively, to B = H r ( Ω ) B=H^r(\Omega) B = H r ( Ω ) . Explicitly, for α ∈ L 2 ( π ) \alpha\in L^2(\pi) α ∈ L 2 ( π ) and f ∈ H r ( Ω ) f\in H^r(\Omega) f ∈ H r ( Ω ) ,
( Φ r α ) ( x ) = ⟨ ϕ ( x , ⋅ ) , α ⟩ L 2 ( π ) , ( Φ r ∗ f ) ( θ ) = ⟨ ϕ ( ⋅ , θ ) , f ⟩ H r ( Ω ) . (39) (\Phi_r\alpha)(x)=\langle\phi(x,\cdot),\alpha\rangle_{L^2(\pi)},\qquad(\Phi_r^*f)(\theta)=\langle\phi(\cdot,\theta),f\rangle_{H^r(\Omega)}. \tag*{(39)} ( Φ r α ) ( x ) = ⟨ ϕ ( x , ⋅ ) , α ⟩ L 2 ( π ) , ( Φ r ∗ f ) ( θ ) = ⟨ ϕ ( ⋅ , θ ) , f ⟩ H r ( Ω ) . (39)
To connect this formulation with kernel ridge regression (KRR), we introduce the kernel on the parameter space Θ \Theta Θ :
k ~ r ( θ , θ ′ ) = ⟨ ϕ ( ⋅ , θ ) , ϕ ( ⋅ , θ ′ ) ⟩ H r ( Ω ) . (40) \widetilde{k}_r(\theta,\theta')=\langle\phi(\cdot,\theta),\phi(\cdot,\theta')\rangle_{H^r(\Omega)}. \tag*{(40)} k r ( θ , θ ′ ) = ⟨ ϕ ( ⋅ , θ ) , ϕ ( ⋅ , θ ′ ) ⟩ H r ( Ω ) . (40)
The corresponding integral operator T ~ r \widetilde{T}_r T r is given by
T ~ r g = ∫ Θ k ~ r ( ⋅ , θ ′ ) g ( θ ′ ) d π ( θ ′ ) = Φ r ∗ Φ r g . (41) \widetilde{T}_r g=\int_\Theta\widetilde{k}_r(\cdot,\theta')g(\theta')\,\mathrm{d}\pi(\theta')=\Phi_r^*\Phi_r g. \tag*{(41)} T r g = ∫ Θ k r ( ⋅ , θ ′ ) g ( θ ′ ) d π ( θ ′ ) = Φ r ∗ Φ r g . (41)
Its spectral decomposition is
T ~ r = ∑ j = 1 ∞ μ ~ j ( r ) e ~ j ( r ) ⊗ e ~ j ( r ) . (42) \widetilde{T}_r=\sum_{j=1}^{\infty}\widetilde{\mu}^{(r)}_j\widetilde{e}^{(r)}_j\otimes\widetilde{e}^{(r)}_j. \tag*{(42)} T r = j = 1 ∑ ∞ μ j ( r ) e j ( r ) ⊗ e j ( r ) . (42)
where { μ ~ j ( r ) } j = 1 ∞ \{\widetilde{\mu}^{(r)}_j\}_{j=1}^{\infty} { μ j ( r ) } j = 1 ∞ are the eigenvalues in a decreasing order and { e ~ j ( r ) } j = 1 ∞ \{\widetilde{e}^{(r)}_j\}_{j=1}^{\infty} { e j ( r ) } j = 1 ∞ are the corresponding orthonormal eigenfunctions in L 2 ( π ) L^2(\pi) L 2 ( π ) . Let H ( r ) s \mathcal{H}_{(r)}^s H ( r ) s and H ~ ( r ) s \widetilde{\mathcal{H}}_{(r)}^s H ( r ) s denote the interpolation spaces defined in (30) and (31) , respectively, with B = H r ( Ω ) B=H^r(\Omega) B = H r ( Ω ) . The RKHS associated with k ~ r \widetilde{k}_r k r is isometrically identified with H ~ ( r ) 1 \widetilde{\mathcal{H}}_{(r)}^1 H ( r ) 1 .
Now consider the following KRR on Θ \Theta Θ with input points { θ i } i = 1 m ∼ i . i . d . π ′ \{\theta_i\}_{i=1}^m \mathrel{\mathop{\sim}^{\mathrm{i.i.d.}}} \pi' { θ i } i = 1 m ∼ i.i.d. π ′ and target function g ∗ ( θ ) = ⟨ ϕ ( ⋅ , θ ) , u ∗ ( ⋅ ) ⟩ H r ( Ω ) = ( Φ r ∗ u ∗ ) ( θ ) g^*(\theta)=\langle\phi(\cdot,\theta),u^*(\cdot)\rangle_{H^r(\Omega)}=(\Phi_r^*u^*)(\theta) g ∗ ( θ ) = ⟨ ϕ ( ⋅ , θ ) , u ∗ ( ⋅ ) ⟩ H r ( Ω ) = ( Φ r ∗ u ∗ ) ( θ ) . It follows that g ∗ ∈ H ~ ( r ) 1 g^*\in\widetilde{\mathcal{H}}_{(r)}^1 g ∗ ∈ H ( r ) 1 . The KRR estimator is obtained by solving:
g ^ m , λ ( r ) = arg min g ∈ H ~ ( r ) 1 ( 1 m ∑ i = 1 m ( g ( θ i ) − g ∗ ( θ i ) ) 2 p ( θ i ) + λ ∥ g ∥ H ~ ( r ) 1 2 ) . (43) \widehat{g}^{(r)}_{m,\lambda}=\underset{g\in\widetilde{\mathcal{H}}_{(r)}^1}{\arg\min}\left(\frac{1}{m}\sum_{i=1}^{m}\frac{\left(g(\theta_i)-g^*(\theta_i)\right)^2}{p(\theta_i)}+\lambda\lVert g\rVert_{\widetilde{\mathcal{H}}_{(r)}^1}^2\right). \tag*{(43)} g m , λ ( r ) = g ∈ H ( r ) 1 arg min ( m 1 i = 1 ∑ m p ( θ i ) ( g ( θ i ) − g ∗ ( θ i ) ) 2 + λ ∥ g ∥ H ( r ) 1 2 ) . (43)
Define k ^ m : Θ → R m \widehat{k}_m:\Theta\to\mathbb{R}^m k m : Θ → R m and K ~ m ∈ R m × m \widetilde{K}_m\in\mathbb{R}^{m\times m} K m ∈ R m × m by
( k ^ m ( ⋅ ) ) i = k ~ r ( θ i , ⋅ ) m p ( θ i ) , K ~ m = ( k ~ r ( θ i , θ j ) m p ( θ i ) p ( θ j ) ) i , j = 1 m . \left(\widehat{k}_m(\cdot)\right)_i=\frac{\widetilde{k}_r(\theta_i,\cdot)}{\sqrt{m p(\theta_i)}},\qquad\widetilde{K}_m=\left(\frac{\widetilde{k}_r(\theta_i,\theta_j)}{m\sqrt{p(\theta_i)p(\theta_j)}}\right)_{i,j=1}^{m}. ( k m ( ⋅ ) ) i = m p ( θ i ) k r ( θ i , ⋅ ) , K m = ( m p ( θ i ) p ( θ j ) k r ( θ i , θ j ) ) i , j = 1 m .
The KRR solution can then be written as g ^ m , λ ( r ) = k ^ m ( ⋅ ) T b ^ m , λ ( r ) \widehat{g}^{(r)}_{m,\lambda}=\widehat{k}_m(\cdot)^\mathsf{T}\widehat{b}^{(r)}_{m,\lambda} g m , λ ( r ) = k m ( ⋅ ) T b m , λ ( r ) with
b ^ m , λ ( r ) = ( K ~ m + λ I m ) − 1 y , (44) \widehat{b}^{(r)}_{m,\lambda}=(\widetilde{K}_m+\lambda I_m)^{-1}y, \tag*{(44)} b m , λ ( r ) = ( K m + λ I m ) − 1 y , (44)
where y y y is defined in (34) . By (40) , K ~ m = K m \widetilde{K}_m=K_m K m = K m , so b ^ m , λ ( r ) = α ^ m , λ ( r ) \widehat{b}^{(r)}_{m,\lambda}=\widehat{\alpha}^{(r)}_{m,\lambda} b m , λ ( r ) = α m , λ ( r ) . Moreover, for any b ^ ∈ R m \widehat{b}\in\mathbb{R}^m b ∈ R m ,
k ^ m ( θ ) T b ^ = 1 m ∑ i = 1 m b ^ i p ( θ i ) ⟨ ϕ ( ⋅ , θ ) , ϕ ( ⋅ , θ i ) ⟩ H r ( Ω ) = ⟨ ϕ ( ⋅ , θ ) , 1 m ∑ i = 1 m b ^ i p ( θ i ) ϕ ( ⋅ , θ i ) ⟩ H r ( Ω ) = ⟨ ϕ ( ⋅ , θ ) , S m , r b ^ ⟩ H r ( Ω ) = ( Φ r ∗ S m , r b ^ ) ( θ ) . \begin{aligned}
\widehat{k}_m(\theta)^\mathsf{T}\widehat{b}
&=\frac{1}{\sqrt{m}}\sum_{i=1}^{m}\frac{\widehat{b}_i}{\sqrt{p(\theta_i)}}\langle\phi(\cdot,\theta),\phi(\cdot,\theta_i)\rangle_{H^r(\Omega)}\\
&=\left\langle\phi(\cdot,\theta),\frac{1}{\sqrt{m}}\sum_{i=1}^{m}\frac{\widehat{b}_i}{\sqrt{p(\theta_i)}}\phi(\cdot,\theta_i)\right\rangle_{H^r(\Omega)}\\
&=\langle\phi(\cdot,\theta),\mathcal{S}_{m,r}\widehat{b}\rangle_{H^r(\Omega)}
=(\Phi_r^*\mathcal{S}_{m,r}\widehat{b})(\theta).
\end{aligned} k m ( θ ) T b = m 1 i = 1 ∑ m p ( θ i ) b i ⟨ ϕ ( ⋅ , θ ) , ϕ ( ⋅ , θ i ) ⟩ H r ( Ω ) = ⟨ ϕ ( ⋅ , θ ) , m 1 i = 1 ∑ m p ( θ i ) b i ϕ ( ⋅ , θ i ) ⟩ H r ( Ω ) = ⟨ ϕ ( ⋅ , θ ) , S m , r b ⟩ H r ( Ω ) = ( Φ r ∗ S m , r b ) ( θ ) .
Taking b ^ = b ^ m , λ ( r ) \widehat{b}=\widehat{b}^{(r)}_{m,\lambda} b = b m , λ ( r ) yields
g ^ m , λ ( r ) = Φ r ∗ S m , r ( S m , r ∗ S m , r + λ I m ) − 1 S m , r ∗ u ∗ = Φ r ∗ u ^ m , λ ( r ) . (45) \widehat{g}^{(r)}_{m,\lambda}=\Phi_r^*\mathcal{S}_{m,r}(\mathcal{S}_{m,r}^*\mathcal{S}_{m,r}+\lambda I_m)^{-1}\mathcal{S}_{m,r}^*u^*=\Phi_r^*\widehat{u}^{(r)}_{m,\lambda}. \tag*{(45)} g m , λ ( r ) = Φ r ∗ S m , r ( S m , r ∗ S m , r + λ I m ) − 1 S m , r ∗ u ∗ = Φ r ∗ u m , λ ( r ) . (45)
Let N ( r ) N^{(r)} N ( r ) and F q ( r ) F_q^{(r)} F q ( r ) denote the DoFs obtained from (4) by replacing ( μ j , e j ) (\mu_j,e_j) ( μ j , e j ) with ( μ ~ j ( r ) , e ~ j ( r ) ) (\widetilde{\mu}^{(r)}_j,\widetilde{e}^{(r)}_j) ( μ j ( r ) , e j ( r ) ) and taking π \pi π as the reference measure on Θ \Theta Θ . We now combine the duality identity with the KRR error bound in [14 Theorem 4.3] to prove Theorem 1.
Proof of Theorem 1. By Lemma 20, N ( r ) ( μ ~ k ( r ) ) ≤ C k N^{(r)}(\tilde{\mu}^{(r)}_k) \le Ck N ( r ) ( μ ~ k ( r ) ) ≤ C k for all k ≥ 1 k \ge1 k ≥ 1 . Choosing c 0 ≤ ( 20 C ) − 1 c_0 \le(20C)^{-1} c 0 ≤ ( 20 C ) − 1 and m 0 m_0 m 0 sufficiently large so that c 0 m 0 ≥ 1 c_0m_0 \ge1 c 0 m 0 ≥ 1 , we obtain
N ( r ) ( μ ~ ⌊ c 0 t ⌋ ( r ) ) ≤ C ⌊ c 0 t ⌋ ≤ t 20 , t ≥ m 0 . N^{(r)}(\tilde{\mu}^{(r)}_{\lfloor c_0t\rfloor}) \le C\lfloor c_0t\rfloor\le\frac{t}{20}, \qquad t \ge m_0. N ( r ) ( μ ~ ⌊ c 0 t ⌋ ( r ) ) ≤ C ⌊ c 0 t ⌋ ≤ 20 t , t ≥ m 0 .
Write L m : = log ( 14 m / δ ) L_m := \log(14m/\delta) L m := log ( 14 m / δ ) and λ m : = μ ~ ⌊ c 0 m / L m ⌋ ( r ) \lambda_m := \tilde{\mu}^{(r)}_{\lfloor c_0m/L_m\rfloor} λ m := μ ~ ⌊ c 0 m / L m ⌋ ( r ) . For m / L m ≥ m 0 m/L_m \ge m_0 m / L m ≥ m 0 ,
N ( r ) ( λ m ) = N ( r ) ( μ ~ ⌊ c 0 m / L m ⌋ ( r ) ) ≤ m 20 L m . N^{(r)}(\lambda_m) = N^{(r)}\left(\tilde{\mu}^{(r)}_{\lfloor c_0m/L_m\rfloor}\right) \le\frac{m}{20L_m}. N ( r ) ( λ m ) = N ( r ) ( μ ~ ⌊ c 0 m / L m ⌋ ( r ) ) ≤ 20 L m m .
By [6 ] , we have
F p m ( r ) ( λ m ) ≤ 2 N ( r ) ( λ m ) ≤ m 10 L m . F^{(r)}_{p_m}(\lambda_m) \le2N^{(r)}(\lambda_m) \le\frac{m}{10L_m}. F p m ( r ) ( λ m ) ≤ 2 N ( r ) ( λ m ) ≤ 10 L m m .
Since L m > 1 L_m > 1 L m > 1 , it follows that F p m ( r ) ( λ m ) ≤ m F^{(r)}_{p_m}(\lambda_m) \le m F p m ( r ) ( λ m ) ≤ m and
5 F p m ( r ) ( λ m ) max { 1 , log 14 F p m ( r ) ( λ m ) δ } ≤ 5 F p m ( r ) ( λ m ) L m ≤ m 2 < m . 5F^{(r)}_{p_m}(\lambda_m)\max\left\{1,\log\frac{14F^{(r)}_{p_m}(\lambda_m)}{\delta}\right\} \le5F^{(r)}_{p_m}(\lambda_m)L_m \le\frac{m}{2} < m. 5 F p m ( r ) ( λ m ) max { 1 , log δ 14 F p m ( r ) ( λ m ) } ≤ 5 F p m ( r ) ( λ m ) L m ≤ 2 m < m .
Thus the critical penalty λ m c r i t \lambda_m^{\mathrm{crit}} λ m crit associated with p m p_m p m , as defined in [14 Definition 4.2] , satisfies
λ m c r i t ≤ λ m . \lambda_m^{\mathrm{crit}} \le\lambda_m. λ m crit ≤ λ m .
Fix λ ∈ ( 0 , λ m c r i t ] \lambda\in(0,\lambda_m^{\mathrm{crit}}] λ ∈ ( 0 , λ m crit ] . Applying [14 Theorem 4.3] with s = 2 s=2 s = 2 and p = 1 p=1 p = 1 yields, with probability at least 1 − δ 1-\delta 1 − δ ,
sup ∥ g ∗ ∥ H r ~ ≤ 1 ∥ g ^ m , λ − g ∗ ∥ H r ~ 2 ≤ 16 λ m c r i t . \sup_{\|g^*\|_{\mathcal{H}^{\tilde{}}_r}\le1}\|\hat{g}_{m,\lambda}-g^*\|^2_{\mathcal{H}^{\tilde{}}_r} \le16\lambda_m^{\mathrm{crit}}. ∥ g ∗ ∥ H r ~ ≤ 1 sup ∥ g ^ m , λ − g ∗ ∥ H r ~ 2 ≤ 16 λ m crit .
Since g ∗ = Φ r ∗ u ∗ g^*=\Phi_r^*u^* g ∗ = Φ r ∗ u ∗ and g ^ m , λ ( r ) = Φ r ∗ u ^ m , λ ( r ) \hat{g}^{(r)}_{m,\lambda}=\Phi_r^*\hat{u}^{(r)}_{m,\lambda} g ^ m , λ ( r ) = Φ r ∗ u ^ m , λ ( r ) , Lemma 11 gives
sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ H m ∥ u − u ∗ ∥ H r ( Ω ) 2 ≤ sup ∥ u ∗ ∥ H k ≤ 1 ∥ u ^ m , λ ( r ) − u ∗ ∥ H r ( Ω ) 2 = sup ∥ g ∗ ∥ H r ~ ≤ 1 ∥ g ^ m , λ ( r ) − g ∗ ∥ H r ~ 2 ≤ 16 λ m c r i t ≤ 16 λ m = 16 μ ~ ⌊ c 0 m / L m ⌋ ( r ) . \begin{aligned}
\sup_{\|u^*\|_{\mathcal{H}_k}\le1}\inf_{u\in\mathcal{H}_m}\|u-u^*\|^2_{H^r(\Omega)}
\le\sup_{\|u^*\|_{\mathcal{H}_k}\le1}\|\hat{u}^{(r)}_{m,\lambda}-u^*\|^2_{H^r(\Omega)} \\
&= \sup_{\|g^*\|_{\mathcal{H}^{\tilde{}}_r}\le1}\|\hat{g}^{(r)}_{m,\lambda}-g^*\|^2_{\mathcal{H}^{\tilde{}}_r}
\le16\lambda_m^{\mathrm{crit}} \le16\lambda_m = 16\tilde{\mu}^{(r)}_{\lfloor c_0m/L_m\rfloor}.
\end{aligned} ∥ u ∗ ∥ H k ≤ 1 sup u ∈ H m inf ∥ u − u ∗ ∥ H r ( Ω ) 2 ≤ ∥ u ∗ ∥ H k ≤ 1 sup ∥ u ^ m , λ ( r ) − u ∗ ∥ H r ( Ω ) 2 = ∥ g ∗ ∥ H r ~ ≤ 1 sup ∥ g ^ m , λ ( r ) − g ∗ ∥ H r ~ 2 ≤ 16 λ m crit ≤ 16 λ m = 16 μ ~ ⌊ c 0 m / L m ⌋ ( r ) .
Remark 5 (Direct and dual representations of the approximation error) Let u ∗ = Φ r b ∗ u^*=\Phi_rb^* u ∗ = Φ r b ∗ with b ∗ ∈ L 2 ( π ) b^*\in L^2(\pi) b ∗ ∈ L 2 ( π ) , and set
T m , r : = S m , r S m , r ∗ , R m , λ : = ( T m , r + λ I ) − 1 . T_{m,r}:=S_{m,r}S^*_{m,r}, \qquad R_{m,\lambda}:=(T_{m,r}+\lambda I)^{-1}. T m , r := S m , r S m , r ∗ , R m , λ := ( T m , r + λ I ) − 1 .
By (38) , the direct approximation error is u ^ m , λ ( r ) − u ∗ = − λ R m , λ u ∗ \hat{u}^{(r)}_{m,\lambda}-u^*=-\lambda R_{m,\lambda}u^* u ^ m , λ ( r ) − u ∗ = − λ R m , λ u ∗ . Since R m , λ R_{m,\lambda} R m , λ is self-adjoint on H r ( Ω ) H^r(\Omega) H r ( Ω ) , the operator A m , λ : = − λ R m , λ Φ r A_{m,\lambda}:=-\lambda R_{m,\lambda}\Phi_r A m , λ := − λ R m , λ Φ r from L 2 ( π ) L^2(\pi) L 2 ( π ) to H r ( Ω ) H^r(\Omega) H r ( Ω ) has adjoint A m , λ ∗ = − λ Φ r ∗ R m , λ A^*_{m,\lambda}=-\lambda\Phi_r^*R_{m,\lambda} A m , λ ∗ = − λ Φ r ∗ R m , λ . Thus, by (45) and g ∗ = Φ r ∗ u ∗ g^*=\Phi_r^*u^* g ∗ = Φ r ∗ u ∗ ,
u ^ m , λ ( r ) − u ∗ = A m , λ b ∗ , g ^ m , λ ( r ) − g ∗ = Φ r ∗ ( u ^ m , λ ( r ) − u ∗ ) = A m , λ ∗ u ∗ . \hat{u}^{(r)}_{m,\lambda}-u^*=A_{m,\lambda}b^*, \qquad\hat{g}^{(r)}_{m,\lambda}-g^*=\Phi_r^*(\hat{u}^{(r)}_{m,\lambda}-u^*)=A^*_{m,\lambda}u^*. u ^ m , λ ( r ) − u ∗ = A m , λ b ∗ , g ^ m , λ ( r ) − g ∗ = Φ r ∗ ( u ^ m , λ ( r ) − u ∗ ) = A m , λ ∗ u ∗ .
By Lemma 11 , these two representations give the same error in their respective norms,
∥ u ^ m , λ ( r ) − u ∗ ∥ H r ( Ω ) = ∥ g ^ m , λ ( r ) − g ∗ ∥ H r ~ . \|\hat{u}^{(r)}_{m,\lambda}-u^*\|_{H^r(\Omega)}=\|\hat{g}^{(r)}_{m,\lambda}-g^*\|_{\mathcal{H}^{\tilde{}}_r}. ∥ u ^ m , λ ( r ) − u ∗ ∥ H r ( Ω ) = ∥ g ^ m , λ ( r ) − g ∗ ∥ H r ~ .
This identity also gives an operator-norm interpretation of the worst-case error. We have
∥ g ^ m , λ ( r ) − g ∗ ∥ H r ~ = ∥ A m , λ b ∗ ∥ H r ( Ω ) = sup ∥ v ∥ H r ( Ω ) ≤ 1 ∣ ⟨ b ∗ , A m , λ ∗ v ⟩ L 2 ( π ) ∣ . \|\hat{g}^{(r)}_{m,\lambda}-g^*\|_{\mathcal{H}^{\tilde{}}_r}
=\|A_{m,\lambda}b^*\|_{H^r(\Omega)}
=\sup_{\|v\|_{H^r(\Omega)}\le1}\left|\langle b^*,A^*_{m,\lambda}v\rangle_{L^2(\pi)}\right|. ∥ g ^ m , λ ( r ) − g ∗ ∥ H r ~ = ∥ A m , λ b ∗ ∥ H r ( Ω ) = ∥ v ∥ H r ( Ω ) ≤ 1 sup ⟨ b ∗ , A m , λ ∗ v ⟩ L 2 ( π ) .
By the feature representation of the RKHS unit ball, taking the supremum over ∥ u ∗ ∥ H k ≤ 1 \|u^*\|_{\mathcal{H}_k} \le1 ∥ u ∗ ∥ H k ≤ 1 is equivalent to taking the supremum over ∥ b ∗ ∥ L 2 ( π ) ≤ 1 \|b^*\|_{L^2(\pi)} \le1 ∥ b ∗ ∥ L 2 ( π ) ≤ 1 . Consequently,
sup ∥ u ∗ ∥ H k ≤ 1 ∥ u ^ m , λ ( r ) − u ∗ ∥ H r ( Ω ) = sup ∥ b ∥ L 2 ( π ) ≤ 1 ∥ A m , λ b ∥ H r ( Ω ) = ∥ A m , λ ∥ L 2 ( π ) → H r ( Ω ) . \sup_{\|u^*\|_{\mathcal{H}_k}\le1}\|\hat{u}^{(r)}_{m,\lambda}-u^*\|_{H^r(\Omega)}
=
\sup_{\|b\|_{L^2(\pi)}\le1}\|A_{m,\lambda}b\|_{H^r(\Omega)}
=
\|A_{m,\lambda}\|_{L^2(\pi)\to H^r(\Omega)}. ∥ u ∗ ∥ H k ≤ 1 sup ∥ u ^ m , λ ( r ) − u ∗ ∥ H r ( Ω ) = ∥ b ∥ L 2 ( π ) ≤ 1 sup ∥ A m , λ b ∥ H r ( Ω ) = ∥ A m , λ ∥ L 2 ( π ) → H r ( Ω ) .
sup ∥ g ∗ ∥ H ~ ( r ) 1 ≤ 1 ∥ g ^ m , λ ( r ) − g ∗ ∥ H ~ ( r ) 1 = sup ∥ b ∥ L 2 ( π ) ≤ 1 ∥ v ∥ H r ( Ω ) ≤ 1 ∣ ⟨ b , A m , λ ∗ v ⟩ L 2 ( π ) ∣ = sup ∥ v ∥ H r ( Ω ) ≤ 1 ∥ A m , λ ∗ v ∥ L 2 ( π ) = ∥ A m , λ ∗ ∥ H r ( Ω ) → L 2 ( π ) . \sup_{\|g^*\|_{\widetilde{\mathcal{H}}^1_{(r)}}\le1}\|\hat{g}^{(r)}_{m,\lambda}-g^*\|_{\widetilde{\mathcal{H}}^1_{(r)}}
=
\sup_{\substack{\|b\|_{L^2(\pi)}\le1\\\|v\|_{H^r(\Omega)}\le1}}
|\langle b,A^*_{m,\lambda}v\rangle_{L^2(\pi)}|
=
\sup_{\|v\|_{H^r(\Omega)}\le1}\|A^*_{m,\lambda}v\|_{L^2(\pi)}
=
\|A^*_{m,\lambda}\|_{H^r(\Omega)\to L^2(\pi)}. ∥ g ∗ ∥ H ( r ) 1 ≤ 1 sup ∥ g ^ m , λ ( r ) − g ∗ ∥ H ( r ) 1 = ∥ b ∥ L 2 ( π ) ≤ 1 ∥ v ∥ H r ( Ω ) ≤ 1 sup ∣ ⟨ b , A m , λ ∗ v ⟩ L 2 ( π ) ∣ = ∥ v ∥ H r ( Ω ) ≤ 1 sup ∥ A m , λ ∗ v ∥ L 2 ( π ) = ∥ A m , λ ∗ ∥ H r ( Ω ) → L 2 ( π ) .
Since ∥ A m , λ ∗ ∥ = ∥ A m , λ ∥ \|A^*_{m,\lambda}\|=\|A_{m,\lambda}\| ∥ A m , λ ∗ ∥ = ∥ A m , λ ∥ , the bound on ∥ A m , λ ∗ ∥ \|A^*_{m,\lambda}\| ∥ A m , λ ∗ ∥ which is established in KRR analysis also bounds the operator norm that governs the direct approximation error. Thus, both analyses control the same worst-case error and yield the same convergence rate under the stated sampling assumptions. We use the dual representation because it places finite-feature approximation and collocation estimation in a common noiseless KRR framework.
We next prove the lower bound on the worst-case Sobolev approximation error stated in Theorem 2 .
Proof of Theorem 2 . Define
V m : = span { k ~ r ( ⋅ , θ i ) : i = 1 , … , m } ⊂ H ~ ( r ) 1 . V_m:=\operatorname{span}\{\tilde{k}_r(\cdot,\theta_i):i=1,\ldots,m\}\subset\widetilde{\mathcal{H}}^1_{(r)}. V m := span { k ~ r ( ⋅ , θ i ) : i = 1 , … , m } ⊂ H ( r ) 1 .
Since for any θ ∈ Θ \theta\in\Theta θ ∈ Θ ,
Φ r ∗ ( ϕ ( ⋅ , θ i ) ) ( θ ) = k ~ r ( θ , θ i ) , \Phi_r^*(\phi(\cdot,\theta_i))(\theta)=\tilde{k}_r(\theta,\theta_i), Φ r ∗ ( ϕ ( ⋅ , θ i )) ( θ ) = k ~ r ( θ , θ i ) ,
we have Φ m ∗ Σ m ϕ = V m \Phi_m^*\Sigma_m^\phi=V_m Φ m ∗ Σ m ϕ = V m . By Lemma 11 ,
sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ Σ m ϕ ∥ u − u ∗ ∥ H r ( Ω ) 2 = sup ∥ g ∗ ∥ H ~ ( r ) 2 ≤ 1 inf h ∈ V m ∥ h − g ∗ ∥ H ~ ( r ) 1 2 . \sup_{\|u^*\|_{\mathcal{H}_k}\le1}\inf_{u\in\Sigma_m^\phi}\|u-u^*\|_{H^r(\Omega)}^2
=
\sup_{\|g^*\|_{\widetilde{\mathcal{H}}^2_{(r)}}\le1}\inf_{h\in V_m}\|h-g^*\|_{\widetilde{\mathcal{H}}^1_{(r)}}^2. ∥ u ∗ ∥ H k ≤ 1 sup u ∈ Σ m ϕ inf ∥ u − u ∗ ∥ H r ( Ω ) 2 = ∥ g ∗ ∥ H ( r ) 2 ≤ 1 sup h ∈ V m inf ∥ h − g ∗ ∥ H ( r ) 1 2 .
Let
S m : = ( θ 1 , … , θ m ) , g ( S m ) : = ( g ( θ 1 ) , … , g ( θ m ) ) ⊤ , S_m:=(\theta_1,\ldots,\theta_m),\qquad g(S_m):=(g(\theta_1),\ldots,g(\theta_m))^\top, S m := ( θ 1 , … , θ m ) , g ( S m ) := ( g ( θ 1 ) , … , g ( θ m ) ) ⊤ ,
and define
M m : = { Q : Q g = G ( S m , g ( S m ) ) , G : Θ m × R m → R Θ } . \mathcal{M}_m:=\left\{Q:Q g=G(S_m,g(S_m)),\ G:\Theta^m\times\mathbb{R}^m\to\mathbb{R}^{\Theta}\right\}. M m := { Q : Q g = G ( S m , g ( S m )) , G : Θ m × R m → R Θ } .
Let Q 0 Q_0 Q 0 denote the orthogonal projection onto V m V_m V m in H ~ ( r ) 1 \widetilde{\mathcal{H}}^1_{(r)} H ( r ) 1 . Then
sup ∥ g ∗ ∥ H ~ ( r ) 2 ≤ 1 inf h ∈ V m ∥ h − g ∗ ∥ H ~ ( r ) 1 2 = sup ∥ g ∗ ∥ H ~ ( r ) 2 ≤ 1 ∥ Q 0 g ∗ − g ∗ ∥ H ~ ( r ) 1 2 . \sup_{\|g^*\|_{\widetilde{\mathcal{H}}^2_{(r)}}\le1}\inf_{h\in V_m}\|h-g^*\|_{\widetilde{\mathcal{H}}^1_{(r)}}^2
=
\sup_{\|g^*\|_{\widetilde{\mathcal{H}}^2_{(r)}}\le1}\|Q_0g^*-g^*\|_{\widetilde{\mathcal{H}}^1_{(r)}}^2. ∥ g ∗ ∥ H ( r ) 2 ≤ 1 sup h ∈ V m inf ∥ h − g ∗ ∥ H ( r ) 1 2 = ∥ g ∗ ∥ H ( r ) 2 ≤ 1 sup ∥ Q 0 g ∗ − g ∗ ∥ H ( r ) 1 2 .
By the reproducing property, Q 0 g ∗ Q_0g^* Q 0 g ∗ depends only on S m S_m S m and g ∗ ( S m ) g^*(S_m) g ∗ ( S m ) , so Q 0 ∈ M m Q_0\in\mathcal{M}_m Q 0 ∈ M m . Then
sup ∥ g ∗ ∥ H ~ ( r ) 2 ≤ 1 ∥ Q 0 g ∗ − g ∗ ∥ H ~ ( r ) 1 2 ≥ inf Q ∈ M m sup ∥ g ∗ ∥ H ~ ( r ) 2 ≤ 1 ∥ Q g ∗ − g ∗ ∥ H ~ ( r ) 1 2 ≥ μ ~ m + 1 ( r ) , \sup_{\|g^*\|_{\widetilde{\mathcal{H}}^2_{(r)}}\le1}\|Q_0g^*-g^*\|_{\widetilde{\mathcal{H}}^1_{(r)}}^2
\ge
\inf_{Q\in\mathcal{M}_m}\sup_{\|g^*\|_{\widetilde{\mathcal{H}}^2_{(r)}}\le1}\|Qg^*-g^*\|_{\widetilde{\mathcal{H}}^1_{(r)}}^2
\ge\tilde{\mu}^{(r)}_{m+1}, ∥ g ∗ ∥ H ( r ) 2 ≤ 1 sup ∥ Q 0 g ∗ − g ∗ ∥ H ( r ) 1 2 ≥ Q ∈ M m inf ∥ g ∗ ∥ H ( r ) 2 ≤ 1 sup ∥ Q g ∗ − g ∗ ∥ H ( r ) 1 2 ≥ μ ~ m + 1 ( r ) ,
where the last inequality follows from the case s = 2 s=2 s = 2 , p = 1 p=1 p = 1 of [14 Theorem 4.1] . □
Remark 6 (A geometric interpretation of the Sobolev approximation lower bounds) Write μ j = μ ~ j ( r ) \mu_j=\tilde{\mu}_j^{(r)} μ j = μ ~ j ( r ) . In the equivalent KRR problem on the parameter space, the target unit ball, expressed in coordinates orthonormal in the error space H ~ ( r ) 1 \widetilde{H}_{(r)}^1 H ( r ) 1 , is the ellipsoid
E : = { ∑ j ≥ 1 μ j b j ψ j : ∑ j ≥ 1 ∣ b j ∣ 2 ≤ 1 } . \mathcal{E}:=\left\{\sum_{j\ge1}\sqrt{\mu_j}b_j\psi_j:\sum_{j\ge1}|b_j|^2\le1\right\}. E := { j ≥ 1 ∑ μ j b j ψ j : j ≥ 1 ∑ ∣ b j ∣ 2 ≤ 1 } .
where ψ j \psi_j ψ j is the j j j th principal direction. Retaining the first m m m directions gives worst-case squared error
sup ∑ j ∣ b j ∣ 2 ≤ 1 ∑ j > m μ j ∣ b j ∣ 2 = μ m + 1 , \sup_{\sum_j|b_j|^2\le1}\sum_{j>m}\mu_j|b_j|^2=\mu_{m+1}, ∑ j ∣ b j ∣ 2 ≤ 1 sup j > m ∑ μ j ∣ b j ∣ 2 = μ m + 1 ,
with equality attained at the target μ m + 1 ψ m + 1 \sqrt{\mu_{m+1}}\psi_{m+1} μ m + 1 ψ m + 1 . This choice of directions is optimal. Indeed, for any trial space V V V of dimension at most m m m , the span of ψ 1 , … , ψ m + 1 \psi_1,\ldots,\psi_{m+1} ψ 1 , … , ψ m + 1 contains a unit vector w w w orthogonal to V V V . Since the first m + 1 m+1 m + 1 semiaxes all have length at least μ m + 1 \sqrt{\mu_{m+1}} μ m + 1 , the target g ∗ = μ m + 1 w g^*=\sqrt{\mu_{m+1}}w g ∗ = μ m + 1 w belongs to E \mathcal{E} E and satisfies
inf v ∈ V ∥ g ∗ − v ∥ H ~ ( r ) 1 2 = ∥ g ∗ ∥ H ~ ( r ) 1 2 = μ m + 1 . \inf_{v\in V}\|g^*-v\|_{\widetilde{H}_{(r)}^1}^2=\|g^*\|_{\widetilde{H}_{(r)}^1}^2=\mu_{m+1}. v ∈ V inf ∥ g ∗ − v ∥ H ( r ) 1 2 = ∥ g ∗ ∥ H ( r ) 1 2 = μ m + 1 .
The space generated by m m m sampled features has dimension at most m m m , so its worst-case squared approximation error is at least μ m + 1 \mu_{m+1} μ m + 1 .
We next study approximation in the residual seminorm ∥ ⋅ ∥ R η \|\cdot\|_{R_\eta} ∥ ⋅ ∥ R η . For the exact solution u ∗ ∈ H k u^*\in H_k u ∗ ∈ H k of (7) , the squared approximation error is
inf u α ∈ Σ m Φ ∥ u α − u ∗ ∥ R η 2 = inf u α ∈ Σ m Φ ( ∥ L u α − f ∗ ∥ L 2 ( ρ I ) 2 + η ∥ B u α − g ∗ ∥ L 2 ( ρ B ) 2 ) . (46) \inf_{u_\alpha\in\Sigma_m^\Phi}\|u_\alpha-u^*\|_{R_\eta}^2
=
\inf_{u_\alpha\in\Sigma_m^\Phi}\left(\|\mathcal{L}u_\alpha-f^*\|_{L^2(\rho_I)}^2+\eta\|\mathcal{B}u_\alpha-g^*\|_{L^2(\rho_B)}^2\right). \tag*{(46)} u α ∈ Σ m Φ inf ∥ u α − u ∗ ∥ R η 2 = u α ∈ Σ m Φ inf ( ∥ L u α − f ∗ ∥ L 2 ( ρ I ) 2 + η ∥ B u α − g ∗ ∥ L 2 ( ρ B ) 2 ) . (46)
For η > 0 \eta>0 η > 0 , define the residual operator P R η : D R → L 2 ( Ω ‾ ) ‾ P_{R_\eta}:D_R\to\overline{L^2(\overline{\Omega})} P R η : D R → L 2 ( Ω ) by
( P R η f ) ( x ) = { L f ( x ) , x ∈ Ω , η B f ( x ) , x ∈ ∂ Ω , (47) (P_{R_\eta}f)(x)=
\begin{cases}
\mathcal{L}f(x), & x\in\Omega,\\
\sqrt{\eta}\,\mathcal{B}f(x), & x\in\partial\Omega,
\end{cases} \tag*{(47)} ( P R η f ) ( x ) = { L f ( x ) , η B f ( x ) , x ∈ Ω , x ∈ ∂ Ω , (47)
with equality understood almost everywhere with respect to the corresponding measures. Define the residual feature function by
ϕ R η ( x , θ ) : = ( P R η ϕ ( ⋅ , θ ) ) ( x ) = { L ϕ ( x , θ ) , x ∈ Ω , η B ϕ ( x , θ ) , x ∈ ∂ Ω . (48) \phi_{R_\eta}(x,\theta):=(P_{R_\eta}\phi(\cdot,\theta))(x)=
\begin{cases}
\mathcal{L}\phi(x,\theta), & x\in\Omega,\\
\sqrt{\eta}\,\mathcal{B}\phi(x,\theta), & x\in\partial\Omega.
\end{cases} \tag*{(48)} ϕ R η ( x , θ ) := ( P R η ϕ ( ⋅ , θ )) ( x ) = { L ϕ ( x , θ ) , η B ϕ ( x , θ ) , x ∈ Ω , x ∈ ∂ Ω. (48)
Assumption [5 ] implies that the map θ ↦ ϕ R η ( ⋅ , θ ) \theta\mapsto\phi_{R_\eta}(\cdot,\theta) θ ↦ ϕ R η ( ⋅ , θ ) belongs to L 2 ( π ; L 2 ( Ω ‾ ) ‾ ) L^2(\pi;\overline{L^2(\overline{\Omega})}) L 2 ( π ; L 2 ( Ω ) ) . Thus, Assumption [7 ] holds with B = L 2 ( Ω ‾ ) ‾ B=\overline{L^2(\overline{\Omega})} B = L 2 ( Ω ) and feature function ϕ R η \phi_{R_\eta} ϕ R η . The optimization problem (46) is equivalent to the following finite-dimensional optimization problem over the coefficient vector α ∈ R m \alpha\in\mathbb{R}^m α ∈ R m :
inf α ∈ R m ( α ⊤ K m R η α − 2 ( y R η ) ⊤ α + c R η ) , \inf_{\alpha\in\mathbb{R}^m}\left(\alpha^\top K_m^{R_\eta}\alpha-2(y^{R_\eta})^\top\alpha+c^{R_\eta}\right), α ∈ R m inf ( α ⊤ K m R η α − 2 ( y R η ) ⊤ α + c R η ) ,
where C R η = ∥ u ∗ ∥ R η 2 \mathcal{C}_{R_\eta}=\|u^*\|_{R_\eta}^2 C R η = ∥ u ∗ ∥ R η 2 and
K m R η = ( ⟨ L ϕ ( ⋅ , θ i ) , L ϕ ( ⋅ , θ j ) ⟩ L 2 ( ρ I ) + η ⟨ B ϕ ( ⋅ , θ i ) , B ϕ ( ⋅ , θ j ) ⟩ L 2 ( ρ B ) m p ( θ i ) p ( θ j ) ) i , j = 1 m = ( ⟨ ϕ R η ( ⋅ , θ i ) , ϕ R η ( ⋅ , θ j ) ⟩ L 2 ( Ω ‾ ) ‾ m p ( θ i ) p ( θ j ) ) i , j = 1 m , \begin{aligned}
K^{R_\eta}_m=\left(\frac{\langle\mathcal{L}\phi(\cdot,\theta_i),\mathcal{L}\phi(\cdot,\theta_j)\rangle_{L^2(\rho_I)}+\eta\langle\mathcal{B}\phi(\cdot,\theta_i),\mathcal{B}\phi(\cdot,\theta_j)\rangle_{L^2(\rho_B)}}{m\sqrt{p(\theta_i)p(\theta_j)}}\right)_{i,j=1}^{m} \\
&=\left(\frac{\langle\phi_{R_\eta}(\cdot,\theta_i),\phi_{R_\eta}(\cdot,\theta_j)\rangle_{\overline{L^2(\overline{\Omega})}}}{m\sqrt{p(\theta_i)p(\theta_j)}}\right)_{i,j=1}^{m},
\end{aligned} K m R η = ( m p ( θ i ) p ( θ j ) ⟨ L ϕ ( ⋅ , θ i ) , L ϕ ( ⋅ , θ j ) ⟩ L 2 ( ρ I ) + η ⟨ B ϕ ( ⋅ , θ i ) , B ϕ ( ⋅ , θ j ) ⟩ L 2 ( ρ B ) ) i , j = 1 m = ( m p ( θ i ) p ( θ j ) ⟨ ϕ R η ( ⋅ , θ i ) , ϕ R η ( ⋅ , θ j ) ⟩ L 2 ( Ω ) ) i , j = 1 m ,
y R η = ( ⟨ L ϕ ( ⋅ , θ i ) , f ∗ ( ⋅ ) ⟩ L 2 ( ρ I ) + η ⟨ B ϕ ( ⋅ , θ i ) , g ∗ ⟩ L 2 ( ρ B ) m p ( θ i ) ) i = 1 m = ( ⟨ ϕ R η ( ⋅ , θ i ) , P R η u ∗ ( ⋅ ) ⟩ L 2 ( Ω ‾ ) ‾ m p ( θ i ) ) i = 1 m . \begin{aligned}
y^{R_\eta}=\left(\frac{\langle\mathcal{L}\phi(\cdot,\theta_i),f^*(\cdot)\rangle_{L^2(\rho_I)}+\eta\langle\mathcal{B}\phi(\cdot,\theta_i),g^*\rangle_{L^2(\rho_B)}}{\sqrt{mp(\theta_i)}}\right)_{i=1}^{m} \\
&=\left(\frac{\langle\phi_{R_\eta}(\cdot,\theta_i),P_{R_\eta}u^*(\cdot)\rangle_{\overline{L^2(\overline{\Omega})}}}{\sqrt{mp(\theta_i)}}\right)_{i=1}^{m}.
\end{aligned} y R η = ( m p ( θ i ) ⟨ L ϕ ( ⋅ , θ i ) , f ∗ ( ⋅ ) ⟩ L 2 ( ρ I ) + η ⟨ B ϕ ( ⋅ , θ i ) , g ∗ ⟩ L 2 ( ρ B ) ) i = 1 m = ( m p ( θ i ) ⟨ ϕ R η ( ⋅ , θ i ) , P R η u ∗ ( ⋅ ) ⟩ L 2 ( Ω ) ) i = 1 m .
As in (35) , we consider
u ^ m , λ R η = arg min u α ∈ Σ m m ( ∥ u α − u ∗ ∥ R η 2 + λ ∥ α ∥ 2 2 ) . \widehat{u}^{R_\eta}_{m,\lambda}=\underset{u_\alpha\in\Sigma_m^m}{\arg\min}\left(\|u_\alpha-u^*\|_{R_\eta}^2+\lambda\|\alpha\|_2^2\right). u m , λ R η = u α ∈ Σ m m arg min ( ∥ u α − u ∗ ∥ R η 2 + λ ∥ α ∥ 2 2 ) .
Let ϕ ^ m : Ω ↦ R m \widehat{\phi}_m:\Omega\mapsto\mathbb{R}^m ϕ m : Ω ↦ R m with ( ϕ ^ m ( ⋅ ) ) i = ϕ ( ⋅ , θ i ) m p ( θ i ) (\widehat{\phi}_m(\cdot))_i=\frac{\phi(\cdot,\theta_i)}{\sqrt{mp(\theta_i)}} ( ϕ m ( ⋅ ) ) i = m p ( θ i ) ϕ ( ⋅ , θ i ) . Then u ^ m , λ R η = ϕ ^ m ( ⋅ ) ⊤ α ^ m , λ R η \widehat{u}^{R_\eta}_{m,\lambda}=\widehat{\phi}_m(\cdot)^\top\widehat{\alpha}^{R_\eta}_{m,\lambda} u m , λ R η = ϕ m ( ⋅ ) ⊤ α m , λ R η , where the coefficient vector α ^ m , λ R η \widehat{\alpha}^{R_\eta}_{m,\lambda} α m , λ R η is determined by
α ^ m , λ R η = arg min α ∈ R m ( α ⊤ ( K m R η + λ I m ) α − 2 ( y R η ) ⊤ α + C R η ) = ( K m R η + λ I m ) − 1 y R η . \widehat{\alpha}^{R_\eta}_{m,\lambda}=\underset{\alpha\in\mathbb{R}^m}{\arg\min}\left(\alpha^\top(K^{R_\eta}_m+\lambda I_m)\alpha-2(y^{R_\eta})^\top\alpha+\mathcal{C}_{R_\eta}\right)=(K^{R_\eta}_m+\lambda I_m)^{-1}y^{R_\eta}. α m , λ R η = α ∈ R m arg min ( α ⊤ ( K m R η + λ I m ) α − 2 ( y R η ) ⊤ α + C R η ) = ( K m R η + λ I m ) − 1 y R η .
We express the approximant in operator form. Let S m , R η : R m → L 2 ( Ω ‾ ) ‾ S_{m,R_\eta}:\mathbb{R}^m\to\overline{L^2(\overline{\Omega})} S m , R η : R m → L 2 ( Ω ) be the discrete operator, and let S m , R η ∗ : L 2 ( Ω ‾ ) ‾ → R m S^*_{m,R_\eta}:\overline{L^2(\overline{\Omega})}\to\mathbb{R}^m S m , R η ∗ : L 2 ( Ω ) → R m denote its adjoint. For a ∈ R m a\in\mathbb{R}^m a ∈ R m and f ∈ L 2 ( Ω ‾ ) ‾ f\in\overline{L^2(\overline{\Omega})} f ∈ L 2 ( Ω ) , these operators are given by
S m , R η a = 1 m ∑ i = 1 m a i p ( θ i ) ϕ R η ( ⋅ , θ i ) , S_{m,R_\eta}a=\frac{1}{\sqrt{m}}\sum_{i=1}^{m}\frac{a_i}{\sqrt{p(\theta_i)}}\phi_{R_\eta}(\cdot,\theta_i), S m , R η a = m 1 i = 1 ∑ m p ( θ i ) a i ϕ R η ( ⋅ , θ i ) ,
( S m , R η ∗ f ) j = ⟨ ϕ R η ( ⋅ , θ j ) , f ⟩ L 2 ( Ω ‾ ) ‾ m p ( θ j ) , j = 1 , … , m . (49) (S^*_{m,R_\eta}f)_j=\frac{\langle\phi_{R_\eta}(\cdot,\theta_j),f\rangle_{\overline{L^2(\overline{\Omega})}}}{\sqrt{mp(\theta_j)}},\qquad j=1,\ldots,m. \tag*{(49)} ( S m , R η ∗ f ) j = m p ( θ j ) ⟨ ϕ R η ( ⋅ , θ j ) , f ⟩ L 2 ( Ω ) , j = 1 , … , m . (49)
It follows that
P R η u ^ m , λ R η = S m , R η ( S m , R η ∗ S m , R η + λ I m ) − 1 S m , R η ∗ P R η u ∗ . (50) P_{R_\eta}\widehat{u}^{R_\eta}_{m,\lambda}=S_{m,R_\eta}(S^*_{m,R_\eta}S_{m,R_\eta}+\lambda I_m)^{-1}S^*_{m,R_\eta}P_{R_\eta}u^*. \tag*{(50)} P R η u m , λ R η = S m , R η ( S m , R η ∗ S m , R η + λ I m ) − 1 S m , R η ∗ P R η u ∗ . (50)
The corresponding operator Φ R η : L 2 ( π ) → L 2 ( Ω ‾ ) ‾ \Phi_{R_\eta}:L^2(\pi)\to\overline{L^2(\overline{\Omega})} Φ R η : L 2 ( π ) → L 2 ( Ω ) and its adjoint Φ R η ∗ \Phi^*_{R_\eta} Φ R η ∗ are obtained by specializing (27) and (28) , respectively, to B = L 2 ( Ω ‾ ) ‾ \mathcal{B}=\overline{L^2(\overline{\Omega})} B = L 2 ( Ω ) with feature function ϕ R η \phi_{R_\eta} ϕ R η . Explicitly, for α ∈ L 2 ( π ) \alpha\in L^2(\pi) α ∈ L 2 ( π ) and f ∈ L 2 ( Ω ‾ ) ‾ f\in\overline{L^2(\overline{\Omega})} f ∈ L 2 ( Ω ) ,
( Φ R η α ) ( x ) = ⟨ ϕ R η ( x , ⋅ ) , α ⟩ L 2 ( π ) , ( Φ R η ∗ f ) ( θ ) = ⟨ ϕ R η ( ⋅ , θ ) , f ⟩ L 2 ( Ω ‾ ) ‾ . (\Phi_{R_\eta}\alpha)(x)=\langle\phi_{R_\eta}(x,\cdot),\alpha\rangle_{L^2(\pi)},\qquad(\Phi^*_{R_\eta}f)(\theta)=\langle\phi_{R_\eta}(\cdot,\theta),f\rangle_{\overline{L^2(\overline{\Omega})}}. ( Φ R η α ) ( x ) = ⟨ ϕ R η ( x , ⋅ ) , α ⟩ L 2 ( π ) , ( Φ R η ∗ f ) ( θ ) = ⟨ ϕ R η ( ⋅ , θ ) , f ⟩ L 2 ( Ω ) .
To connect this formulation with KRR on Θ \Theta Θ , we introduce the kernel on the parameter space Θ \Theta Θ :
k ~ R η ( θ , θ ′ ) = ⟨ ϕ R η ( ⋅ , θ ) , ϕ R η ( ⋅ , θ ′ ) ⟩ L 2 ( Ω ‾ ) ‾ = ⟨ L ϕ ( ⋅ , θ ) , L ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ I ) + η ⟨ B ϕ ( ⋅ , θ ) , B ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ B ) . \begin{aligned}
\widetilde{k}_{R_\eta}(\theta,\theta')=\left\langle\phi_{R_\eta}(\cdot,\theta),\phi_{R_\eta}(\cdot,\theta')\right\rangle_{\overline{L^2(\overline{\Omega})}} \\
&=\left\langle L\phi(\cdot,\theta),L\phi(\cdot,\theta')\right\rangle_{L^2(\rho_I)}+\eta\left\langle B\phi(\cdot,\theta),B\phi(\cdot,\theta')\right\rangle_{L^2(\rho_B)}.
\end{aligned} k R η ( θ , θ ′ ) = ⟨ ϕ R η ( ⋅ , θ ) , ϕ R η ( ⋅ , θ ′ ) ⟩ L 2 ( Ω ) = ⟨ L ϕ ( ⋅ , θ ) , L ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ I ) + η ⟨ B ϕ ( ⋅ , θ ) , B ϕ ( ⋅ , θ ′ ) ⟩ L 2 ( ρ B ) .
Denote by H ~ R η \widetilde{\mathcal{H}}_{R_\eta} H R η the RKHS associated with k ~ R η \widetilde{k}_{R_\eta} k R η . The corresponding integral operator T ~ R η \widetilde{T}_{R_\eta} T R η is given by
T ~ R η g = ∫ Θ k ~ R η ( ⋅ , θ ′ ) g ( θ ′ ) d π ( θ ′ ) = Φ R η ∗ Φ R η g . (51) \widetilde{T}_{R_\eta}g=\int_{\Theta}\widetilde{k}_{R_\eta}(\cdot,\theta')g(\theta')\,\mathrm{d}\pi(\theta')=\Phi_{R_\eta}^{*}\Phi_{R_\eta}g. \tag*{(51)} T R η g = ∫ Θ k R η ( ⋅ , θ ′ ) g ( θ ′ ) d π ( θ ′ ) = Φ R η ∗ Φ R η g . (51)
Its spectral decomposition is
T ~ R η = ∑ j = 1 ∞ μ ~ j ( R η ) e ~ j ( R η ) ⊗ e ~ j ( R η ) , \widetilde{T}_{R_\eta}=\sum_{j=1}^{\infty}\widetilde{\mu}_{j}^{(R_\eta)}\widetilde{e}_{j}^{(R_\eta)}\otimes\widetilde{e}_{j}^{(R_\eta)}, T R η = j = 1 ∑ ∞ μ j ( R η ) e j ( R η ) ⊗ e j ( R η ) ,
where { μ ~ j ( R η ) } j = 1 ∞ \{\widetilde{\mu}_{j}^{(R_\eta)}\}_{j=1}^{\infty} { μ j ( R η ) } j = 1 ∞ are the positive eigenvalues in nonincreasing order and { e ~ j ( R η ) } j = 1 ∞ \{\widetilde{e}_{j}^{(R_\eta)}\}_{j=1}^{\infty} { e j ( R η ) } j = 1 ∞ are the corresponding orthonormal eigenfunctions in L 2 ( π ) L^2(\pi) L 2 ( π ) . For the target function ⟨ ϕ R η ( ⋅ , θ ) , P R η u ∗ ( ⋅ ) ⟩ L 2 ( Ω ‾ ) ‾ = ( Φ R η ∗ P R η u ∗ ) ( θ ) \langle\phi_{R_\eta}(\cdot,\theta),P_{R_\eta}u^*(\cdot)\rangle_{\overline{L^2(\overline{\Omega})}}=(\Phi_{R_\eta}^{*}P_{R_\eta}u^*)(\theta) ⟨ ϕ R η ( ⋅ , θ ) , P R η u ∗ ( ⋅ ) ⟩ L 2 ( Ω ) = ( Φ R η ∗ P R η u ∗ ) ( θ ) , the corresponding KRR estimator admits the operator representation
g ^ m , λ R η = Φ R η ∗ S m , R η ( S m , R η ∗ S m , R η + λ I m ) − 1 S m , R η ∗ P R η u ∗ = Φ R η ∗ P R η u ^ m , λ R η . (52) \begin{aligned}
\widehat{g}_{m,\lambda}^{R_\eta}=\Phi_{R_\eta}^{*}S_{m,R_\eta}\left(S_{m,R_\eta}^{*}S_{m,R_\eta}+\lambda I_m\right)^{-1}S_{m,R_\eta}^{*}P_{R_\eta}u^* \\
&=\Phi_{R_\eta}^{*}P_{R_\eta}\widehat{u}_{m,\lambda}^{R_\eta}. \tag*{(52)}
\end{aligned} g m , λ R η = Φ R η ∗ S m , R η ( S m , R η ∗ S m , R η + λ I m ) − 1 S m , R η ∗ P R η u ∗ = Φ R η ∗ P R η u m , λ R η . (52)
Let H R η s \mathcal{H}_{R_\eta}^{s} H R η s and H ~ R η s \widetilde{\mathcal{H}}_{R_\eta}^{s} H R η s denote the interpolation spaces defined in (30) and (31) , respectively, with B = L 2 ( Ω ‾ ) ‾ B=\overline{L^2(\overline{\Omega})} B = L 2 ( Ω ) and feature function ϕ R η \phi_{R_\eta} ϕ R η . The minimum-coefficient characterization identifies H R η 1 \mathcal{H}_{R_\eta}^{1} H R η 1 with P R η H k P_{R_\eta}\mathcal{H}_k P R η H k , equipped with the norm
∥ h ∥ H R η 1 = min α ∈ L 2 ( π ) Φ R η α = h ∥ α ∥ L 2 ( π ) = min v ∈ H k P R η v = h ∥ v ∥ H k . \|h\|_{\mathcal{H}_{R_\eta}^{1}}=\min_{\substack{\alpha\in L^2(\pi)\\ \Phi_{R_\eta}\alpha=h}}\|\alpha\|_{L^2(\pi)}=\min_{\substack{v\in\mathcal{H}_k\\ P_{R_\eta}v=h}}\|v\|_{\mathcal{H}_k}. ∥ h ∥ H R η 1 = α ∈ L 2 ( π ) Φ R η α = h min ∥ α ∥ L 2 ( π ) = v ∈ H k P R η v = h min ∥ v ∥ H k .
Thus, P R η P_{R_\eta} P R η maps the unit ball of H k \mathcal{H}_k H k onto that of H R η 1 \mathcal{H}_{R_\eta}^{1} H R η 1 . For h ∗ ∈ H R η 1 h^*\in\mathcal{H}_{R_\eta}^{1} h ∗ ∈ H R η 1 , let g ∗ ( θ ) : = Φ R η ∗ h ∗ g^*(\theta):=\Phi_{R_\eta}^{*}h^* g ∗ ( θ ) := Φ R η ∗ h ∗ . By Lemma 10 , the map h ∗ ↦ g ∗ h^*\mapsto g^* h ∗ ↦ g ∗ is an isometric isomorphism from H R η 1 \mathcal{H}_{R_\eta}^{1} H R η 1 onto H ~ R η 2 \widetilde{\mathcal{H}}_{R_\eta}^{2} H R η 2 . In particular, ∥ g ∗ ∥ H ~ R η 2 = ∥ h ∗ ∥ H R η 1 \|g^*\|_{\widetilde{\mathcal{H}}_{R_\eta}^{2}}=\|h^*\|_{\mathcal{H}_{R_\eta}^{1}} ∥ g ∗ ∥ H R η 2 = ∥ h ∗ ∥ H R η 1 . Lemma 11 with s = 2 s=2 s = 2 and p = 1 p=1 p = 1 gives
sup ∥ u ∗ ∥ H k ≤ 1 inf u ∈ Σ m ϕ ∥ u − u ∗ ∥ R η 2 = sup ∥ h ∗ ∥ H R η 1 ≤ 1 inf h ∈ P R η Σ m ϕ ∥ h − h ∗ ∥ L 2 ( Ω ‾ ) ‾ 2 = sup ∥ g ∗ ∥ H ~ R η 2 ≤ 1 inf a ∈ R m ∥ ∑ i = 1 m a i k ~ R η ( ⋅ , θ i ) − g ∗ ∥ H ~ R η 1 2 . \begin{aligned}
\sup_{\|u^*\|_{\mathcal{H}_k}\le1}\inf_{u\in\Sigma_m^\phi}\|u-u^*\|_{R_\eta}^{2}
=
\sup_{\|h^*\|_{\mathcal{H}_{R_\eta}^{1}}\le1}\inf_{h\in P_{R_\eta}\Sigma_m^\phi}\|h-h^*\|_{\overline{L^2(\overline{\Omega})}}^{2} \\
&=
\sup_{\|g^*\|_{\widetilde{\mathcal{H}}_{R_\eta}^{2}}\le1}\inf_{a\in\mathbb{R}^{m}}
\left\|\sum_{i=1}^{m}a_i\widetilde{k}_{R_\eta}(\cdot,\theta_i)-g^*\right\|_{\widetilde{\mathcal{H}}_{R_\eta}^{1}}^{2}.
\end{aligned} ∥ u ∗ ∥ H k ≤ 1 sup u ∈ Σ m ϕ inf ∥ u − u ∗ ∥ R η 2 = ∥ h ∗ ∥ H R η 1 ≤ 1 sup h ∈ P R η Σ m ϕ inf ∥ h − h ∗ ∥ L 2 ( Ω ) 2 = ∥ g ∗ ∥ H R η 2 ≤ 1 sup a ∈ R m inf i = 1 ∑ m a i k R η ( ⋅ , θ i ) − g ∗ H R η 1 2 .
Consequently, the upper and lower bounds for the approximation error in the residual seminorm follow by the same arguments as in the proofs of Theorems 1 and 2 , respectively.
Estimation Error Analysis
We now study the estimation error associated with a finite number of collocation observations. For λ > 0 \lambda> 0 λ > 0 , consider the regularized estimator defined in (17) . By the feature-space characterization of H k \mathcal{H}_k H k in (13) ,
u ^ n , η , λ = Φ ( arg min b ∈ L 2 ( π ) { R ^ n , η ( Φ b ) + λ ∥ b ∥ L 2 ( π ) 2 } ) . (53) \widehat{u}_{n,\eta,\lambda}=\Phi\left(\underset{b\in L^2(\pi)}{\arg\min}\left\{\widehat{\mathcal{R}}_{n,\eta}(\Phi b)+\lambda\lVert b\rVert_{L^2(\pi)}^2\right\}\right). \tag*{(53)} u n , η , λ = Φ ( b ∈ L 2 ( π ) arg min { R n , η ( Φ b ) + λ ∥ b ∥ L 2 ( π ) 2 } ) . (53)
We next reformulate (53) as a KRR problem. Writing ν I = n / n I \nu_I=n/n_I ν I = n / n I and ν B = n / n B \nu_B=n/n_B ν B = n / n B , we have
R ^ n , η ( u ) = 1 n [ ∑ i = 1 n I ∣ ν I L u ( x i ) − ν I f ∗ ( x i ) ∣ 2 q I ( x i ) + ∑ i = n I + 1 n ∣ η ν B B u ( x i ) − η ν B g ∗ ( x i ) ∣ 2 q B ( x i ) ] . \widehat{\mathcal{R}}_{n,\eta}(u)=\frac{1}{n}\left[\sum_{i=1}^{n_I}\frac{\left\lvert\sqrt{\nu_I}\mathcal{L}u(x_i)-\sqrt{\nu_I}f^*(x_i)\right\rvert^2}{q_I(x_i)}+\sum_{i=n_I+1}^{n}\frac{\left\lvert\sqrt{\eta\nu_B}\mathcal{B}u(x_i)-\sqrt{\eta\nu_B}g^*(x_i)\right\rvert^2}{q_B(x_i)}\right]. R n , η ( u ) = n 1 [ i = 1 ∑ n I q I ( x i ) ν I L u ( x i ) − ν I f ∗ ( x i ) 2 + i = n I + 1 ∑ n q B ( x i ) η ν B B u ( x i ) − η ν B g ∗ ( x i ) 2 ] .
Thus we define P P D E : D R → L 2 ( Ω ‾ ) P_{\mathrm{PDE}}:D_R\to L^2(\overline{\Omega}) P PDE : D R → L 2 ( Ω ) by
( P P D E u ) ( x ) : = { ν I L u ( x ) , x ∈ Ω , η ν B B u ( x ) , x ∈ ∂ Ω . (P_{\mathrm{PDE}}u)(x):=
\begin{cases}
\sqrt{\nu_I}\mathcal{L}u(x), & x\in\Omega,\\
\sqrt{\eta\nu_B}\mathcal{B}u(x), & x\in\partial\Omega.
\end{cases} ( P PDE u ) ( x ) := { ν I L u ( x ) , η ν B B u ( x ) , x ∈ Ω , x ∈ ∂ Ω.
The corresponding feature function is
ϕ P D E ( x , θ ) : = ( P P D E ϕ ( ⋅ , θ ) ) ( x ) = { ν I L ϕ ( x , θ ) , x ∈ Ω , η ν B B ϕ ( x , θ ) , x ∈ ∂ Ω . \phi_{\mathrm{PDE}}(x,\theta):=(P_{\mathrm{PDE}}\phi(\cdot,\theta))(x)=
\begin{cases}
\sqrt{\nu_I}\mathcal{L}\phi(x,\theta), & x\in\Omega,\\
\sqrt{\eta\nu_B}\mathcal{B}\phi(x,\theta), & x\in\partial\Omega.
\end{cases} ϕ PDE ( x , θ ) := ( P PDE ϕ ( ⋅ , θ )) ( x ) = { ν I L ϕ ( x , θ ) , η ν B B ϕ ( x , θ ) , x ∈ Ω , x ∈ ∂ Ω.
The transformed features therefore induce the kernel
k P D E ( x , x ′ ) = ∫ Θ ϕ P D E ( x , θ ) ϕ P D E ( x ′ , θ ) d π ( θ ) = ⟨ ϕ P D E ( x , ⋅ ) , ϕ P D E ( x ′ , ⋅ ) ⟩ L 2 ( π ) . k_{\mathrm{PDE}}(x,x')=\int_{\Theta}\phi_{\mathrm{PDE}}(x,\theta)\phi_{\mathrm{PDE}}(x',\theta)\,\mathrm{d}\pi(\theta)=\langle\phi_{\mathrm{PDE}}(x,\cdot),\phi_{\mathrm{PDE}}(x',\cdot)\rangle_{L^2(\pi)}. k PDE ( x , x ′ ) = ∫ Θ ϕ PDE ( x , θ ) ϕ PDE ( x ′ , θ ) d π ( θ ) = ⟨ ϕ PDE ( x , ⋅ ) , ϕ PDE ( x ′ , ⋅ ) ⟩ L 2 ( π ) .
Let H P D E \mathcal{H}_{\mathrm{PDE}} H PDE denote the associated RKHS, viewed as continuously embedded in L 2 ( Ω ‾ ) L^2(\overline{\Omega}) L 2 ( Ω ) . The feature operator Φ P D E : L 2 ( π ) → L 2 ( Ω ‾ ) \Phi_{\mathrm{PDE}}:L^2(\pi)\to L^2(\overline{\Omega}) Φ PDE : L 2 ( π ) → L 2 ( Ω ) associated with ϕ P D E \phi_{\mathrm{PDE}} ϕ PDE and its adjoint Φ P D E ∗ : L 2 ( Ω ‾ ) → L 2 ( π ) \Phi_{\mathrm{PDE}}^*:L^2(\overline{\Omega})\to L^2(\pi) Φ PDE ∗ : L 2 ( Ω ) → L 2 ( π ) are given by
( Φ P D E b ) ( x ) = ⟨ ϕ P D E ( x , ⋅ ) , b ⟩ L 2 ( π ) , ( Φ P D E ∗ f ) ( θ ) = ⟨ ϕ P D E ( ⋅ , θ ) , f ⟩ L 2 ( Ω ‾ ) . (\Phi_{\mathrm{PDE}}b)(x)=\langle\phi_{\mathrm{PDE}}(x,\cdot),b\rangle_{L^2(\pi)},\qquad(\Phi_{\mathrm{PDE}}^*f)(\theta)=\langle\phi_{\mathrm{PDE}}(\cdot,\theta),f\rangle_{L^2(\overline{\Omega})}. ( Φ PDE b ) ( x ) = ⟨ ϕ PDE ( x , ⋅ ) , b ⟩ L 2 ( π ) , ( Φ PDE ∗ f ) ( θ ) = ⟨ ϕ PDE ( ⋅ , θ ) , f ⟩ L 2 ( Ω ) .
Since Assumption 5 implies Φ P D E = P P D E Φ \Phi_{\mathrm{PDE}}=P_{\mathrm{PDE}}\Phi Φ PDE = P PDE Φ , the minimum-coefficient characterization identifies H P D E \mathcal{H}_{\mathrm{PDE}} H PDE with P P D E H k P_{\mathrm{PDE}}\mathcal{H}_k P PDE H k , equipped with the norm
∥ h ∥ H P D E = min b ∈ L 2 ( π ) Φ P D E b = h ∥ b ∥ L 2 ( π ) = min u ∈ H k P P D E u = h ∥ u ∥ H k . \lVert h\rVert_{\mathcal{H}_{\mathrm{PDE}}}=\min_{\substack{b\in L^2(\pi)\\\Phi_{\mathrm{PDE}}b=h}}\lVert b\rVert_{L^2(\pi)}=\min_{\substack{u\in\mathcal{H}_k\\P_{\mathrm{PDE}}u=h}}\lVert u\rVert_{\mathcal{H}_k}. ∥ h ∥ H PDE = b ∈ L 2 ( π ) Φ PDE b = h min ∥ b ∥ L 2 ( π ) = u ∈ H k P PDE u = h min ∥ u ∥ H k .
Thus, P P D E P_{\mathrm{PDE}} P PDE maps the closed unit ball of H k \mathcal{H}_k H k onto that of H P D E \mathcal{H}_{\mathrm{PDE}} H PDE . Let h ∗ = P P D E u ∗ h^*=P_{\mathrm{PDE}}u^* h ∗ = P PDE u ∗ . Then (53) is equivalent to a KRR problem in H P D E \mathcal{H}_{\mathrm{PDE}} H PDE , with
P P D E u ^ n , η , λ = argmin h ∈ H P D E { 1 n ∑ i = 1 n ∣ h ( x i ) − h ∗ ( x i ) ∣ 2 q ( x i ) + λ ∥ h ∥ H P D E 2 } . P_{\mathrm{PDE}}\widehat{u}_{n,\eta,\lambda}=\underset{h\in\mathcal{H}_{\mathrm{PDE}}}{\operatorname{argmin}}\left\{\frac{1}{n}\sum_{i=1}^{n}\frac{\left|h(x_i)-h^*(x_i)\right|^2}{q(x_i)}+\lambda\|h\|_{\mathcal{H}_{\mathrm{PDE}}}^2\right\}. P PDE u n , η , λ = h ∈ H PDE argmin { n 1 i = 1 ∑ n q ( x i ) ∣ h ( x i ) − h ∗ ( x i ) ∣ 2 + λ ∥ h ∥ H PDE 2 } .
where q ( x ) : = q I ( x ) q(x):=q_I(x) q ( x ) := q I ( x ) for x ∈ Ω x\in\Omega x ∈ Ω and q ( x ) : = q B ( x ) q(x):=q_B(x) q ( x ) := q B ( x ) for x ∈ ∂ Ω x\in\partial\Omega x ∈ ∂ Ω . To derive the estimator explicitly, set
( ϕ n , P D E ( θ ) ) i : = ϕ P D E ( x i , θ ) n q ( x i ) , ( y P D E ) i : = h ∗ ( x i ) n q ( x i ) , i = 1 , … , n . (\phi_{n,\mathrm{PDE}}(\theta))_i:=\frac{\phi_{\mathrm{PDE}}(x_i,\theta)}{\sqrt{nq(x_i)}},\qquad(y_{\mathrm{PDE}})_i:=\frac{h^*(x_i)}{\sqrt{nq(x_i)}},\qquad i=1,\ldots,n. ( ϕ n , PDE ( θ ) ) i := n q ( x i ) ϕ PDE ( x i , θ ) , ( y PDE ) i := n q ( x i ) h ∗ ( x i ) , i = 1 , … , n .
The corresponding discrete observation operator S n , P D E : L 2 ( π ) → R n S_{n,\mathrm{PDE}}:L^2(\pi)\to\mathbb{R}^n S n , PDE : L 2 ( π ) → R n and its adjoint are given by
( S n , P D E b ) i = ⟨ ϕ P D E ( x i , ⋅ ) , b ⟩ L 2 ( π ) n q ( x i ) , i = 1 , … , n , S n , P D E ∗ a = ϕ n , P D E ( ⋅ ) T a , (S_{n,\mathrm{PDE}}b)_i=\frac{\langle\phi_{\mathrm{PDE}}(x_i,\cdot),b\rangle_{L^2(\pi)}}{\sqrt{nq(x_i)}},\qquad i=1,\ldots,n,\qquad S^*_{n,\mathrm{PDE}}a=\phi_{n,\mathrm{PDE}}(\cdot)^\mathsf{T}a, ( S n , PDE b ) i = n q ( x i ) ⟨ ϕ PDE ( x i , ⋅ ) , b ⟩ L 2 ( π ) , i = 1 , … , n , S n , PDE ∗ a = ϕ n , PDE ( ⋅ ) T a ,
for b ∈ L 2 ( π ) b\in L^2(\pi) b ∈ L 2 ( π ) and a ∈ R n a\in\mathbb{R}^n a ∈ R n . The coefficient minimization in (53) is thus finding
b ^ n , η , λ = argmin b ∈ L 2 ( π ) { ∥ S n , P D E b − y P D E ∥ R n 2 + λ ∥ b ∥ L 2 ( π ) 2 } . \widehat{b}_{n,\eta,\lambda}=\underset{b\in L^2(\pi)}{\operatorname{argmin}}\left\{\|S_{n,\mathrm{PDE}}b-y_{\mathrm{PDE}}\|_{\mathbb{R}^n}^2+\lambda\|b\|_{L^2(\pi)}^2\right\}. b n , η , λ = b ∈ L 2 ( π ) argmin { ∥ S n , PDE b − y PDE ∥ R n 2 + λ ∥ b ∥ L 2 ( π ) 2 } .
For every v ∈ L 2 ( π ) v\in L^2(\pi) v ∈ L 2 ( π ) , the first-order optimality condition gives
⟨ S n , P D E b ^ n , η , λ − y P D E , S n , P D E v ⟩ R n + λ ⟨ b ^ n , η , λ , v ⟩ L 2 ( π ) = 0. \left\langle S_{n,\mathrm{PDE}}\widehat{b}_{n,\eta,\lambda}-y_{\mathrm{PDE}},S_{n,\mathrm{PDE}}v\right\rangle_{\mathbb{R}^n}+\lambda\left\langle\widehat{b}_{n,\eta,\lambda},v\right\rangle_{L^2(\pi)}=0. ⟨ S n , PDE b n , η , λ − y PDE , S n , PDE v ⟩ R n + λ ⟨ b n , η , λ , v ⟩ L 2 ( π ) = 0.
Since v v v is arbitrary, this is equivalent to
S n , P D E ∗ ( S n , P D E b ^ n , η , λ − y P D E ) + λ b ^ n , η , λ = 0. S^*_{n,\mathrm{PDE}}\left(S_{n,\mathrm{PDE}}\widehat{b}_{n,\eta,\lambda}-y_{\mathrm{PDE}}\right)+\lambda\widehat{b}_{n,\eta,\lambda}=0. S n , PDE ∗ ( S n , PDE b n , η , λ − y PDE ) + λ b n , η , λ = 0.
Hence
b ^ n , η , λ = ( S n , P D E ∗ S n , P D E + λ I ) − 1 S n , P D E ∗ y P D E = S n , P D E ∗ ( S n , P D E S n , P D E ∗ + λ I n ) − 1 y P D E , \widehat{b}_{n,\eta,\lambda}=\left(S^*_{n,\mathrm{PDE}}S_{n,\mathrm{PDE}}+\lambda I\right)^{-1}S^*_{n,\mathrm{PDE}}y_{\mathrm{PDE}}=S^*_{n,\mathrm{PDE}}\left(S_{n,\mathrm{PDE}}S^*_{n,\mathrm{PDE}}+\lambda I_n\right)^{-1}y_{\mathrm{PDE}}, b n , η , λ = ( S n , PDE ∗ S n , PDE + λ I ) − 1 S n , PDE ∗ y PDE = S n , PDE ∗ ( S n , PDE S n , PDE ∗ + λ I n ) − 1 y PDE ,
where the second equality follows from (A.2). Since y P D E = S n , P D E b ∗ y_{\mathrm{PDE}}=S_{n,\mathrm{PDE}}b^* y PDE = S n , PDE b ∗ , we obtain
u ^ n , η , λ = Φ b ^ n , η , λ = Φ ( S n , P D E ∗ S n , P D E + λ I n ) − 1 S n , P D E b ∗ . (54) \widehat{u}_{n,\eta,\lambda}=\Phi\widehat{b}_{n,\eta,\lambda}=\Phi\left(S^*_{n,\mathrm{PDE}}S_{n,\mathrm{PDE}}+\lambda I_n\right)^{-1}S_{n,\mathrm{PDE}}b^*. \tag*{(54)} u n , η , λ = Φ b n , η , λ = Φ ( S n , PDE ∗ S n , PDE + λ I n ) − 1 S n , PDE b ∗ . (54)
For any u = Φ b u=\Phi b u = Φ b , the estimation error in terms of the residual seminorm is determined by:
∥ u − u ∗ ∥ R η 2 = ∥ P R η ( u − u ∗ ) ∥ L 2 ( Ω ‾ ) 2 = ∥ Φ R η ( b − b ∗ ) ∥ L 2 ( Ω ‾ ) 2 = ⟨ R ~ η ( b − b ∗ ) , b − b ∗ ⟩ L 2 ( π ) = ∥ R ~ η 1 / 2 ( b − b ∗ ) ∥ L 2 ( π ) 2 . \begin{aligned}
\|u-u^*\|_{\mathcal{R}_\eta}^2=\|P_{\mathcal{R}_\eta}(u-u^*)\|_{L^2(\overline{\Omega})}^2=\|\Phi\mathcal{R}_\eta(b-b^*)\|_{L^2(\overline{\Omega})}^2 \\
&=\left\langle\widetilde{\mathcal{R}}_\eta(b-b^*),b-b^*\right\rangle_{L^2(\pi)}=\left\|\widetilde{\mathcal{R}}_\eta^{1/2}(b-b^*)\right\|_{L^2(\pi)}^2.
\end{aligned} ∥ u − u ∗ ∥ R η 2 = ∥ P R η ( u − u ∗ ) ∥ L 2 ( Ω ) 2 = ∥Φ R η ( b − b ∗ ) ∥ L 2 ( Ω ) 2 = ⟨ R η ( b − b ∗ ) , b − b ∗ ⟩ L 2 ( π ) = R η 1/2 ( b − b ∗ ) L 2 ( π ) 2 .
Here T ~ R η = Φ R η ∗ Φ R η \widetilde{T}_{R_\eta}=\Phi_{R_\eta}^{*}\Phi_{R_\eta} T R η = Φ R η ∗ Φ R η , and T ~ R η 1 / 2 \widetilde{T}_{R_\eta}^{1/2} T R η 1/2 denotes its positive square root. We therefore relate this operator to T ~ P D E : = Φ P D E ∗ Φ P D E \widetilde{T}_{\mathrm{PDE}}:=\Phi_{\mathrm{PDE}}^{*}\Phi_{\mathrm{PDE}} T PDE := Φ PDE ∗ Φ PDE . To express T ~ P D E \widetilde{T}_{\mathrm{PDE}} T PDE and T ~ R η \widetilde{T}_{R_\eta} T R η using the same interior and boundary components, we introduce Φ L : L 2 ( π ) → L 2 ( ρ I ) \Phi_{\mathcal{L}}:L^2(\pi)\to L^2(\rho_I) Φ L : L 2 ( π ) → L 2 ( ρ I ) and Φ B : L 2 ( π ) → L 2 ( ρ B ) \Phi_{\mathcal{B}}:L^2(\pi)\to L^2(\rho_B) Φ B : L 2 ( π ) → L 2 ( ρ B ) . These operators and their adjoints are given by
Φ L b = ∫ Θ L ϕ ( ⋅ , θ ) b ( θ ) d π ( θ ) , Φ L ∗ f = ∫ Ω L ϕ ( x , ⋅ ) f ( x ) d ρ I ( x ) . (55) \Phi_{\mathcal{L}}b=\int_{\Theta}\mathcal{L}\phi(\cdot,\theta)b(\theta)\,\mathrm{d}\pi(\theta),\qquad
\Phi_{\mathcal{L}}^{*}f=\int_{\Omega}\mathcal{L}\phi(x,\cdot)f(x)\,\mathrm{d}\rho_I(x). \tag*{(55)} Φ L b = ∫ Θ L ϕ ( ⋅ , θ ) b ( θ ) d π ( θ ) , Φ L ∗ f = ∫ Ω L ϕ ( x , ⋅ ) f ( x ) d ρ I ( x ) . (55)
and
Φ B b = ∫ Θ B ϕ ( ⋅ , θ ) b ( θ ) d π ( θ ) , Φ B ∗ f = ∫ ∂ Ω B ϕ ( x , ⋅ ) f ( x ) d ρ B ( x ) . (56) \Phi_{\mathcal{B}}b=\int_{\Theta}\mathcal{B}\phi(\cdot,\theta)b(\theta)\,\mathrm{d}\pi(\theta),\qquad
\Phi_{\mathcal{B}}^{*}f=\int_{\partial\Omega}\mathcal{B}\phi(x,\cdot)f(x)\,\mathrm{d}\rho_B(x). \tag*{(56)} Φ B b = ∫ Θ B ϕ ( ⋅ , θ ) b ( θ ) d π ( θ ) , Φ B ∗ f = ∫ ∂ Ω B ϕ ( x , ⋅ ) f ( x ) d ρ B ( x ) . (56)
Writing T ~ L : = Φ L ∗ Φ L \widetilde{T}_{\mathcal{L}}:=\Phi_{\mathcal{L}}^{*}\Phi_{\mathcal{L}} T L := Φ L ∗ Φ L and T ~ B : = Φ B ∗ Φ B \widetilde{T}_{\mathcal{B}}:=\Phi_{\mathcal{B}}^{*}\Phi_{\mathcal{B}} T B := Φ B ∗ Φ B , we obtain T ~ R η = T ~ L + η T ~ B \widetilde{T}_{R_\eta}=\widetilde{T}_{\mathcal{L}}+\eta\widetilde{T}_{\mathcal{B}} T R η = T L + η T B , and
T ~ P D E : = Φ P D E ∗ Φ P D E = ν I T ~ L + η ν B T ~ B . (57) \widetilde{T}_{\mathrm{PDE}}:=\Phi_{\mathrm{PDE}}^{*}\Phi_{\mathrm{PDE}}=\nu_I\widetilde{T}_{\mathcal{L}}+\eta\nu_B\widetilde{T}_{\mathcal{B}}. \tag*{(57)} T PDE := Φ PDE ∗ Φ PDE = ν I T L + η ν B T B . (57)
Thus, setting ν − : = min { ν I , ν B } \nu_-:=\min\{\nu_I,\nu_B\} ν − := min { ν I , ν B } and ν + : = max { ν I , ν B } \nu_+:=\max\{\nu_I,\nu_B\} ν + := max { ν I , ν B } , we have
ν − T ~ R η ⪯ T ~ P D E ⪯ ν + T ~ R η . (58) \nu_-\widetilde{T}_{R_\eta}\preceq\widetilde{T}_{\mathrm{PDE}}\preceq\nu_+\widetilde{T}_{R_\eta}. \tag*{(58)} ν − T R η ⪯ T PDE ⪯ ν + T R η . (58)
The corresponding comparison in the function space is
ν − ∥ v ∥ R η 2 ≤ ∥ P P D E v ∥ L 2 ( Ω ‾ ) 2 ≤ ν + ∥ v ∥ R η 2 , v ∈ D R . (59) \nu_-\|v\|_{R_\eta}^{2}\leq\|P_{\mathrm{PDE}}v\|_{L^2(\overline{\Omega})}^{2}\leq\nu_+\|v\|_{R_\eta}^{2},\qquad v\in D_R. \tag*{(59)} ν − ∥ v ∥ R η 2 ≤ ∥ P PDE v ∥ L 2 ( Ω ) 2 ≤ ν + ∥ v ∥ R η 2 , v ∈ D R . (59)
To prove Theorem 5, we introduce the relevant degrees of freedom. Let ( μ j L , e j L ) (\mu_j^{\mathcal{L}},e_j^{\mathcal{L}}) ( μ j L , e j L ) and ( μ j B , e j B ) (\mu_j^{\mathcal{B}},e_j^{\mathcal{B}}) ( μ j B , e j B ) denote the positive eigenpairs of the integral operators Φ L Φ L ∗ : L 2 ( ρ I ) → L 2 ( ρ I ) \Phi_{\mathcal{L}}\Phi_{\mathcal{L}}^{*}:L^2(\rho_I)\to L^2(\rho_I) Φ L Φ L ∗ : L 2 ( ρ I ) → L 2 ( ρ I ) and Φ B Φ B ∗ : L 2 ( ρ B ) → L 2 ( ρ B ) \Phi_{\mathcal{B}}\Phi_{\mathcal{B}}^{*}:L^2(\rho_B)\to L^2(\rho_B) Φ B Φ B ∗ : L 2 ( ρ B ) → L 2 ( ρ B ) , respectively. Their kernels k L k_{\mathcal{L}} k L and k B k_{\mathcal{B}} k B are defined in (18) and (19) . Following (4) , for λ > 0 \lambda>0 λ > 0 , define
N L ( λ ) : = ∑ j ≥ 1 μ j L μ j L + λ , N B ( λ ) : = ∑ j ≥ 1 μ j B μ j B + λ , N P D E ( λ ) : = ∑ j ≥ 1 μ j P D E μ j P D E + λ . N_{\mathcal{L}}(\lambda):=\sum_{j\geq1}\frac{\mu_j^{\mathcal{L}}}{\mu_j^{\mathcal{L}}+\lambda},\qquad
N_{\mathcal{B}}(\lambda):=\sum_{j\geq1}\frac{\mu_j^{\mathcal{B}}}{\mu_j^{\mathcal{B}}+\lambda},\qquad
N_{\mathrm{PDE}}(\lambda):=\sum_{j\geq1}\frac{\mu_j^{\mathrm{PDE}}}{\mu_j^{\mathrm{PDE}}+\lambda}. N L ( λ ) := j ≥ 1 ∑ μ j L + λ μ j L , N B ( λ ) := j ≥ 1 ∑ μ j B + λ μ j B , N PDE ( λ ) := j ≥ 1 ∑ μ j PDE + λ μ j PDE .
For the sampling densities q I q_I q I and q B q_B q B , define
F L ( λ ) : = ess sup x ∈ Ω 1 q I ( x ) ∑ j ≥ 1 μ j L μ j L + λ ∣ e j L ( x ) ∣ 2 , F_{\mathcal{L}}(\lambda):=\operatorname*{ess\,sup}_{x\in\Omega}\frac{1}{q_I(x)}\sum_{j\geq1}\frac{\mu_j^{\mathcal{L}}}{\mu_j^{\mathcal{L}}+\lambda}|e_j^{\mathcal{L}}(x)|^2, F L ( λ ) := x ∈ Ω ess sup q I ( x ) 1 j ≥ 1 ∑ μ j L + λ μ j L ∣ e j L ( x ) ∣ 2 ,
and
F B ( λ ) : = ess sup x ∈ ∂ Ω 1 q B ( x ) ∑ j ≥ 1 μ j B μ j B + λ ∣ e j B ( x ) ∣ 2 . F_{\mathcal{B}}(\lambda):=\operatorname*{ess\,sup}_{x\in\partial\Omega}\frac{1}{q_B(x)}\sum_{j\geq1}\frac{\mu_j^{\mathcal{B}}}{\mu_j^{\mathcal{B}}+\lambda}|e_j^{\mathcal{B}}(x)|^2. F B ( λ ) := x ∈ ∂ Ω ess sup q B ( x ) 1 j ≥ 1 ∑ μ j B + λ μ j B ∣ e j B ( x ) ∣ 2 .
The dependence of F L F_{\mathcal{L}} F L and F B F_{\mathcal{B}} F B on q I q_I q I and q B q_B q B is suppressed in the notation. To characterize the admissible range of λ \lambda λ in Theorem 5, we introduce a concept of critical penalty.
Definition 4 (Critical penalty) For λ > 0 \lambda> 0 λ > 0 , set
F I ( λ ) : = max { N P D E ( λ ) , F C ( λ ) } , F_I(\lambda) := \max\{N_{\mathrm{PDE}}(\lambda), F_C(\lambda)\}, F I ( λ ) := max { N PDE ( λ ) , F C ( λ )} , and
F B ( λ ) : = max { N P D E ( λ ) , F B ( λ / η ) } . F_B(\lambda) := \max\{N_{\mathrm{PDE}}(\lambda), F_B(\lambda/\eta)\}. F B ( λ ) := max { N PDE ( λ ) , F B ( λ / η )} . For n ∈ N + n \in\mathbb{N}^{+} n ∈ N + and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , define
Λ I ( n , δ ) : = inf { λ > 0 : n ≥ 16 F I ( λ ) max { 1 , log 14 F I ( λ ) δ } } . (60) \Lambda_I(n,\delta) := \inf\left\{ \lambda> 0 : n \ge16F_I(\lambda) \max\left\{1,\log\frac{14F_I(\lambda)}{\delta}\right\} \right\}. \tag*{(60)} Λ I ( n , δ ) := inf { λ > 0 : n ≥ 16 F I ( λ ) max { 1 , log δ 14 F I ( λ ) } } . (60) and
Λ B ( n , δ ) : = inf { λ > 0 : n ≥ 16 F B ( λ ) max { 1 , log 14 F B ( λ ) δ } } . (61) \Lambda_B(n,\delta) := \inf\left\{ \lambda> 0 : n \ge16F_B(\lambda) \max\left\{1,\log\frac{14F_B(\lambda)}{\delta}\right\} \right\}. \tag*{(61)} Λ B ( n , δ ) := inf { λ > 0 : n ≥ 16 F B ( λ ) max { 1 , log δ 14 F B ( λ ) } } . (61)
We define the weighted empirical operators as
T ~ L , n I = 1 n I ∑ i = 1 n I L ϕ ( x i , ⋅ ) ⊗ L ϕ ( x i , ⋅ ) q I ( x i ) , T ~ B , n B = 1 n B ∑ i = 1 n B B ϕ ( x n I + i , ⋅ ) ⊗ B ϕ ( x n I + i , ⋅ ) q B ( x n I + i ) . \widetilde{\mathcal{T}}_{\mathcal{L},n_I} = \frac{1}{n_I}\sum_{i=1}^{n_I}\frac{\mathcal{L}\phi(x_i,\cdot)\otimes\mathcal{L}\phi(x_i,\cdot)}{q_I(x_i)}, \qquad\widetilde{\mathcal{T}}_{\mathcal{B},n_B} = \frac{1}{n_B}\sum_{i=1}^{n_B}\frac{\mathcal{B}\phi(x_{n_I+i},\cdot)\otimes\mathcal{B}\phi(x_{n_I+i},\cdot)}{q_B(x_{n_I+i})}. T L , n I = n I 1 i = 1 ∑ n I q I ( x i ) L ϕ ( x i , ⋅ ) ⊗ L ϕ ( x i , ⋅ ) , T B , n B = n B 1 i = 1 ∑ n B q B ( x n I + i ) B ϕ ( x n I + i , ⋅ ) ⊗ B ϕ ( x n I + i , ⋅ ) .
Thus, A n : = S n , P D E ∗ S n , P D E = T ~ L , n I + η T ~ B , n B A_n := S_{n,\mathrm{PDE}}^{*}S_{n,\mathrm{PDE}} = \widetilde{\mathcal{T}}_{\mathcal{L},n_I}+\eta\widetilde{\mathcal{T}}_{\mathcal{B},n_B} A n := S n , PDE ∗ S n , PDE = T L , n I + η T B , n B , and E [ A n ] = T ~ L + η T ~ B = T ~ R η \mathbb{E}[A_n] = \widetilde{\mathcal{T}}_{\mathcal{L}}+\eta\widetilde{\mathcal{T}}_{\mathcal{B}}=\widetilde{\mathcal{T}}_{R_\eta} E [ A n ] = T L + η T B = T R η .
We now prove the estimation upper bound in Theorem 5.
Proof of Theorem 5 Let b ∗ b^{*} b ∗ be the minimum-norm coefficient representing u ∗ = Φ b ∗ u^{*}=\Phi b^{*} u ∗ = Φ b ∗ . Then
∥ b ∗ ∥ L 2 ( π ) = ∥ u ∗ ∥ H K . \|b^{*}\|_{L^2(\pi)}=\|u^{*}\|_{\mathcal{H}_K}. ∥ b ∗ ∥ L 2 ( π ) = ∥ u ∗ ∥ H K .
By (54) and (A.2),
b ^ n , η , λ − b ∗ = [ S n , P D E ∗ ( S n , P D E S n , P D E ∗ + λ I n ) − 1 S n , P D E − I ] b ∗ = [ ( A n + λ I ) − 1 A n − I ] b ∗ = − λ ( A n + λ I ) − 1 b ∗ . \begin{aligned}
\widehat{b}_{n,\eta,\lambda}-b^{*}
=\left[S_{n,\mathrm{PDE}}^{*}(S_{n,\mathrm{PDE}}S_{n,\mathrm{PDE}}^{*}+\lambda I_n)^{-1}S_{n,\mathrm{PDE}}-I\right]b^{*} \\
&=\left[(A_n+\lambda I)^{-1}A_n-I\right]b^{*} \\
&=-\lambda(A_n+\lambda I)^{-1}b^{*}.
\end{aligned} b n , η , λ − b ∗ = [ S n , PDE ∗ ( S n , PDE S n , PDE ∗ + λ I n ) − 1 S n , PDE − I ] b ∗ = [ ( A n + λ I ) − 1 A n − I ] b ∗ = − λ ( A n + λ I ) − 1 b ∗ .
It follows that
∥ u ^ n , η , λ − u ∗ ∥ R η = ∥ T ~ R η 1 / 2 ( b ^ n , η , λ − b ∗ ) ∥ L 2 ( π ) ≤ λ ∥ T ~ R η 1 / 2 ( A n + λ I ) − 1 ∥ ∥ b ∗ ∥ L 2 ( π ) . \begin{aligned}
\|\widehat{u}_{n,\eta,\lambda}-u^{*}\|_{R_\eta}
=\|\widetilde{\mathcal{T}}_{R_\eta}^{1/2}(\widehat{b}_{n,\eta,\lambda}-b^{*})\|_{L^2(\pi)} \\
&\le\lambda\|\widetilde{\mathcal{T}}_{R_\eta}^{1/2}(A_n+\lambda I)^{-1}\|\|b^{*}\|_{L^2(\pi)}.
\end{aligned} ∥ u n , η , λ − u ∗ ∥ R η = ∥ T R η 1/2 ( b n , η , λ − b ∗ ) ∥ L 2 ( π ) ≤ λ ∥ T R η 1/2 ( A n + λ I ) − 1 ∥∥ b ∗ ∥ L 2 ( π ) .
Fix a deterministic λ n > 0 \lambda_n>0 λ n > 0 satisfying (A.7), specifically
λ n ≥ max { Λ I ( n I , δ / 2 ) , Λ B ( n B , δ / 2 ) } . \lambda_n\ge\max\{\Lambda_I(n_I,\delta/2),\Lambda_B(n_B,\delta/2)\}. λ n ≥ max { Λ I ( n I , δ /2 ) , Λ B ( n B , δ /2 )} .
By Lemmas 22 and 23, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ T ~ R η 1 / 2 ( A n + λ n I ) − 1 / 2 ∥ ≤ ∥ T ~ R η 1 / 2 ( T ~ R η + λ n I ) − 1 / 2 ∥ ∥ ( T ~ R η + λ n I ) 1 / 2 ( A n + λ n I ) − 1 / 2 ∥ ≤ 2. \left\|\widetilde{\mathcal{T}}_{R_\eta}^{1/2}(A_n+\lambda_n I)^{-1/2}\right\|
\le
\left\|\widetilde{\mathcal{T}}_{R_\eta}^{1/2}(\widetilde{\mathcal{T}}_{R_\eta}+\lambda_n I)^{-1/2}\right\|
\left\|(\widetilde{\mathcal{T}}_{R_\eta}+\lambda_n I)^{1/2}(A_n+\lambda_n I)^{-1/2}\right\|
\le2. T R η 1/2 ( A n + λ n I ) − 1/2 ≤ T R η 1/2 ( T R η + λ n I ) − 1/2 ( T R η + λ n I ) 1/2 ( A n + λ n I ) − 1/2 ≤ 2.
On this event, for every 0 < λ ≤ λ n 0<\lambda\le\lambda_n 0 < λ ≤ λ n , Lemma 18 gives
λ ∥ T ~ R η 1 / 2 ( A n + λ I ) − 1 ∥ ≤ λ ∥ T ~ R η 1 / 2 ( A n + λ n I ) − 1 / 2 ∥ × ∥ ( A n + λ n I ) 1 / 2 ( A n + λ I ) − 1 / 2 ∥ ∥ ( A n + λ I ) − 1 / 2 ∥ \begin{aligned}
\lambda\left\|\widetilde{\mathcal{T}}_{R_\eta}^{1/2}(A_n+\lambda I)^{-1}\right\|
\le
\lambda\left\|\widetilde{\mathcal{T}}_{R_\eta}^{1/2}(A_n+\lambda_n I)^{-1/2}\right\| \\
&\times\left\|(A_n+\lambda_n I)^{1/2}(A_n+\lambda I)^{-1/2}\right\|\left\|(A_n+\lambda I)^{-1/2}\right\|
\end{aligned} λ T R η 1/2 ( A n + λ I ) − 1 ≤ λ T R η 1/2 ( A n + λ n I ) − 1/2 × ( A n + λ n I ) 1/2 ( A n + λ I ) − 1/2 ( A n + λ I ) − 1/2
≤ 2 λ ( λ n λ ) 1 / 2 λ − 1 / 2 = 2 λ n 1 / 2 . \le2\lambda\left(\frac{\lambda_n}{\lambda}\right)^{1/2}\lambda^{-1/2}
=2\lambda_n^{1/2}. ≤ 2 λ ( λ λ n ) 1/2 λ − 1/2 = 2 λ n 1/2 .
Consequently,
sup ∥ u ∗ ∥ H k ≤ 1 R η ( u ^ n , η , λ ) ≤ 4 λ n , 0 < λ ≤ λ n . \sup_{\|u^*\|_{H^k}\le1} \mathcal{R}_\eta(\hat{u}_{n,\eta,\lambda}) \le4\lambda_n,\qquad0<\lambda\le\lambda_n. ∥ u ∗ ∥ H k ≤ 1 sup R η ( u ^ n , η , λ ) ≤ 4 λ n , 0 < λ ≤ λ n .
We now choose the sampling densities so that λ n \lambda_n λ n can be selected from the PDE eigenvalues. By Lemma 20, there exists a constant C > 0 C>0 C > 0 such that
N P D E ( μ j P D E ) ≤ C j , j ≥ 1. N_{\mathrm{PDE}}(\mu_j^{\mathrm{PDE}})\le Cj,\qquad j\ge1. N PDE ( μ j PDE ) ≤ C j , j ≥ 1.
Since the sampling proportions 1 / ν I 1/\nu_I 1/ ν I and 1 / ν B 1/\nu_B 1/ ν B are fixed and positive, choose c 0 > 0 c_0>0 c 0 > 0 such that
C c 0 ≤ 1 40 min { 1 / ν I , 1 / ν B } , Cc_0\le\frac{1}{40}\min\{1/\nu_I,1/\nu_B\}, C c 0 ≤ 40 1 min { 1/ ν I , 1/ ν B } ,
and then choose n 0 > 0 n_0>0 n 0 > 0 sufficiently large that ⌊ c 0 n 0 ⌋ ≥ 1 \lfloor c_0n_0\rfloor\ge1 ⌊ c 0 n 0 ⌋ ≥ 1 . For every t ≥ n 0 t\ge n_0 t ≥ n 0 , we therefore have
N P D E ( μ ⌊ c 0 t ⌋ P D E ) ≤ C ⌊ c 0 t ⌋ ≤ C c 0 t ≤ t 40 min { 1 / ν I , 1 / ν B } . N_{\mathrm{PDE}}\left(\mu_{\lfloor c_0t\rfloor}^{\mathrm{PDE}}\right)\le C\lfloor c_0t\rfloor\le Cc_0t\le\frac{t}{40}\min\{1/\nu_I,1/\nu_B\}. N PDE ( μ ⌊ c 0 t ⌋ PDE ) ≤ C ⌊ c 0 t ⌋ ≤ C c 0 t ≤ 40 t min { 1/ ν I , 1/ ν B } .
The constants c 0 c_0 c 0 and n 0 n_0 n 0 are independent of n n n and δ \delta δ . For n log ( 28 n / δ ) ≥ n 0 \frac{n}{\log(28n/\delta)}\ge n_0 l o g ( 28 n / δ ) n ≥ n 0 , set
l n : = ⌊ c 0 n log ( 28 n / δ ) ⌋ , λ n : = μ l n P D E > 0. l_n:=\left\lfloor\frac{c_0n}{\log(28n/\delta)}\right\rfloor,\qquad\lambda_n:=\mu_{l_n}^{\mathrm{PDE}}>0. l n := ⌊ log ( 28 n / δ ) c 0 n ⌋ , λ n := μ l n PDE > 0.
Choose
q I ( x ) = q ∗ ( k L , ρ I , λ n ; x ) = 1 2 + 1 2 N L ( λ n ) ∑ j ≥ 1 μ j L μ j L + λ n ∣ e j L ( x ) ∣ 2 , q_I(x)=q^*(k_{\mathcal{L}},\rho_I,\lambda_n;x)=\frac{1}{2}+\frac{1}{2N_{\mathcal{L}}(\lambda_n)}\sum_{j\ge1}\frac{\mu_j^{\mathcal{L}}}{\mu_j^{\mathcal{L}}+\lambda_n}|e_j^{\mathcal{L}}(x)|^2, q I ( x ) = q ∗ ( k L , ρ I , λ n ; x ) = 2 1 + 2 N L ( λ n ) 1 j ≥ 1 ∑ μ j L + λ n μ j L ∣ e j L ( x ) ∣ 2 ,
and
q B ( x ) = q ∗ ( k B , ρ B , λ n / η ; x ) = 1 2 + 1 2 N B ( λ n / η ) ∑ j ≥ 1 μ j B μ j B + λ n / η ∣ e j B ( x ) ∣ 2 . q_B(x)=q^*(k_B,\rho_B,\lambda_n/\eta;x)=\frac{1}{2}+\frac{1}{2N_B(\lambda_n/\eta)}\sum_{j\ge1}\frac{\mu_j^B}{\mu_j^B+\lambda_n/\eta}|e_j^B(x)|^2. q B ( x ) = q ∗ ( k B , ρ B , λ n / η ; x ) = 2 1 + 2 N B ( λ n / η ) 1 j ≥ 1 ∑ μ j B + λ n / η μ j B ∣ e j B ( x ) ∣ 2 .
By construction,
F L ( λ n ) ≤ 2 N L ( λ n ) , F B ( λ n / η ) ≤ 2 N B ( λ n / η ) . F_{\mathcal{L}}(\lambda_n)\le2N_{\mathcal{L}}(\lambda_n),\qquad F_B(\lambda_n/\eta)\le2N_B(\lambda_n/\eta). F L ( λ n ) ≤ 2 N L ( λ n ) , F B ( λ n / η ) ≤ 2 N B ( λ n / η ) .
Since ν I , ν B ≥ 1 \nu_I,\nu_B\ge1 ν I , ν B ≥ 1 , the decomposition of T ~ P D E \widetilde{T}_{\mathrm{PDE}} T PDE gives
T ~ L ⪯ T ~ P D E , η T ~ B ⪯ T ~ P D E . \widetilde{T}_{\mathcal{L}}\preceq\widetilde{T}_{\mathrm{PDE}},\qquad\eta\widetilde{T}_B\preceq\widetilde{T}_{\mathrm{PDE}}. T L ⪯ T PDE , η T B ⪯ T PDE .
Then
T ~ L ( T ~ L + λ n I ) − 1 = I − λ n ( T ~ L + λ n I ) − 1 ⪯ I − λ n ( T ~ P D E + λ n I ) − 1 = T ~ P D E ( T ~ P D E + λ n I ) − 1 . \widetilde{T}_{\mathcal{L}}(\widetilde{T}_{\mathcal{L}}+\lambda_nI)^{-1}
=I-\lambda_n(\widetilde{T}_{\mathcal{L}}+\lambda_nI)^{-1}
\preceq I-\lambda_n(\widetilde{T}_{\mathrm{PDE}}+\lambda_nI)^{-1}
=\widetilde{T}_{\mathrm{PDE}}(\widetilde{T}_{\mathrm{PDE}}+\lambda_nI)^{-1}. T L ( T L + λ n I ) − 1 = I − λ n ( T L + λ n I ) − 1 ⪯ I − λ n ( T PDE + λ n I ) − 1 = T PDE ( T PDE + λ n I ) − 1 .
Taking traces yields
N L ( λ n ) = tr [ T ~ L ( T ~ L + λ n I ) − 1 ] ≤ N P D E ( λ n ) . N_{\mathcal{L}}(\lambda_n)=\operatorname{tr}\left[\widetilde{T}_{\mathcal{L}}(\widetilde{T}_{\mathcal{L}}+\lambda_nI)^{-1}\right]\le N_{\mathrm{PDE}}(\lambda_n). N L ( λ n ) = tr [ T L ( T L + λ n I ) − 1 ] ≤ N PDE ( λ n ) .
Applying the same argument to η T ~ B \eta\widetilde{T}_B η T B gives
η T ~ B ( η T ~ B + λ n I ) − 1 ⪯ T ~ P D E ( T ~ P D E + λ n I ) − 1 . \eta\widetilde{T}_B(\eta\widetilde{T}_B+\lambda_nI)^{-1}\preceq\widetilde{T}_{\mathrm{PDE}}(\widetilde{T}_{\mathrm{PDE}}+\lambda_nI)^{-1}. η T B ( η T B + λ n I ) − 1 ⪯ T PDE ( T PDE + λ n I ) − 1 .
Consequently,
N B ( λ n / η ) = tr [ T ~ B ( T ~ B + ( λ n / η ) I ) − 1 ] = tr [ η T ~ B ( η T ~ B + λ n I ) − 1 ] ≤ N P D E ( λ n ) . N_B(\lambda_n/\eta)=\operatorname{tr}\left[\widetilde{T}_B\left(\widetilde{T}_B+(\lambda_n/\eta)I\right)^{-1}\right]
=\operatorname{tr}\left[\eta\widetilde{T}_B(\eta\widetilde{T}_B+\lambda_nI)^{-1}\right]
\le N_{\mathrm{PDE}}(\lambda_n). N B ( λ n / η ) = tr [ T B ( T B + ( λ n / η ) I ) − 1 ] = tr [ η T B ( η T B + λ n I ) − 1 ] ≤ N PDE ( λ n ) .
Therefore,
F I ( λ n ) , F B ( λ n ) ≤ 2 N P D E ( λ n ) ≤ min { n I , n B } 20 log ( 28 n / δ ) . F_I(\lambda_n),F_B(\lambda_n)\le2N_{\mathrm{PDE}}(\lambda_n)\le\frac{\min\{n_I,n_B\}}{20\log(28n/\delta)}. F I ( λ n ) , F B ( λ n ) ≤ 2 N PDE ( λ n ) ≤ 20 log ( 28 n / δ ) min { n I , n B } .
For J ∈ { I , B } J\in\{I,B\} J ∈ { I , B } , it follows that
16 F J ( λ n ) max { 1 , log 28 F J ( λ n ) δ } ≤ 4 5 min { n I , n B } ≤ n J . 16F_J(\lambda_n)\max\left\{1,\log\frac{28F_J(\lambda_n)}{\delta}\right\}\le\frac{4}{5}\min\{n_I,n_B\}\le n_J. 16 F J ( λ n ) max { 1 , log δ 28 F J ( λ n ) } ≤ 5 4 min { n I , n B } ≤ n J .
Thus the chosen λ n \lambda_n λ n satisfies (A.7). Therefore with probability at least 1 − δ 1-\delta 1 − δ ,
sup ∥ u ∗ ∥ H k ≤ 1 ∥ u ^ n , η , λ − u ∗ ∥ R η 2 ≤ 4 μ ⌊ c 0 n / log ( 28 n / δ ) ⌋ P D E , 0 < λ ≤ λ n . \sup_{\lVert u^*\rVert_{\mathcal{H}_k}\leq1}\lVert\widehat{u}_{n,\eta,\lambda}-u^*\rVert_{\mathcal{R}_\eta}^2 \leq4\mu^{\mathrm{PDE}}_{\left\lfloor c_0n/\log(28n/\delta)\right\rfloor}, \qquad0<\lambda\leq\lambda_n. ∥ u ∗ ∥ H k ≤ 1 sup ∥ u n , η , λ − u ∗ ∥ R η 2 ≤ 4 μ ⌊ c 0 n / l o g ( 28 n / δ ) ⌋ PDE , 0 < λ ≤ λ n .
We next establish the lower bound on the worst-case estimation error stated in Theorem 6 .
Proof of Theorem 6 Set h ∗ : = P P D E u ∗ h^*:=P_{\mathrm{PDE}}u^* h ∗ := P PDE u ∗ . For h ∈ H P D E h\in\mathcal{H}_{\mathrm{PDE}} h ∈ H PDE , write
h ( X n ) : = ( h ( x 1 ) , … , h ( x n ) ) ⊤ , D ν : = diag ( ν I I n I , ν B I n B ) . h(X_n):=(h(x_1),\ldots,h(x_n))^\top,\qquad D_\nu:=\operatorname{diag}\left(\sqrt{\nu_I}I_{n_I},\sqrt{\nu_B}I_{n_B}\right). h ( X n ) := ( h ( x 1 ) , … , h ( x n ) ) ⊤ , D ν := diag ( ν I I n I , ν B I n B ) .
Then
h ∗ ( X n ) = D ν z n ( X n , u ∗ ) . h^*(X_n)=D_\nu z_n(X_n,u^*). h ∗ ( X n ) = D ν z n ( X n , u ∗ ) .
Since D ν D_\nu D ν is invertible, the PDE observations and the pointwise observations of h ∗ h^* h ∗ determine each other. Each G ∈ M n P D E G\in\mathcal{M}_n^{\mathrm{PDE}} G ∈ M n PDE therefore induces an estimator
Q G h : = P P D E G ( X n , D ν − 1 h ( X n ) ) . Q_Gh:=P_{\mathrm{PDE}}G(X_n,D_\nu^{-1}h(X_n)). Q G h := P PDE G ( X n , D ν − 1 h ( X n )) .
By (59) ,
∥ G ( S n ( u ∗ ) ) − u ∗ ∥ R η 2 ≥ 1 ν + ∥ Q G h ∗ − h ∗ ∥ L 2 ( Ω ‾ ) ‾ 2 . \lVert G(S_n(u^*))-u^*\rVert_{\mathcal{R}_\eta}^2\geq\frac{1}{\nu_+}\lVert Q_Gh^*-h^*\rVert_{\overline{L^2(\overline{\Omega})}}^2. ∥ G ( S n ( u ∗ )) − u ∗ ∥ R η 2 ≥ ν + 1 ∥ Q G h ∗ − h ∗ ∥ L 2 ( Ω ) 2 .
Let M n \mathcal{M}_n M n denote the class of all estimators based only on ( X n , h ( X n ) ) (X_n,h(X_n)) ( X n , h ( X n )) , with outputs in L 2 ( Ω ‾ ) ‾ \overline{L^2(\overline{\Omega})} L 2 ( Ω ) . Since { Q G : G ∈ M n P D E } ⊆ M n \{Q_G:G\in\mathcal{M}_n^{\mathrm{PDE}}\}\subseteq\mathcal{M}_n { Q G : G ∈ M n PDE } ⊆ M n , we obtain
inf G ∈ M n P D E sup ∥ u ∗ ∥ H k ≤ 1 ∥ G ( S n ( u ∗ ) ) − u ∗ ∥ R η 2 ≥ 1 ν + inf Q ∈ M n sup ∥ h ∗ ∥ H P D E ≤ 1 ∥ Q h ∗ − h ∗ ∥ L 2 ( Ω ‾ ) ‾ 2 ≥ μ n + 1 P D E ν + . \begin{aligned}
\inf_{G\in\mathcal{M}_n^{\mathrm{PDE}}}\sup_{\lVert u^*\rVert_{\mathcal{H}_k}\leq1}\lVert G(S_n(u^*))-u^*\rVert_{\mathcal{R}_\eta}^2
\geq\frac{1}{\nu_+}\inf_{Q\in\mathcal{M}_n}\sup_{\lVert h^*\rVert_{\mathcal{H}_{\mathrm{PDE}}}\leq1}\lVert Qh^*-h^*\rVert_{\overline{L^2(\overline{\Omega})}}^2 \\
&\geq\frac{\mu_{n+1}^{\mathrm{PDE}}}{\nu_+}.
\end{aligned} G ∈ M n PDE inf ∥ u ∗ ∥ H k ≤ 1 sup ∥ G ( S n ( u ∗ )) − u ∗ ∥ R η 2 ≥ ν + 1 Q ∈ M n inf ∥ h ∗ ∥ H PDE ≤ 1 sup ∥ Q h ∗ − h ∗ ∥ L 2 ( Ω ) 2 ≥ ν + μ n + 1 PDE .
The last inequality follows from [14 Theorem 4.1] with s = 1 s=1 s = 1 and p = 0 p=0 p = 0 .
We now consider the simultaneous restrictions imposed by finitely many features and finitely many collocation observations. Fix feature parameters θ 1 , … , θ m ∈ Θ \theta_1,\ldots,\theta_m\in\Theta θ 1 , … , θ m ∈ Θ and collocation points
X n = ( x 1 , … , x n ) ∈ Ω n I × ( ∂ Ω ) n B , n = n I + n B . X_n=(x_1,\ldots,x_n)\in\Omega^{n_I}\times(\partial\Omega)^{n_B},\qquad n=n_I+n_B. X n = ( x 1 , … , x n ) ∈ Ω n I × ( ∂ Ω ) n B , n = n I + n B .
Let M n P D E \mathcal{M}_n^{\mathrm{PDE}} M n PDE be the class of estimators based only on collocation observations, as defined in (20) . Let M m , n \mathcal{M}_{m,n} M m , n denote the class of estimators defined in (21) , which use collocation observations and take values in Σ m ϕ \Sigma_m^\phi Σ m ϕ .
To distinguish the restriction imposed by finitely many features from that imposed by finitely many collocation observations, define
E 1 : = sup ∥ u ∗ ∥ H k ≤ 1 inf v ∈ Σ m ϕ ∥ v − u ∗ ∥ R η 2 , E_1:=\sup_{\lVert u^*\rVert_{\mathcal{H}_k}\leq1}\inf_{v\in\Sigma_m^\phi}\lVert v-u^*\rVert_{\mathcal{R}_\eta}^2, E 1 := ∥ u ∗ ∥ H k ≤ 1 sup v ∈ Σ m ϕ inf ∥ v − u ∗ ∥ R η 2 ,
E 2 : = inf G ∈ M n P D E sup ∥ u ∗ ∥ H k ≤ 1 ∥ G ( S n ( u ∗ ) ) − u ∗ ∥ R η 2 . E_2:=\inf_{G\in\mathcal{M}_n^{\mathrm{PDE}}}\sup_{\lVert u^*\rVert_{\mathcal{H}_k}\leq1}\lVert G(S_n(u^*))-u^*\rVert_{\mathcal{R}_\eta}^2. E 2 := G ∈ M n PDE inf ∥ u ∗ ∥ H k ≤ 1 sup ∥ G ( S n ( u ∗ )) − u ∗ ∥ R η 2 .
Here, E 1 E_1 E 1 measures the best approximation error in the random feature model when the target is fully known, whereas E 2 E_2 E 2 measures the optimal recovery error from the collocation observations without restricting the output to Σ m ϕ \Sigma_m^\phi Σ m ϕ . Proof of Corollary 7 For every G ∈ M m , n G \in\mathcal{M}_{m,n} G ∈ M m , n and every u ∗ ∈ H k u^* \in\mathcal{H}_k u ∗ ∈ H k , the inclusion G ( S n ( u ∗ ) ) ∈ Σ m ϕ G(S_n(u^*)) \in\Sigma_m^\phi G ( S n ( u ∗ )) ∈ Σ m ϕ implies
∥ G ( S n ( u ∗ ) ) − u ∗ ∥ R η 2 ≥ inf v ∈ Σ m ϕ ∥ v − u ∗ ∥ R η 2 . \lVert G(S_n(u^*))-u^*\rVert_{\mathcal{R}_\eta}^2 \ge\inf_{v\in\Sigma_m^\phi}\lVert v-u^*\rVert_{\mathcal{R}_\eta}^2. ∥ G ( S n ( u ∗ )) − u ∗ ∥ R η 2 ≥ v ∈ Σ m ϕ inf ∥ v − u ∗ ∥ R η 2 .
Taking the supremum over the unit ball of H k \mathcal{H}_k H k and then the infimum over G ∈ M m , n G \in\mathcal{M}_{m,n} G ∈ M m , n gives E m , n ≥ E 1 E_{m,n} \ge E_1 E m , n ≥ E 1 . Moreover, M m , n ⊆ M n P D E \mathcal{M}_{m,n} \subseteq\mathcal{M}_n^{\mathrm{PDE}} M m , n ⊆ M n PDE , so E m , n ≥ E 2 E_{m,n} \ge E_2 E m , n ≥ E 2 .
It remains to relate these two errors to the PDE spectrum. Since T ~ P D E ⪯ ν + T ~ R η \widetilde{T}_{\mathrm{PDE}} \preceq\nu_+ \widetilde{T}_{\mathcal{R}_\eta} T PDE ⪯ ν + T R η ,
μ j P D E ≤ ν + μ ~ j ( R η ) , j ≥ 1. \mu_j^{\mathrm{PDE}} \le\nu_+ \widetilde{\mu}_j^{(\mathcal{R}_\eta)}, \qquad j \ge1. μ j PDE ≤ ν + μ j ( R η ) , j ≥ 1.
Theorem 4 therefore gives
E 1 ≥ μ ~ m + 1 ( R η ) ≥ μ m + 1 P D E ν + . E_1 \ge\widetilde{\mu}_{m+1}^{(\mathcal{R}_\eta)} \ge\frac{\mu_{m+1}^{\mathrm{PDE}}}{\nu_+}. E 1 ≥ μ m + 1 ( R η ) ≥ ν + μ m + 1 PDE .
On the other hand, Theorem 6 gives
E 2 ≥ μ n + 1 P D E ν + . E_2 \ge\frac{\mu_{n+1}^{\mathrm{PDE}}}{\nu_+}. E 2 ≥ ν + μ n + 1 PDE .
Combining these inequalities proves (23) .
Remark 7 (A two-stage approximator) Fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Choose l n , q I , q B l_n,q_I,q_B l n , q I , q B according to Theorem 5 and l m , p m l_m,p_m l m , p m according to Theorem 3, in both cases replacing the failure probability by δ / 2 \delta/2 δ /2 . Assume that the hypotheses and sample-size requirements of both theorems are satisfied. First, sample the interior and boundary collocation points independently from q I d ρ I q_I\,d\rho_I q I d ρ I and q B d ρ B q_B\,d\rho_B q B d ρ B , respectively, with the two collections independent. Compute u ^ n , η , λ \hat{u}_{n,\eta,\lambda} u ^ n , η , λ according to (17) with some fixed 0 < λ ≤ μ l n P D E 0 < \lambda\le\mu_{l_n}^{\mathrm{PDE}} 0 < λ ≤ μ l n PDE . Next, independently of the collocation points, draw θ 1 , … , θ m \theta_1,\ldots,\theta_m θ 1 , … , θ m independently from p m d π p_m\,d\pi p m d π . Taking u ^ n , η , λ ∈ H k \hat{u}_{n,\eta,\lambda} \in\mathcal{H}_k u ^ n , η , λ ∈ H k as a target function, define
u ^ n m = argmin u ∈ Σ m ϕ ∥ u − u ^ n , η , λ ∥ R η . (62) \hat{u}_n^m = \underset{u\in\Sigma_m^\phi}{\operatorname{argmin}}\lVert u-\hat{u}_{n,\eta,\lambda}\rVert_{\mathcal{R}_\eta}. \tag*{(62)} u ^ n m = u ∈ Σ m ϕ argmin ∥ u − u ^ n , η , λ ∥ R η . (62)
Hence,
∥ u ^ n m − u ∗ ∥ R η 2 ≤ 2 ∥ u ^ n m − u ^ n , η , λ ∥ R η 2 + 2 ∥ u ^ n , η , λ − u ∗ ∥ R η 2 . \lVert\hat{u}_n^m-u^*\rVert_{\mathcal{R}_\eta}^2 \le2\lVert\hat{u}_n^m-\hat{u}_{n,\eta,\lambda}\rVert_{\mathcal{R}_\eta}^2+2\lVert\hat{u}_{n,\eta,\lambda}-u^*\rVert_{\mathcal{R}_\eta}^2. ∥ u ^ n m − u ∗ ∥ R η 2 ≤ 2 ∥ u ^ n m − u ^ n , η , λ ∥ R η 2 + 2 ∥ u ^ n , η , λ − u ∗ ∥ R η 2 .
Therefore, Theorems 5 and 3 imply that, with probability at least 1 − δ 1-\delta 1 − δ ,
sup ∥ u ∗ ∥ H k ≤ 1 ∥ u ^ n m − u ∗ ∥ R η 2 ≤ 8 μ l n P D E + 32 μ ~ l m ( R η ) . \sup_{\lVert u^*\rVert_{\mathcal{H}_k}\le1}\lVert\hat{u}_n^m-u^*\rVert_{\mathcal{R}_\eta}^2 \le8\mu_{l_n}^{\mathrm{PDE}}+32\widetilde{\mu}_{l_m}^{(\mathcal{R}_\eta)}. ∥ u ∗ ∥ H k ≤ 1 sup ∥ u ^ n m − u ∗ ∥ R η 2 ≤ 8 μ l n PDE + 32 μ l m ( R η ) .
This construction differs from the empirical least-squares estimator u ^ n , m , η \hat{u}_{n,m,\eta} u ^ n , m , η defined in (11) , which directly minimizes the weighted empirical residual in (10) over Σ m ϕ \Sigma_m^\phi Σ m ϕ . We leave it to future work to determine whether this estimator attains the same bound under suitable sampling assumptions.
Conclusions
In this work, we analyze RFM within a kernel framework, separating finite-feature approximation from recovery over the full RKHS using finitely many collocation observations. Under the stated assumptions, we establish eigenvalue-based lower bounds and high-probability upper bounds for worst-case errors over the unit ball of H k \mathcal{H}_k H k . The approximation bounds cover Sobolev norms and the PDE residual seminorm, while the estimation bounds are measured in the residual seminorm. Combining the approximation and estimation lower bounds, we obtain a lower bound for reconstruction with m m m features and n n n collocation observations. This bound identifies the separate limitations imposed by the feature space and the available observations. We further investigate how activation regularity affects the relevant eigenvalues, obtaining algebraic eigenvalue upper bounds for finitely smooth activations, and stretched-exponential bounds for analytic activations. The upper bounds in the residual seminorm can also be combined through the two-stage construction in (62) , which first performs recovery over the full RKHS and then projects onto the random feature model using exact residual inner products. Establishing a comparable bound for the direct empirical least-squares RFM estimator in (11) remains an open problem.
Acknowledgements
Acknowledgements. This work is partially supported by the NSFC Major Research Plan - Interpretable and General Purpose Next-generation Artificial Intelligence (Nos. 92570001 and 92370205), NSFC grant 12425113, the Basic Research Program of Jiangsu under Grant BK20253039 and Grant BK20250480, and Key Laboratory of the Ministry of Education for Mathematical Foundations and Applications of Digital Technology, University of Science and Technology of China.
Data Availability
Data Availability. Data will be made available on request.
Declarations
Competing interests
Competing interests. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Auxiliary Results
Singular value decomposition (SVD)
For a compact operator A : H 1 ↦ H 2 A : \mathcal{H}_1 \mapsto\mathcal{H}_2 A : H 1 ↦ H 2 , both A A ∗ AA^* A A ∗ and A ∗ A A^*A A ∗ A are compact, self-adjoint, and positive operators. Moreover, A A ∗ AA^* A A ∗ and A ∗ A A^*A A ∗ A share the same non-zero eigenvalues, and the singular value decomposition of A A A is given by
A = ∑ j = 1 ∞ μ j 1 2 ψ j ′ ⊗ ψ j (63) A = \sum_{j=1}^{\infty} \mu_j^{\frac{1}{2}} \psi'_j \otimes\psi_j \tag*{(63)} A = j = 1 ∑ ∞ μ j 2 1 ψ j ′ ⊗ ψ j (63)
where { μ j } j = 1 ∞ \{\mu_j\}_{j=1}^{\infty} { μ j } j = 1 ∞ are the eigenvalues of A A ∗ AA^* A A ∗ in a decreasing order, { ψ j } j = 1 ∞ \{\psi_j\}_{j=1}^{\infty} { ψ j } j = 1 ∞ and { ψ j ′ } j = 1 ∞ \{\psi'_j\}_{j=1}^{\infty} { ψ j ′ } j = 1 ∞ are the corresponding orthonormal eigenfunctions of A ∗ A A^*A A ∗ A and A A ∗ AA^* A A ∗ , respectively. Through the singular value decomposition, it is easy to prove that for any r ∈ R r \in\mathbb{R} r ∈ R , λ > 0 \lambda> 0 λ > 0 , it holds that
( A ∗ A + λ ) r A ∗ = A ∗ ( A A ∗ + λ ) r . (64) (A^*A + \lambda)^r A^* = A^*(AA^* + \lambda)^r. \tag*{(64)} ( A ∗ A + λ ) r A ∗ = A ∗ ( A A ∗ + λ ) r . (64)
For more information on bounded linear operators in Hilbert spaces, we refer to [37 ] .
Preliminary Results on Singular Values of Compact Operators
Proposition 13 ([15 Proposition 2.1] ) Let H 1 , H 2 \mathcal{H}_1,\mathcal{H}_2 H 1 , H 2 be two Hilbert spaces. Let T : H 1 → H 2 T:\mathcal{H}_1\to\mathcal{H}_2 T : H 1 → H 2 be a compact linear operator. Fix an integer n ≥ 1 n\ge1 n ≥ 1 and let S n \mathcal{S}_n S n be the set of all subspaces of H 1 \mathcal{H}_1 H 1 with co-dimension less than or equal to n − 1 n-1 n − 1 . Then the singular value
s n ( T ) = min X ∈ S n max x ∈ X , ∥ x ∥ = 1 ∥ T x ∥ . (65) s_n(T)=\min_{X\in\mathcal{S}_n}\max_{x\in X,\lVert x\rVert=1}\lVert Tx\rVert. \tag*{(65)} s n ( T ) = X ∈ S n min x ∈ X , ∥ x ∥ = 1 max ∥ T x ∥ . (65)
Proposition 14 ([15 Proposition 2.5] ) Let H 1 , H 2 \mathcal{H}_1,\mathcal{H}_2 H 1 , H 2 be two Hilbert spaces. Let S , T : H 1 → H 2 S,T:\mathcal{H}_1\to\mathcal{H}_2 S , T : H 1 → H 2 be compact linear operators, and let p , q ≥ 1 p,q\ge1 p , q ≥ 1 be two integers. Then
s p + q − 1 ( S + T ) ≤ s p ( S ) + s q ( T ) . (66) s_{p+q-1}(S+T)\le s_p(S)+s_q(T). \tag*{(66)} s p + q − 1 ( S + T ) ≤ s p ( S ) + s q ( T ) . (66)
Proposition 15 Let H 1 \mathcal{H}_1 H 1 and H 2 \mathcal{H}_2 H 2 be Hilbert spaces, and let T i : H 1 → H 2 T_i:\mathcal{H}_1\to\mathcal{H}_2 T i : H 1 → H 2 , i = 1 , … , M i=1,\ldots,M i = 1 , … , M , be compact linear operators. Then, for every n ≥ 1 n\ge1 n ≥ 1 ,
s n ( ∑ i = 1 M T i ) ≤ ∑ i = 1 M s ⌈ n / M ⌉ ( T i ) . s_n\left(\sum_{i=1}^{M}T_i\right)\le\sum_{i=1}^{M}s_{\lceil n/M\rceil}(T_i). s n ( i = 1 ∑ M T i ) ≤ i = 1 ∑ M s ⌈ n / M ⌉ ( T i ) . If, in addition, there exist T max ∈ { T 1 , … , T M } T_{\max}\in\{T_1,\ldots,T_M\} T m a x ∈ { T 1 , … , T M } and a constant C > 0 C>0 C > 0 such that
s k ( T i ) ≤ C s k ( T max ) , i = 1 , … , M , k ≥ 1 , s_k(T_i)\le Cs_k(T_{\max}),\qquad i=1,\ldots,M,\quad k\ge1, s k ( T i ) ≤ C s k ( T m a x ) , i = 1 , … , M , k ≥ 1 , then
s n ( ∑ i = 1 M T i ) ≤ C M s ⌈ n / M ⌉ ( T max ) , n ≥ 1. s_n\left(\sum_{i=1}^{M}T_i\right)\le CMs_{\lceil n/M\rceil}(T_{\max}),\qquad n\ge1. s n ( i = 1 ∑ M T i ) ≤ C M s ⌈ n / M ⌉ ( T m a x ) , n ≥ 1. Proof. Repeated application of Proposition 14 gives
s M ( k − 1 ) + 1 ( ∑ i = 1 M T i ) ≤ ∑ i = 1 M s k ( T i ) , k ≥ 1. s_{M(k-1)+1}\left(\sum_{i=1}^{M}T_i\right)\le\sum_{i=1}^{M}s_k(T_i),\qquad k\ge1. s M ( k − 1 ) + 1 ( i = 1 ∑ M T i ) ≤ i = 1 ∑ M s k ( T i ) , k ≥ 1. For any n ≥ 1 n\ge1 n ≥ 1 , set k = ⌊ n / M ⌋ k=\lfloor n/M\rfloor k = ⌊ n / M ⌋ . Since M ( k − 1 ) + 1 ≤ n M(k-1)+1\le n M ( k − 1 ) + 1 ≤ n , monotonicity of the singular values yields
s n ( ∑ i = 1 M T i ) ≤ s M ( k − 1 ) + 1 ( ∑ i = 1 M T i ) ≤ ∑ i = 1 M s k ( T i ) . s_n\left(\sum_{i=1}^{M}T_i\right)\le s_{M(k-1)+1}\left(\sum_{i=1}^{M}T_i\right)\le\sum_{i=1}^{M}s_k(T_i). s n ( i = 1 ∑ M T i ) ≤ s M ( k − 1 ) + 1 ( i = 1 ∑ M T i ) ≤ i = 1 ∑ M s k ( T i ) . The final assertion follows from s k ( T i ) ≤ C s k ( T max ) s_k(T_i)\le Cs_k(T_{\max}) s k ( T i ) ≤ C s k ( T m a x ) . □
Lemma 16 Let Ω \Omega Ω be a bounded open subset of R d \mathbb{R}^{d} R d or a compact d d d -dimensional C 1 C^1 C 1 manifold embedded in R d ′ \mathbb{R}^{d'} R d ′ , and let Ω ′ \Omega' Ω ′ be an open subset of R d ′ \mathbb{R}^{d'} R d ′ . Integration over Ω \Omega Ω is with respect to Lebesgue measure in the Euclidean case and the induced Riemannian volume measure in the manifold case; both are denoted by d y \mathrm{d}y d y . Let k ∈ L 2 ( Ω ′ × Ω ) k\in L^2(\Omega'\times\Omega) k ∈ L 2 ( Ω ′ × Ω ) with respect to the corresponding product measure, and define
K : L 2 ( Ω ) → L 2 ( Ω ′ ) , ( K φ ) ( x ) = ∫ Ω k ( x , y ) φ ( y ) d y . K:L^2(\Omega)\to L^2(\Omega'),\qquad(K\varphi)(x)=\int_{\Omega}k(x,y)\varphi(y)\,\mathrm{d}y. K : L 2 ( Ω ) → L 2 ( Ω ′ ) , ( K φ ) ( x ) = ∫ Ω k ( x , y ) φ ( y ) d y . Let a ∈ L ∞ ( Ω ) a\in L^\infty(\Omega) a ∈ L ∞ ( Ω ) satisfy ∥ a ∥ L ∞ ( Ω ) ≤ M \lVert a\rVert_{L^\infty(\Omega)}\le M ∥ a ∥ L ∞ ( Ω ) ≤ M for some M ≥ 0 M\ge0 M ≥ 0 , and define
K ~ : L 2 ( Ω ) → L 2 ( Ω ′ ) , ( K ~ φ ~ ) ( x ) = ∫ Ω a ( y ) k ( x , y ) φ ~ ( y ) d y . \widetilde{K}:L^2(\Omega)\to L^2(\Omega'),\qquad(\widetilde{K}\widetilde{\varphi})(x)=\int_{\Omega}a(y)k(x,y)\widetilde{\varphi}(y)\,\mathrm{d}y. K : L 2 ( Ω ) → L 2 ( Ω ′ ) , ( K φ ) ( x ) = ∫ Ω a ( y ) k ( x , y ) φ ( y ) d y . Then
s n ( K ~ ) ≤ M s n ( K ) , n ≥ 1. s_n(\widetilde{K})\le Ms_n(K),\qquad n\ge1. s n ( K ) ≤ M s n ( K ) , n ≥ 1.
Proof. Define M a : L 2 ( Ω ) → L 2 ( Ω ) M_a : L^2(\Omega) \to L^2(\Omega) M a : L 2 ( Ω ) → L 2 ( Ω ) by M a φ = a φ M_a\varphi= a\varphi M a φ = a φ . Then ∥ M a ∥ ≤ M \lVert M_a\rVert\le M ∥ M a ∥ ≤ M and K ~ = K M a \widetilde{K} = KM_a K = K M a . Since K K K is Hilbert–Schmidt, both K K K and K ~ \widetilde{K} K are compact. By the finite-rank approximation characterization of singular values [38 Chapter VI, Theorem 1.5] , for every bounded linear operator Q : L 2 ( Ω ) → L 2 ( Ω ′ ) Q : L^2(\Omega) \to L^2(\Omega') Q : L 2 ( Ω ) → L 2 ( Ω ′ ) with rank ( Q ) < n \operatorname{rank}(Q) < n rank ( Q ) < n , we have
s n ( K ~ ) ≤ ∥ K M a − Q M a ∥ ≤ M ∥ K − Q ∥ , s_n(\widetilde{K}) \le\lVert KM_a-QM_a\rVert\le M\lVert K-Q\rVert, s n ( K ) ≤ ∥ K M a − Q M a ∥ ≤ M ∥ K − Q ∥ ,
where rank ( Q M a ) ≤ rank ( Q ) < n \operatorname{rank}(QM_a) \le\operatorname{rank}(Q) < n rank ( Q M a ) ≤ rank ( Q ) < n . Taking the infimum over such Q Q Q yields s n ( K ~ ) ≤ M s n ( K ) s_n(\widetilde{K}) \le M s_n(K) s n ( K ) ≤ M s n ( K ) .
Theorem 17 ([15 Theorems 1.1 and 1.3] ) Let Ω \Omega Ω be a bounded open set of R d \mathbb{R}^d R d such that ∂ Ω \partial\Omega ∂ Ω is Lipschitz regular. Let Ω ′ \Omega' Ω ′ be an open set of R d ′ \mathbb{R}^{d'} R d ′ and k : Ω ′ × Ω → R k : \Omega' \times\Omega\to\mathbb{R} k : Ω ′ × Ω → R be in L 2 ( Ω ′ ) × H p ( Ω ) L^2(\Omega') \times H^p(\Omega) L 2 ( Ω ′ ) × H p ( Ω ) . Define the linear operator
K : L 2 ( Ω ) → L 2 ( Ω ′ ) , K f ( x ) = ∫ Ω k ( x , y ) f ( y ) d y . K : L^2(\Omega) \to L^2(\Omega'), \qquad Kf(x) = \int_{\Omega} k(x,y)f(y)\,\mathrm{d}y. K : L 2 ( Ω ) → L 2 ( Ω ′ ) , K f ( x ) = ∫ Ω k ( x , y ) f ( y ) d y . Then the singular values satisfy s n ( K ) = o ( n − b d − 1 2 ) s_n(K) = o\left(n^{-\frac{b}{d}-\frac{1}{2}}\right) s n ( K ) = o ( n − d b − 2 1 ) . In addition, if k k k is analytic in a neighborhood of Ω ‾ \overline{\Omega} Ω as a function valued in L 2 ( Ω ′ ) L^2(\Omega') L 2 ( Ω ′ ) , there exist τ ∈ ( 0 , 1 ) \tau\in(0,1) τ ∈ ( 0 , 1 ) and C > 0 C > 0 C > 0 such that
s n ( K ) ≤ C τ n 1 / d . s_n(K) \le C\tau^{n^{1/d}}. s n ( K ) ≤ C τ n 1/ d .
We remark that if k ( x , y ) k(x,y) k ( x , y ) is real analytic for ( x , y ) (x,y) ( x , y ) in a neighborhood of Ω ′ ‾ × Ω ‾ \overline{\Omega'} \times\overline{\Omega} Ω ′ × Ω , it satisfies the analyticity conditions; we refer to [15 Section 4.4] for more details.
Useful Operator Inequalities
Lemma 18 ([14 Lemma A.10] ) Let A A A be a positive self-adjoint compact operator defined over a separable Hilbert space and α , β , γ ∈ ( 0 , ∞ ) \alpha,\beta,\gamma\in(0,\infty) α , β , γ ∈ ( 0 , ∞ ) . If β ≤ α \beta\le\alpha β ≤ α , we have ∥ ( A + α ) γ ( A + β ) − γ ∥ ≤ ( α / β ) γ \lVert(A+\alpha)^{\gamma}(A+\beta)^{-\gamma}\rVert\le(\alpha/\beta)^\gamma ∥( A + α ) γ ( A + β ) − γ ∥ ≤ ( α / β ) γ .
The following concentration inequality originates from [39 Theorem 3.1] . We use the formulation in [14 Theorem A.2] .
Theorem 19 ([39 Theorem 3.1] ; [14 Theorem A.2] ) Let X 1 , … , X n X_1,\ldots,X_n X 1 , … , X n be a sequence of independent self-adjoint random operators on a Hilbert space H \mathcal{H} H such that E X i = 0 \mathbb{E}X_i=0 E X i = 0 for i = 1 , … , n i=1,\ldots,n i = 1 , … , n and ∥ ∑ i = 1 n E X i 2 ∥ ≤ σ 2 \left\lVert\sum_{i=1}^{n}\mathbb{E}X_i^2\right\rVert\le\sigma^2 ∑ i = 1 n E X i 2 ≤ σ 2 . Assume that ∥ X i ∥ ≤ U \lVert X_i\rVert\le U ∥ X i ∥ ≤ U almost surely for all 1 ≤ i ≤ n 1\le i\le n 1 ≤ i ≤ n and some positive U ∈ R U\in\mathbb{R} U ∈ R . Then, for any t ≥ 1 6 ( U + U 2 + 36 σ 2 ) t\ge\frac{1}{6}\left(U+\sqrt{U^2+36\sigma^2}\right) t ≥ 6 1 ( U + U 2 + 36 σ 2 ) ,
P ( ∥ ∑ i = 1 n X i ∥ > t ) ≤ 14 Tr ( ∑ i = 1 n E X i 2 ) σ 2 exp [ − t 2 / 2 σ 2 + t U / 3 ] . (67) \mathbb{P}\left(\left\lVert\sum_{i=1}^{n}X_i\right\rVert>t\right)\le14\frac{\operatorname{Tr}\left(\sum_{i=1}^{n}\mathbb{E}X_i^2\right)}{\sigma^2}\exp\left[-\frac{t^2/2}{\sigma^2+tU/3}\right]. \tag*{(67)} P ( i = 1 ∑ n X i > t ) ≤ 14 σ 2 Tr ( ∑ i = 1 n E X i 2 ) exp [ − σ 2 + t U /3 t 2 /2 ] . (67)
DoF and eigenvalue decay
The behavior of N ( γ ) N(\gamma) N ( γ ) can be explicitly characterized using the eigenvalue decay.
Lemma 20. Let { μ j } j = 1 ∞ \{\mu_j\}_{j=1}^{\infty} { μ j } j = 1 ∞ be a nonincreasing sequence of positive numbers, and define
N ( γ ) : = ∑ j = 1 ∞ μ j μ j + γ , γ > 0. N(\gamma) := \sum_{j=1}^{\infty} \frac{\mu_j}{\mu_j+\gamma}, \qquad\gamma>0. N ( γ ) := j = 1 ∑ ∞ μ j + γ μ j , γ > 0. Suppose that { μ j } j = 1 ∞ \{\mu_j\}_{j=1}^{\infty} { μ j } j = 1 ∞ satisfies Assumption 1. Then, as γ → 0 + \gamma\to0^{+} γ → 0 + ,
N ( γ ) ≍ { γ − 1 / t , if μ j ≲ j − t , [ log ( 1 / γ ) ] d , if μ j ≲ τ − j 1 / d . N(\gamma) \asymp
\begin{cases}
\gamma^{-1/t}, & \text{if }\mu_j \lesssim j^{-t},\\
[\log(1/\gamma)]^d, & \text{if }\mu_j \lesssim\tau^{-j^{1/d}}.
\end{cases} N ( γ ) ≍ { γ − 1/ t , [ log ( 1/ γ ) ] d , if μ j ≲ j − t , if μ j ≲ τ − j 1/ d . Consequently, N ( μ k ) ≲ k N(\mu_k)\lesssim k N ( μ k ) ≲ k for all k ≥ 1 k\ge1 k ≥ 1 .
Proof. The asymptotic estimates for polynomial and exponential decay are established in [14 Lemma 3.3] . Here we provide a direct proof for the stretched exponential case μ j ≲ τ − j 1 / d \mu_j\lesssim\tau^{-j^{1/d}} μ j ≲ τ − j 1/ d , noting that the argument below also recovers the exponential decay regime when d = 1 d=1 d = 1 .
Suppose that μ j ≍ e − a j 1 / d \mu_j\asymp e^{-aj^{1/d}} μ j ≍ e − a j 1/ d with a : = log τ > 0 a:=\log\tau>0 a := log τ > 0 and d ≥ 1 d\ge1 d ≥ 1 . Then there exist constants 0 < c − ≤ c + 0<c_{-}\le c_{+} 0 < c − ≤ c + such that
c − e − a j 1 / d ≤ μ j ≤ c + e − a j 1 / d , j ≥ 1. c_{-}e^{-aj^{1/d}}\le\mu_j\le c_{+}e^{-aj^{1/d}},\qquad j\ge1. c − e − a j 1/ d ≤ μ j ≤ c + e − a j 1/ d , j ≥ 1.
For sufficiently small γ ∈ ( 0 , min { 1 / 2 , c − e − a } ) \gamma\in(0,\min\{1/2,c_{-}e^{-a}\}) γ ∈ ( 0 , min { 1/2 , c − e − a }) , define
J − : = ⌊ ( log ( c − / γ ) a ) d ⌋ ≥ 1 , J + : = ⌈ ( log ( c + / γ ) a ) d ⌉ ≥ 1. J_{-}:=\left\lfloor\left(\frac{\log(c_{-}/\gamma)}{a}\right)^d\right\rfloor\ge1,\qquad
J_{+}:=\left\lceil\left(\frac{\log(c_{+}/\gamma)}{a}\right)^d\right\rceil\ge1. J − := ⌊ ( a log ( c − / γ ) ) d ⌋ ≥ 1 , J + := ⌈ ( a log ( c + / γ ) ) d ⌉ ≥ 1.
For 1 ≤ j ≤ J − 1\le j\le J_{-} 1 ≤ j ≤ J − , monotonicity yields μ j ≥ c − e − a J − 1 / d ≥ γ \mu_j\ge c_{-}e^{-aJ_{-}^{1/d}}\ge\gamma μ j ≥ c − e − a J − 1/ d ≥ γ , so that μ j / ( μ j + γ ) ≥ 1 / 2 \mu_j/(\mu_j+\gamma)\ge1/2 μ j / ( μ j + γ ) ≥ 1/2 . Retaining only the first J − J_{-} J − terms in the series gives the lower bound
N ( γ ) ≥ ∑ j = 1 J − μ j μ j + γ ≥ J − 2 ≳ [ log ( 1 / γ ) ] d . N(\gamma)\ge\sum_{j=1}^{J_{-}}\frac{\mu_j}{\mu_j+\gamma}\ge\frac{J_{-}}{2}\gtrsim[\log(1/\gamma)]^d. N ( γ ) ≥ j = 1 ∑ J − μ j + γ μ j ≥ 2 J − ≳ [ log ( 1/ γ ) ] d .
For the upper bound, we split the series at J + J_{+} J + . Bounding the first J + J_{+} J + terms by 1 and applying integral comparison to the tail, we obtain
N ( γ ) ≤ J + + c + γ ∑ j > J + e − a j 1 / d ≤ J + + c + γ ∫ J + ∞ e − a x 1 / d d x = J + + c + d γ ∫ J + 1 / d ∞ y d − 1 e − a y d y ≤ J + + C c + γ J + ( d − 1 ) / d e − a J + 1 / d ≤ J + + C J + ( d − 1 ) / d ≲ [ log ( 1 / γ ) ] d . \begin{aligned}
N(\gamma)&\le J_{+}+\frac{c_{+}}{\gamma}\sum_{j>J_{+}}e^{-aj^{1/d}}
\le J_{+}+\frac{c_{+}}{\gamma}\int_{J_{+}}^{\infty}e^{-ax^{1/d}}\,dx\\
&=J_{+}+\frac{c_{+}d}{\gamma}\int_{J_{+}^{1/d}}^{\infty}y^{d-1}e^{-ay}\,dy
\le J_{+}+C\frac{c_{+}}{\gamma}J_{+}^{(d-1)/d}e^{-aJ_{+}^{1/d}}\\
&\le J_{+}+CJ_{+}^{(d-1)/d}\lesssim[\log(1/\gamma)]^d.
\end{aligned} N ( γ ) ≤ J + + γ c + j > J + ∑ e − a j 1/ d ≤ J + + γ c + ∫ J + ∞ e − a x 1/ d d x = J + + γ c + d ∫ J + 1/ d ∞ y d − 1 e − a y d y ≤ J + + C γ c + J + ( d − 1 ) / d e − a J + 1/ d ≤ J + + C J + ( d − 1 ) / d ≲ [ log ( 1/ γ ) ] d .
Here, setting Y : = J + 1 / d ≥ 1 Y:=J_{+}^{1/d}\ge1 Y := J + 1/ d ≥ 1 , the second-to-last inequality follows from the exact integration-by-parts identity
∫ Y ∞ y d − 1 e − a y d y = e − a Y ∑ m = 0 d − 1 ( d − 1 ) ! m ! a d − m Y m ≤ ( ∑ m = 0 d − 1 ( d − 1 ) ! m ! a d − m ) Y d − 1 e − a Y , \int_Y^\infty y^{d-1}e^{-ay}\,dy
=e^{-aY}\sum_{m=0}^{d-1}\frac{(d-1)!}{m!a^{d-m}}Y^m
\le\left(\sum_{m=0}^{d-1}\frac{(d-1)!}{m!a^{d-m}}\right)Y^{d-1}e^{-aY}, ∫ Y ∞ y d − 1 e − a y d y = e − aY m = 0 ∑ d − 1 m ! a d − m ( d − 1 )! Y m ≤ ( m = 0 ∑ d − 1 m ! a d − m ( d − 1 )! ) Y d − 1 e − aY ,
with C : = d ∑ m = 0 d − 1 ( d − 1 ) ! m ! a d − m > 0 C:=d\sum_{m=0}^{d-1}\frac{(d-1)!}{m!a^{d-m}}>0 C := d ∑ m = 0 d − 1 m ! a d − m ( d − 1 )! > 0 depending only on a a a and d d d , while the last inequality uses c + e − a J + 1 / d ≤ γ c_{+}e^{-aJ_{+}^{1/d}}\le\gamma c + e − a J + 1/ d ≤ γ . Finally, since μ k → 0 + \mu_k\to0^{+} μ k → 0 + as k → ∞ k\to\infty k → ∞ , substituting γ = μ k \gamma=\mu_k γ = μ k into the above asymptotic bounds on N ( γ ) N(\gamma) N ( γ ) immediately yields N ( μ k ) ≲ k N(\mu_k)\lesssim k N ( μ k ) ≲ k for all k ≥ 1 k\ge1 k ≥ 1 . □
Useful Lemmas in the Proof of Theorem 5
Lemma 21. For every λ > 0 \lambda> 0 λ > 0 ,
F L ( λ ) = ess sup x ∈ Ω ∥ ( T ~ L + λ I ) − 1 / 2 L ϕ ( x , ⋅ ) ∥ L 2 ( π ) 2 q I ( x ) , F_{\mathcal L}(\lambda)=\operatorname{ess\,sup}_{x\in\Omega}\frac{\left\|(\widetilde{\mathcal T}_{\mathcal L}+\lambda I)^{-1/2}\mathcal L\phi(x,\cdot)\right\|_{L^2(\pi)}^2}{q_I(x)}, F L ( λ ) = ess sup x ∈ Ω q I ( x ) ( T L + λ I ) − 1/2 L ϕ ( x , ⋅ ) L 2 ( π ) 2 , and
F B ( λ ) = ess sup x ∈ ∂ Ω ∥ ( T ~ B + λ I ) − 1 / 2 B ϕ ( x , ⋅ ) ∥ L 2 ( π ) 2 q B ( x ) . F_{\mathcal B}(\lambda)=\operatorname{ess\,sup}_{x\in\partial\Omega}\frac{\left\|(\widetilde{\mathcal T}_{\mathcal B}+\lambda I)^{-1/2}\mathcal B\phi(x,\cdot)\right\|_{L^2(\pi)}^2}{q_B(x)}. F B ( λ ) = ess sup x ∈ ∂ Ω q B ( x ) ( T B + λ I ) − 1/2 B ϕ ( x , ⋅ ) L 2 ( π ) 2 . Proof. We prove the first identity; the second follows by the same argument. Write the singular value decompositions as
Φ L = ∑ j ≥ 1 ( μ j L ) 1 / 2 e j L ⊗ e ~ j L , Φ L ∗ = ∑ j ≥ 1 ( μ j L ) 1 / 2 e ~ j L ⊗ e j L , (68) \Phi_{\mathcal L}=\sum_{j\geq1}(\mu_j^{\mathcal L})^{1/2}e_j^{\mathcal L}\otimes\widetilde e_j^{\mathcal L},\qquad
\Phi_{\mathcal L}^*=\sum_{j\geq1}(\mu_j^{\mathcal L})^{1/2}\widetilde e_j^{\mathcal L}\otimes e_j^{\mathcal L}, \tag*{(68)} Φ L = j ≥ 1 ∑ ( μ j L ) 1/2 e j L ⊗ e j L , Φ L ∗ = j ≥ 1 ∑ ( μ j L ) 1/2 e j L ⊗ e j L , (68) where { e j L } j ≥ 1 ⊂ L 2 ( ρ I ) \{e_j^{\mathcal L}\}_{j\geq1}\subset L^2(\rho_I) { e j L } j ≥ 1 ⊂ L 2 ( ρ I ) and { e ~ j L } j ≥ 1 ⊂ L 2 ( π ) \{\widetilde e_j^{\mathcal L}\}_{j\geq1}\subset L^2(\pi) { e j L } j ≥ 1 ⊂ L 2 ( π ) are orthonormal eigenfunctions of Φ L Φ L ∗ \Phi_{\mathcal L}\Phi_{\mathcal L}^* Φ L Φ L ∗ and T ~ L = Φ L ∗ Φ L \widetilde{\mathcal T}_{\mathcal L}=\Phi_{\mathcal L}^*\Phi_{\mathcal L} T L = Φ L ∗ Φ L , respectively. As in [14 Lemma D.2] , for ρ I \rho_I ρ I -almost every x x x ,
L ϕ ( x , ⋅ ) = ∑ j ≥ 1 ( μ j L ) 1 / 2 e j L ( x ) e ~ j L ( ⋅ ) in L 2 ( π ) . \mathcal L\phi(x,\cdot)=\sum_{j\geq1}(\mu_j^{\mathcal L})^{1/2}e_j^{\mathcal L}(x)\widetilde e_j^{\mathcal L}(\cdot)\quad\text{in }L^2(\pi). L ϕ ( x , ⋅ ) = j ≥ 1 ∑ ( μ j L ) 1/2 e j L ( x ) e j L ( ⋅ ) in L 2 ( π ) . Consequently,
∥ ( T ~ L + λ I ) − 1 / 2 L ϕ ( x , ⋅ ) ∥ L 2 ( π ) 2 = ∥ ∑ j ≥ 1 ( μ j L μ j L + λ ) 1 / 2 e j L ( x ) e ~ j L ( ⋅ ) ∥ L 2 ( π ) 2 \left\|(\widetilde{\mathcal T}_{\mathcal L}+\lambda I)^{-1/2}\mathcal L\phi(x,\cdot)\right\|_{L^2(\pi)}^2
=
\left\|\sum_{j\geq1}\left(\frac{\mu_j^{\mathcal L}}{\mu_j^{\mathcal L}+\lambda}\right)^{1/2}e_j^{\mathcal L}(x)\widetilde e_j^{\mathcal L}(\cdot)\right\|_{L^2(\pi)}^2 ( T L + λ I ) − 1/2 L ϕ ( x , ⋅ ) L 2 ( π ) 2 = j ≥ 1 ∑ ( μ j L + λ μ j L ) 1/2 e j L ( x ) e j L ( ⋅ ) L 2 ( π ) 2 = ∑ j ≥ 1 μ j L μ j L + λ ∣ e j L ( x ) ∣ 2 . =\sum_{j\geq1}\frac{\mu_j^{\mathcal L}}{\mu_j^{\mathcal L}+\lambda}\left|e_j^{\mathcal L}(x)\right|^2. = j ≥ 1 ∑ μ j L + λ μ j L e j L ( x ) 2 . Dividing by q I ( x ) q_I(x) q I ( x ) and taking the essential supremum proves the claim. □
Lemma 22 (Concentration of the empirical operator) Let δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and λ > 0 \lambda>0 λ > 0 . If
λ ≥ max { Λ I ( n I , δ / 2 ) , Λ B ( n B , δ / 2 ) } , (69) \lambda\geq\max\{\Lambda_I(n_I,\delta/2),\Lambda_B(n_B,\delta/2)\}, \tag*{(69)} λ ≥ max { Λ I ( n I , δ /2 ) , Λ B ( n B , δ /2 )} , (69) then, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ ( T ~ R η + λ I ) − 1 / 2 ( A n − T ~ R η ) ( T ~ R η + λ I ) − 1 / 2 ∥ ≤ 3 4 . (70) \left\|(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2}(A_n-\widetilde{\mathcal T}_{R_\eta})(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2}\right\|\leq\frac{3}{4}. \tag*{(70)} ( T R η + λ I ) − 1/2 ( A n − T R η ) ( T R η + λ I ) − 1/2 ≤ 4 3 . (70) Proof. Let Δ n = ( T ~ R η + λ I ) − 1 / 2 ( A n − T ~ R η ) ( T ~ R η + λ I ) − 1 / 2 \Delta_n=(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2}(A_n-\widetilde{\mathcal T}_{R_\eta})(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2} Δ n = ( T R η + λ I ) − 1/2 ( A n − T R η ) ( T R η + λ I ) − 1/2 . By the triangle inequality,
∥ Δ n ∥ ≤ ∥ ( T ~ R η + λ I ) − 1 / 2 ( T ~ L , n I − T ~ L ) ( T ~ R η + λ I ) − 1 / 2 ∥ + ∥ ( T ~ R η + λ I ) − 1 / 2 η ( T ~ B , n B − T ~ B ) ( T ~ R η + λ I ) − 1 / 2 ∥ = : M 1 + M 2 . \begin{aligned}
\|\Delta_n\|
&\leq\left\|(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2}(\widetilde{\mathcal T}_{\mathcal L,n_I}-\widetilde{\mathcal T}_{\mathcal L})(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2}\right\|\\
&\quad+\left\|(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2}\eta(\widetilde{\mathcal T}_{\mathcal B,n_B}-\widetilde{\mathcal T}_{\mathcal B})(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2}\right\|\\
&=:M_1+M_2.
\end{aligned} ∥ Δ n ∥ ≤ ( T R η + λ I ) − 1/2 ( T L , n I − T L ) ( T R η + λ I ) − 1/2 + ( T R η + λ I ) − 1/2 η ( T B , n B − T B ) ( T R η + λ I ) − 1/2 =: M 1 + M 2 . Bounding M 1 M_1 M 1 . Define
h i I = ( T ~ R η + λ I ) − 1 / 2 L ϕ ( x i , ⋅ ) q I ( x i ) . h_i^I=\frac{(\widetilde{\mathcal T}_{R_\eta}+\lambda I)^{-1/2}\mathcal L\phi(x_i,\cdot)}{\sqrt{q_I(x_i)}}. h i I = q I ( x i ) ( T R η + λ I ) − 1/2 L ϕ ( x i , ⋅ ) .
and
Z i I : = 1 n I [ h i I ⊗ h i I − ( T ~ R η + λ I ) − 1 / 2 T ~ L ( T ~ R η + λ I ) − 1 / 2 ] . Z_i^I := \frac{1}{n_I}\left[h_i^I \otimes h_i^I - (\widetilde{T}_{R_\eta}+\lambda I)^{-1/2}\widetilde{T}_{\mathcal{L}}(\widetilde{T}_{R_\eta}+\lambda I)^{-1/2}\right]. Z i I := n I 1 [ h i I ⊗ h i I − ( T R η + λ I ) − 1/2 T L ( T R η + λ I ) − 1/2 ] .
Note that
E [ h i I ⊗ h i I ] = ( T ~ R η + λ I ) − 1 / 2 T ~ L ( T ~ R η + λ I ) − 1 / 2 , \mathbb{E}[h_i^I \otimes h_i^I] = (\widetilde{T}_{R_\eta}+\lambda I)^{-1/2}\widetilde{T}_{\mathcal{L}}(\widetilde{T}_{R_\eta}+\lambda I)^{-1/2}, E [ h i I ⊗ h i I ] = ( T R η + λ I ) − 1/2 T L ( T R η + λ I ) − 1/2 ,
the Z i I Z_i^I Z i I are independent, centered, self-adjoint operators, and M 1 = ∥ ∑ i = 1 n I Z i I ∥ M_1 = \left\|\sum_{i=1}^{n_I} Z_i^I\right\| M 1 = ∑ i = 1 n I Z i I .
Since T ~ L ≼ T ~ R η \widetilde{T}_{\mathcal{L}} \preccurlyeq\widetilde{T}_{R_\eta} T L ≼ T R η , ( T ~ R η + λ I ) − 1 ≼ ( T ~ L + λ I ) − 1 (\widetilde{T}_{R_\eta}+\lambda I)^{-1} \preccurlyeq(\widetilde{T}_{\mathcal{L}}+\lambda I)^{-1} ( T R η + λ I ) − 1 ≼ ( T L + λ I ) − 1 . Combining this inequality with Lemma 21 gives
∥ h i I ∥ L 2 ( π ) 2 = ⟨ L ϕ ( x i , ⋅ ) , ( T ~ R η + λ I ) − 1 L ϕ ( x i , ⋅ ) ⟩ L 2 ( π ) q I ( x i ) ≤ ∥ ( T ~ L + λ I ) − 1 / 2 L ϕ ( x i , ⋅ ) ∥ L 2 ( π ) 2 q I ( x i ) ≤ F L ( λ ) ≤ F I ( λ ) , \begin{aligned}
\|h_i^I\|_{L^2(\pi)}^2 =
\frac{\langle\mathcal{L}\phi(x_i,\cdot),(\widetilde{T}_{R_\eta}+\lambda I)^{-1}\mathcal{L}\phi(x_i,\cdot)\rangle_{L^2(\pi)}}{q_I(x_i)} \\
&\leq\frac{\|(\widetilde{T}_{\mathcal{L}}+\lambda I)^{-1/2}\mathcal{L}\phi(x_i,\cdot)\|_{L^2(\pi)}^2}{q_I(x_i)} \\
&\leq F_{\mathcal{L}}(\lambda) \leq F_I(\lambda),
\end{aligned} ∥ h i I ∥ L 2 ( π ) 2 = q I ( x i ) ⟨ L ϕ ( x i , ⋅ ) , ( T R η + λ I ) − 1 L ϕ ( x i , ⋅ ) ⟩ L 2 ( π ) ≤ q I ( x i ) ∥ ( T L + λ I ) − 1/2 L ϕ ( x i , ⋅ ) ∥ L 2 ( π ) 2 ≤ F L ( λ ) ≤ F I ( λ ) ,
almost surely. Hence ∥ h i I ⊗ h i I ∥ ≤ F I ( λ ) \|h_i^I \otimes h_i^I\| \leq F_I(\lambda) ∥ h i I ⊗ h i I ∥ ≤ F I ( λ ) and ∥ E [ h i I ⊗ h i I ] ∥ ≤ F I ( λ ) \|\mathbb{E}[h_i^I \otimes h_i^I]\| \leq F_I(\lambda) ∥ E [ h i I ⊗ h i I ] ∥ ≤ F I ( λ ) . Thus
∥ Z i I ∥ ≤ F I ( λ ) n I . \|Z_i^I\| \leq\frac{F_I(\lambda)}{n_I}. ∥ Z i I ∥ ≤ n I F I ( λ ) .
Moreover,
E [ ( Z i I ) 2 ] ≼ 1 n I 2 E [ ∥ h i I ∥ L 2 ( π ) 2 h i I ⊗ h i I ] \mathbb{E}[(Z_i^I)^2] \preccurlyeq\frac{1}{n_I^2}\mathbb{E}\left[\|h_i^I\|_{L^2(\pi)}^2 h_i^I \otimes h_i^I\right] E [( Z i I ) 2 ] ≼ n I 2 1 E [ ∥ h i I ∥ L 2 ( π ) 2 h i I ⊗ h i I ]
≼ F I ( λ ) n I 2 ( T ~ R η + λ I ) − 1 / 2 T ~ L ( T ~ R η + λ I ) − 1 / 2 \preccurlyeq\frac{F_I(\lambda)}{n_I^2}(\widetilde{T}_{R_\eta}+\lambda I)^{-1/2}\widetilde{T}_{\mathcal{L}}(\widetilde{T}_{R_\eta}+\lambda I)^{-1/2} ≼ n I 2 F I ( λ ) ( T R η + λ I ) − 1/2 T L ( T R η + λ I ) − 1/2
≼ F I ( λ ) n I 2 T ~ R η ( T ~ R η + λ I ) − 1 . \preccurlyeq\frac{F_I(\lambda)}{n_I^2}\widetilde{T}_{R_\eta}(\widetilde{T}_{R_\eta}+\lambda I)^{-1}. ≼ n I 2 F I ( λ ) T R η ( T R η + λ I ) − 1 .
Since ∥ T ~ R η ( T ~ R η + λ I ) − 1 ∥ ≤ 1 \|\widetilde{T}_{R_\eta}(\widetilde{T}_{R_\eta}+\lambda I)^{-1}\| \leq1 ∥ T R η ( T R η + λ I ) − 1 ∥ ≤ 1 ,
∥ ∑ i = 1 n I E [ ( Z i I ) 2 ] ∥ ≤ F I ( λ ) n I . \left\|\sum_{i=1}^{n_I}\mathbb{E}[(Z_i^I)^2]\right\| \leq\frac{F_I(\lambda)}{n_I}. i = 1 ∑ n I E [( Z i I ) 2 ] ≤ n I F I ( λ ) .
Furthermore, T ~ R η ≼ 1 ν − T ~ P D E \widetilde{T}_{R_\eta} \preccurlyeq\frac{1}{\nu_-}\widetilde{T}_{\mathrm{PDE}} T R η ≼ ν − 1 T PDE implies
T ~ R η ( T ~ R η + λ I ) − 1 ≼ T ~ P D E ( T ~ P D E + ν − λ I ) − 1 . \widetilde{T}_{R_\eta}(\widetilde{T}_{R_\eta}+\lambda I)^{-1} \preccurlyeq\widetilde{T}_{\mathrm{PDE}}(\widetilde{T}_{\mathrm{PDE}}+\nu_-\lambda I)^{-1}. T R η ( T R η + λ I ) − 1 ≼ T PDE ( T PDE + ν − λ I ) − 1 .
Taking traces yields
tr [ T ~ R η ( T ~ R η + λ I ) − 1 ] ≤ N P D E ( ν − λ ) ≤ N P D E ( λ ) , \operatorname{tr}\left[\widetilde{T}_{R_\eta}(\widetilde{T}_{R_\eta}+\lambda I)^{-1}\right] \leq N_{\mathrm{PDE}}(\nu_-\lambda) \leq N_{\mathrm{PDE}}(\lambda), tr [ T R η ( T R η + λ I ) − 1 ] ≤ N PDE ( ν − λ ) ≤ N PDE ( λ ) ,
where the last inequality follows from ν − ≥ 1 \nu_- \geq1 ν − ≥ 1 and the monotonicity of N P D E N_{\mathrm{PDE}} N PDE . Thus
tr ( ∑ i = 1 n I E [ ( Z i I ) 2 ] ) ≤ F I ( λ ) N P D E ( λ ) n I ≤ F I ( λ ) 2 n I . \operatorname{tr}\left(\sum_{i=1}^{n_I}\mathbb{E}[(Z_i^I)^2]\right) \leq\frac{F_I(\lambda)N_{\mathrm{PDE}}(\lambda)}{n_I} \leq\frac{F_I(\lambda)^2}{n_I}. tr ( i = 1 ∑ n I E [( Z i I ) 2 ] ) ≤ n I F I ( λ ) N PDE ( λ ) ≤ n I F I ( λ ) 2 .
Applying Theorem 19 with U = σ 2 = F I ( λ ) / n I U=\sigma^2=F_I(\lambda)/n_I U = σ 2 = F I ( λ ) / n I yields
P ( M 1 > t ) ≤ 14 F I ( λ ) exp ( − 3 n I t 2 F I ( λ ) ( 6 + 2 t ) ) , \mathbb{P}(M_1>t) \leq14F_I(\lambda)\exp\left(-\frac{3n_I t^2}{F_I(\lambda)(6+2t)}\right), P ( M 1 > t ) ≤ 14 F I ( λ ) exp ( − F I ( λ ) ( 6 + 2 t ) 3 n I t 2 ) ,
provided
3 n I t 2 ≥ 3 F I ( λ ) + F I ( λ ) t . 3n_I t^2 \geq3F_I(\lambda)+F_I(\lambda)t. 3 n I t 2 ≥ 3 F I ( λ ) + F I ( λ ) t .
For t = 3 / 8 t = 3/8 t = 3/8 , this condition is n I ≥ 8 F I ( λ ) n_I \ge8F_I(\lambda) n I ≥ 8 F I ( λ ) , and exp [ − 3 n I t 2 / ( F I ( λ ) ( 6 + 2 t ) ) ] = exp [ − n I / ( 16 F I ( λ ) ) ] \exp\left[-3n_I t^2/\left(F_I(\lambda)(6+2t)\right)\right] = \exp\left[-n_I/\left(16F_I(\lambda)\right)\right] exp [ − 3 n I t 2 / ( F I ( λ ) ( 6 + 2 t ) ) ] = exp [ − n I / ( 16 F I ( λ ) ) ] . Thus,
n I ≥ 16 F I ( λ ) max { 1 , log 28 F I ( λ ) δ } (71) n_I \ge16F_I(\lambda)\max\left\{1,\log\frac{28F_I(\lambda)}{\delta}\right\} \tag*{(71)} n I ≥ 16 F I ( λ ) max { 1 , log δ 28 F I ( λ ) } (71)
implies P ( M 1 ≤ 3 / 8 ) ≥ 1 − δ / 2 \mathbb{P}(M_1 \le3/8) \ge1-\delta/2 P ( M 1 ≤ 3/8 ) ≥ 1 − δ /2 .
Bounding M 2 M_2 M 2 . Define
h i B : = η q B ( x n I + i ) ( T ~ R η + λ I ) − 1 / 2 B ϕ ( x n I + i , ⋅ ) , h_i^B := \sqrt{\frac{\eta}{q_B(x_{n_I+i})}}\left(\widetilde{\mathcal{T}}_{R_\eta}+\lambda I\right)^{-1/2}B\phi(x_{n_I+i},\cdot), h i B := q B ( x n I + i ) η ( T R η + λ I ) − 1/2 B ϕ ( x n I + i , ⋅ ) ,
and
Z i B : = 1 n B [ h i B ⊗ h i B − η ( T ~ R η + λ I ) − 1 / 2 T ~ B ( T ~ R η + λ I ) − 1 / 2 ] . Z_i^B := \frac{1}{n_B}\left[h_i^B\otimes h_i^B-\eta\left(\widetilde{\mathcal{T}}_{R_\eta}+\lambda I\right)^{-1/2}\widetilde{\mathcal{T}}_B\left(\widetilde{\mathcal{T}}_{R_\eta}+\lambda I\right)^{-1/2}\right]. Z i B := n B 1 [ h i B ⊗ h i B − η ( T R η + λ I ) − 1/2 T B ( T R η + λ I ) − 1/2 ] .
These operators are independent and centered, and M 2 = ∥ ∑ i = 1 n B Z i B ∥ M_2=\left\|\sum_{i=1}^{n_B}Z_i^B\right\| M 2 = ∑ i = 1 n B Z i B .
Since η T ~ B ⪯ T ~ R η \eta\widetilde{\mathcal{T}}_B\preceq\widetilde{\mathcal{T}}_{R_\eta} η T B ⪯ T R η ,
∥ h i B ∥ L 2 ( π ) 2 ≤ η q B ( x n I + i ) ⟨ B ϕ ( x n I + i , ⋅ ) , ( η T ~ B + λ I ) − 1 B ϕ ( x n I + i , ⋅ ) ⟩ L 2 ( π ) = ∥ ( T ~ B + ( λ / η ) I ) − 1 / 2 B ϕ ( x n I + i , ⋅ ) ∥ L 2 ( π ) 2 q B ( x n I + i ) ≤ F B ( λ / η ) ≤ F B ( λ ) , \begin{aligned}
\left\|h_i^B\right\|_{L^2(\pi)}^2\le\frac{\eta}{q_B(x_{n_I+i})}\left\langle B\phi(x_{n_I+i},\cdot),(\eta\widetilde{\mathcal{T}}_B+\lambda I)^{-1}B\phi(x_{n_I+i},\cdot)\right\rangle_{L^2(\pi)} \\
&=\frac{\left\|(\widetilde{\mathcal{T}}_B+(\lambda/\eta)I)^{-1/2}B\phi(x_{n_I+i},\cdot)\right\|_{L^2(\pi)}^2}{q_B(x_{n_I+i})} \\
&\le F_B(\lambda/\eta)\le F_B(\lambda),
\end{aligned} h i B L 2 ( π ) 2 ≤ q B ( x n I + i ) η ⟨ B ϕ ( x n I + i , ⋅ ) , ( η T B + λ I ) − 1 B ϕ ( x n I + i , ⋅ ) ⟩ L 2 ( π ) = q B ( x n I + i ) ( T B + ( λ / η ) I ) − 1/2 B ϕ ( x n I + i , ⋅ ) L 2 ( π ) 2 ≤ F B ( λ / η ) ≤ F B ( λ ) ,
almost surely. The same moment calculation gives
∥ Z i B ∥ ≤ F B ( λ ) n B , \left\|Z_i^B\right\|\le\frac{F_B(\lambda)}{n_B}, Z i B ≤ n B F B ( λ ) ,
and
∑ i = 1 n B E [ ( Z i B ) 2 ] ⪯ F B ( λ ) n B T ~ R η ( T ~ R η + λ I ) − 1 . \sum_{i=1}^{n_B}\mathbb{E}\left[(Z_i^B)^2\right]\preceq\frac{F_B(\lambda)}{n_B}\widetilde{\mathcal{T}}_{R_\eta}\left(\widetilde{\mathcal{T}}_{R_\eta}+\lambda I\right)^{-1}. i = 1 ∑ n B E [ ( Z i B ) 2 ] ⪯ n B F B ( λ ) T R η ( T R η + λ I ) − 1 .
Therefore,
∥ ∑ i = 1 n B E [ ( Z i B ) 2 ] ∥ ≤ F B ( λ ) n B , tr ( ∑ i = 1 n B E [ ( Z i B ) 2 ] ) ≤ F B ( λ ) 2 n B . \left\|\sum_{i=1}^{n_B}\mathbb{E}\left[(Z_i^B)^2\right]\right\|\le\frac{F_B(\lambda)}{n_B},\qquad
\operatorname{tr}\left(\sum_{i=1}^{n_B}\mathbb{E}\left[(Z_i^B)^2\right]\right)\le\frac{F_B(\lambda)^2}{n_B}. i = 1 ∑ n B E [ ( Z i B ) 2 ] ≤ n B F B ( λ ) , tr ( i = 1 ∑ n B E [ ( Z i B ) 2 ] ) ≤ n B F B ( λ ) 2 .
Another application of Theorem 19 with t = 3 / 8 t=3/8 t = 3/8 shows that
n B ≥ 16 F B ( λ ) max { 1 , log 28 F B ( λ ) δ } (72) n_B\ge16F_B(\lambda)\max\left\{1,\log\frac{28F_B(\lambda)}{\delta}\right\} \tag*{(72)} n B ≥ 16 F B ( λ ) max { 1 , log δ 28 F B ( λ ) } (72)
implies P ( M 2 ≤ 3 / 8 ) ≥ 1 − δ / 2 \mathbb{P}(M_2\le3/8)\ge1-\delta/2 P ( M 2 ≤ 3/8 ) ≥ 1 − δ /2 .
The condition (A.7) ensures that (A.9) and (A.10) hold. The union bound now gives
P ( ∥ Δ n ∥ ≤ 3 4 ) ≥ P ( M 1 ≤ 3 8 , M 2 ≤ 3 8 ) ≥ 1 − δ . \mathbb{P}\left(\|\Delta_n\|\le\frac{3}{4}\right)\ge\mathbb{P}\left(M_1\le\frac{3}{8},\,M_2\le\frac{3}{8}\right)\ge1-\delta. P ( ∥ Δ n ∥ ≤ 4 3 ) ≥ P ( M 1 ≤ 8 3 , M 2 ≤ 8 3 ) ≥ 1 − δ .
Lemma 23 On the event (A.8),
∥ ( T ~ R η + λ I ) 1 / 2 ( A n + λ I ) − 1 / 2 ∥ ≤ 2. \left\|\left(\widetilde{\mathcal{T}}_{R_\eta}+\lambda I\right)^{1/2}(A_n+\lambda I)^{-1/2}\right\|\le2. ( T R η + λ I ) 1/2 ( A n + λ I ) − 1/2 ≤ 2. :::`
Proof. Since the operator in (A.8) is self-adjoint, its norm bound implies
( R ~ η + λ I ) − 1 / 2 ( R ~ η − A n ) ( R ~ η + λ I ) − 1 / 2 ≼ 3 4 I . (\widetilde{\mathcal{R}}_{\eta}+\lambda I)^{-1/2}(\widetilde{\mathcal{R}}_{\eta}-A_n)(\widetilde{\mathcal{R}}_{\eta}+\lambda I)^{-1/2}\preccurlyeq\frac{3}{4}I. ( R η + λ I ) − 1/2 ( R η − A n ) ( R η + λ I ) − 1/2 ≼ 4 3 I .
Multiplying both sides of this inequality on the left and right by ( R ~ η + λ I ) 1 / 2 (\widetilde{\mathcal{R}}_{\eta}+\lambda I)^{1/2} ( R η + λ I ) 1/2 gives
R ~ η − A n ≼ 3 4 ( R ~ η + λ I ) . \widetilde{\mathcal{R}}_{\eta}-A_n\preccurlyeq\frac{3}{4}(\widetilde{\mathcal{R}}_{\eta}+\lambda I). R η − A n ≼ 4 3 ( R η + λ I ) .
Hence
A n + λ I ⪰ 1 4 ( R ~ η + λ I ) . A_n+\lambda I\succeq\frac{1}{4}(\widetilde{\mathcal{R}}_{\eta}+\lambda I). A n + λ I ⪰ 4 1 ( R η + λ I ) .
Multiplying both sides of this inequality on the left and right by ( A n + λ I ) − 1 / 2 (A_n+\lambda I)^{-1/2} ( A n + λ I ) − 1/2 gives
( A n + λ I ) − 1 / 2 ( R ~ η + λ I ) ( A n + λ I ) − 1 / 2 ≼ 4 I . (A_n+\lambda I)^{-1/2}(\widetilde{\mathcal{R}}_{\eta}+\lambda I)(A_n+\lambda I)^{-1/2}\preccurlyeq4I. ( A n + λ I ) − 1/2 ( R η + λ I ) ( A n + λ I ) − 1/2 ≼ 4 I .
Taking operator norms proves the claim. □
Proofs of Activation-Dependent Eigenvalue Bounds
Proof of Theorem 8
We first introduce a criterion that converts finite-rank approximation estimates in the Hilbert–Schmidt norm into singular-value decay bounds.
Lemma 24 (Spectral decay from finite-rank approximation) Let T : H 1 → H 2 T:H_1\to H_2 T : H 1 → H 2 be Hilbert–Schmidt between separable Hilbert spaces, and let d ≥ 1 d\ge1 d ≥ 1 and s > 0 s>0 s > 0 . Suppose that for all sufficiently small dyadic h = 2 − ℓ h=2^{-\ell} h = 2 − ℓ there is a bounded linear operator T h : H 1 → H 2 T_h:H_1\to H_2 T h : H 1 → H 2 such that
rank T h ≤ C 0 h − d , ∥ T − T h ∥ H S ≤ C 1 h s , \operatorname{rank} T_h\le C_0h^{-d},\qquad\lVert T-T_h\rVert_{\mathrm{HS}}\le C_1h^s, rank T h ≤ C 0 h − d , ∥ T − T h ∥ HS ≤ C 1 h s , with constants independent of h h h . Then s j ( T ) = O ( j − 1 / 2 − s / d ) s_j(T)=O(j^{-1/2-s/d}) s j ( T ) = O ( j − 1/2 − s / d ) . If instead ∥ T − T h ∥ H S = o ( h s ) \lVert T-T_h\rVert_{\mathrm{HS}}=o(h^s) ∥ T − T h ∥ HS = o ( h s ) , then s j ( T ) = o ( j − 1 / 2 − s / d ) s_j(T)=o(j^{-1/2-s/d}) s j ( T ) = o ( j − 1/2 − s / d ) .
Proof. Set N h : = ⌈ max { C 0 , 1 } h − d ⌉ N_h:=\lceil\max\{C_0,1\}h^{-d}\rceil N h := ⌈ max { C 0 , 1 } h − d ⌉ . Since s N h + 1 ( T h ) = 0 s_{N_h+1}(T_h)=0 s N h + 1 ( T h ) = 0 , Proposition 14 gives
s 2 N h ( T ) ≤ s N h ( T − T h ) ≤ N h − 1 / 2 ∥ T − T h ∥ H S ≤ C h s + d / 2 . s_{2N_h}(T)\le s_{N_h}(T-T_h)\le N_h^{-1/2}\lVert T-T_h\rVert_{\mathrm{HS}}\le Ch^{s+d/2}. s 2 N h ( T ) ≤ s N h ( T − T h ) ≤ N h − 1/2 ∥ T − T h ∥ HS ≤ C h s + d /2 .
The second inequality follows from s N h ( T − T h ) 2 ≤ 1 / N h ∑ j s j ( T − T h ) 2 s_{N_h}(T-T_h)^2\le1/N_h\sum_j s_j(T-T_h)^2 s N h ( T − T h ) 2 ≤ 1/ N h ∑ j s j ( T − T h ) 2 . For every sufficiently large j j j , choose a dyadic h h h with 2 N h ≤ j < 2 N h / 2 2N_h\le j<2N_{h/2} 2 N h ≤ j < 2 N h /2 . Then j ≲ h − d j\lesssim h^{-d} j ≲ h − d , and
s j ( T ) ≤ s 2 N h ( T ) ≤ C h s + d / 2 = O ( j − 1 / 2 − s / d ) . s_j(T)\le s_{2N_h}(T)\le Ch^{s+d/2}=O(j^{-1/2-s/d}). s j ( T ) ≤ s 2 N h ( T ) ≤ C h s + d /2 = O ( j − 1/2 − s / d ) .
The same argument preserves little-o estimates. □
To estimate the singular value decay of the boundary feature operator Φ B \Phi_B Φ B without imposing additional smoothness on ∂ Ω \partial\Omega ∂ Ω , we introduce the following two lemmas.
Lemma 25 (Ambient finite regularity) Let d ≥ 1 d\ge1 d ≥ 1 , let q ≥ 1 q\ge1 q ≥ 1 be an integer, and let X ⊂ R d X\subset\mathbb{R}^d X ⊂ R d be compact with a finite positive Borel measure ρ \rho ρ . Let E E E be a separable real Hilbert space. Suppose a : X → E a:X\to E a : X → E admits an E E E -valued C q C^q C q extension to an open neighborhood of X X X . Then
K : L 2 ( X , ρ ; R ) → E , K f = ∫ X a ( x ) f ( x ) d ρ ( x ) , K:L^2(X,\rho;\mathbb{R})\to E,\qquad Kf=\int_X a(x)f(x)\,\mathrm{d}\rho(x), K : L 2 ( X , ρ ; R ) → E , K f = ∫ X a ( x ) f ( x ) d ρ ( x ) , is compact and s j ( K ) = o ( j − 1 / 2 − q / d ) s_j(K)=o(j^{-1/2-q/d}) s j ( K ) = o ( j − 1/2 − q / d ) .
Proof. If ρ ( X ) = 0 \rho(X)=0 ρ ( X ) = 0 , the assertion is immediate. Otherwise choose a compact neighborhood of X X X contained in the extension domain. For sufficiently small dyadic h h h , a grid partitions X X X into at most C h − d Ch^{-d} C h − d Borel sets of diameter at most h h h . On each nonempty set use the degree-q q q Taylor polynomial of the extension about a point of that set. All line segments used in Taylor’s formula lie in the chosen compact neighborhood. Uniform continuity of the q q q th derivatives gives the uniform remainder bound
sup x ∈ X ∥ a ( x ) − a h ( x ) ∥ E = o ( h q ) . \sup_{x\in X}\lVert a(x)-a_h(x)\rVert_E=o(h^q). x ∈ X sup ∥ a ( x ) − a h ( x ) ∥ E = o ( h q ) .
The integral operator K h K_h K h with kernel a h a_h a h consequently satisfies
rank K h ≤ C q h − d , ∥ K − K h ∥ H S 2 = ∫ X ∥ a ( x ) − a h ( x ) ∥ E 2 d ρ ( x ) = o ( h 2 q ) . \operatorname{rank}K_h\le C_qh^{-d},\qquad\lVert K-K_h\rVert_{\mathrm{HS}}^2=\int_X\lVert a(x)-a_h(x)\rVert_E^2\,d\rho(x)=o(h^{2q}). rank K h ≤ C q h − d , ∥ K − K h ∥ HS 2 = ∫ X ∥ a ( x ) − a h ( x ) ∥ E 2 d ρ ( x ) = o ( h 2 q ) .
The boundedness of a a a and finiteness of ρ \rho ρ make K K K Hilbert–Schmidt. Applying Lemma 24 with s = q s=q s = q proves the lemma. □ \square □
Lemma 26 (Ambient analyticity) Let d ≥ 1 d\ge1 d ≥ 1 , let X ⊂ R d X\subset\mathbb{R}^d X ⊂ R d be compact, and let ρ \rho ρ be a finite positive Borel measure on X X X . Let E E E be a separable real Hilbert space and E C E_{\mathbb{C}} E C its complexification. Suppose a : X → E a:X\to E a : X → E has a norm-holomorphic extension A : U → E C A:U\to E_{\mathbb{C}} A : U → E C on an open neighborhood U ⊂ C d U\subset\mathbb{C}^d U ⊂ C d of X X X . Then the Bochner integral operator
K : L 2 ( X , ρ ; R ) → E , K f = ∫ X a ( x ) f ( x ) d ρ ( x ) , K:L^2(X,\rho;\mathbb{R})\to E,\qquad Kf=\int_X a(x)f(x)\,d\rho(x), K : L 2 ( X , ρ ; R ) → E , K f = ∫ X a ( x ) f ( x ) d ρ ( x ) , is compact, and there exist C , c > 0 C,c>0 C , c > 0 , independent of j j j , such that
s j ( K ) ≤ C exp ( − c j 1 / d ) , j ≥ 1. (73) s_j(K)\le C\exp(-cj^{1/d}),\qquad j\ge1. \tag*{(73)} s j ( K ) ≤ C exp ( − c j 1/ d ) , j ≥ 1. (73)
Proof. The case ρ ( X ) = 0 \rho(X)=0 ρ ( X ) = 0 is immediate. Write P R ( x ) = { z ∈ C d : max i ∣ z i − x i ∣ < R } P_R(x)=\{z\in\mathbb{C}^d:\max_i|z_i-x_i|<R\} P R ( x ) = { z ∈ C d : max i ∣ z i − x i ∣ < R } . By compactness, choose R > 0 R>0 R > 0 , centers x 1 , … , x N ∈ X x_1,\ldots,x_N\in X x 1 , … , x N ∈ X , and a disjoint Borel partition X = ⋃ ℓ = 1 N F ℓ X=\bigcup_{\ell=1}^N F_\ell X = ⋃ ℓ = 1 N F ℓ such that F ℓ ⊂ P R / 2 ( x ℓ ) F_\ell\subset P_{R/2}(x_\ell) F ℓ ⊂ P R /2 ( x ℓ ) and P R ( x ℓ ) ⊂ U P_R(x_\ell)\subset U P R ( x ℓ ) ⊂ U . Set
M : = max 1 ≤ ℓ ≤ N sup z ∈ P R ( x ℓ ) ∥ A ( z ) ∥ E C < ∞ . M:=\max_{1\le\ell\le N}\sup_{z\in P_R(x_\ell)}\lVert A(z)\rVert_{E_{\mathbb{C}}}<\infty. M := 1 ≤ ℓ ≤ N max z ∈ P R ( x ℓ ) sup ∥ A ( z ) ∥ E C < ∞.
In each polydisc, the Taylor expansion and the vector-valued Cauchy formula [40 ] give
A ( z ) = ∑ α ∈ N 0 d a ℓ , α ( z − x ℓ ) α , ∥ a ℓ , α ∥ E C ≤ M R − ∣ α ∣ . A(z)=\sum_{\alpha\in\mathbb{N}_0^d}a_{\ell,\alpha}(z-x_\ell)^\alpha,\qquad\lVert a_{\ell,\alpha}\rVert_{E_{\mathbb{C}}}\le MR^{-|\alpha|}. A ( z ) = α ∈ N 0 d ∑ a ℓ , α ( z − x ℓ ) α , ∥ a ℓ , α ∥ E C ≤ M R − ∣ α ∣ .
For an integer m ≥ 1 m\ge1 m ≥ 1 and x ∈ F ℓ x\in F_\ell x ∈ F ℓ , define
a m ( x ) : = ∑ 0 ≤ α 1 , … , α d < m Re ( a ℓ , α ) ( x − x ℓ ) α . a_m(x):=\sum_{0\le\alpha_1,\ldots,\alpha_d<m}\operatorname{Re}(a_{\ell,\alpha})(x-x_\ell)^\alpha. a m ( x ) := 0 ≤ α 1 , … , α d < m ∑ Re ( a ℓ , α ) ( x − x ℓ ) α .
Since A ( x ) = a ( x ) ∈ E A(x)=a(x)\in E A ( x ) = a ( x ) ∈ E and real-part projection is contractive, absolute convergence and the Cauchy estimate imply
∥ a ( x ) − a m ( x ) ∥ E ≤ ∑ max i α i ≥ m ∥ a ℓ , α ∥ E C ∏ i ∣ x i − x ℓ , i ∣ α i ≤ M ∑ max i α i ≥ m 2 − ∣ α ∣ ≤ M d ( ∑ q = m ∞ 2 − q ) ( ∑ q = 0 ∞ 2 − q ) d − 1 = M d 2 d 2 − m . \begin{aligned}
\lVert a(x)-a_m(x)\rVert_E\le\sum_{\max_i\alpha_i\ge m}\lVert a_{\ell,\alpha}\rVert_{E_{\mathbb{C}}}\prod_i|x_i-x_{\ell,i}|^{\alpha_i} \\
&\le M\sum_{\max_i\alpha_i\ge m}2^{-|\alpha|} \\
&\le Md\left(\sum_{q=m}^{\infty}2^{-q}\right)\left(\sum_{q=0}^{\infty}2^{-q}\right)^{d-1}=Md\,2^{d^2-m}.
\end{aligned} ∥ a ( x ) − a m ( x ) ∥ E ≤ m a x i α i ≥ m ∑ ∥ a ℓ , α ∥ E C i ∏ ∣ x i − x ℓ , i ∣ α i ≤ M m a x i α i ≥ m ∑ 2 − ∣ α ∣ ≤ M d ( q = m ∑ ∞ 2 − q ) ( q = 0 ∑ ∞ 2 − q ) d − 1 = M d 2 d 2 − m .
All multi-index sums in this display are over N 0 d \mathbb{N}_0^d N 0 d . Let K m K_m K m be the integral operator with kernel a m a_m a m . Its range is contained in the span of the N m d Nm^d N m d retained coefficient vectors, and Cauchy–Schwarz gives
rank K m ≤ N m d , ∥ K − K m ∥ ≤ M d 2 d 2 ρ ( X ) 1 / 2 2 − m . \operatorname{rank} K_m \le Nm^d,\qquad\lVert K-K_m\rVert\le Md2^{d^2}\rho(X)^{1/2}2^{-m}. rank K m ≤ N m d , ∥ K − K m ∥ ≤ M d 2 d 2 ρ ( X ) 1/2 2 − m .
Thus K K K is compact and s N m d + 1 ( K ) ≤ ∥ K − K m ∥ s_{Nm^d+1}(K) \le\lVert K-K_m\rVert s N m d + 1 ( K ) ≤ ∥ K − K m ∥ . For j > N j>N j > N , take m = ⌊ ( ( j − 1 ) / N ) 1 / d ⌋ m=\left\lfloor((j-1)/N)^{1/d}\right\rfloor m = ⌊ (( j − 1 ) / N ) 1/ d ⌋ . Increasing C C C to cover the remaining indices proves (B.1). □
Proof of Theorem 8 Set E : = L 2 ( π ) E:=L^2(\pi) E := L 2 ( π ) and, for each derivative occurring below, write
ϕ β ( x ) : = D x β ϕ ( x , ⋅ ) . \phi_\beta(x):=D_x^\beta\phi(x,\cdot). ϕ β ( x ) := D x β ϕ ( x , ⋅ ) .
Define
K β f : = ∫ Ω ϕ β ( x ) f ( x ) d x , K β , ρ I f : = ∫ Ω ϕ β ( x ) f ( x ) d ρ I ( x ) , K_\beta f:=\int_\Omega\phi_\beta(x)f(x)\,\mathrm{d}x,\qquad K_{\beta,\rho_I}f:=\int_\Omega\phi_\beta(x)f(x)\,\mathrm{d}\rho_I(x), K β f := ∫ Ω ϕ β ( x ) f ( x ) d x , K β , ρ I f := ∫ Ω ϕ β ( x ) f ( x ) d ρ I ( x ) ,
with domains L 2 ( Ω , d x ) L^2(\Omega,\mathrm{d}x) L 2 ( Ω , d x ) and L 2 ( ρ I ) L^2(\rho_I) L 2 ( ρ I ) , respectively, and common codomain E E E . Write M g M_g M g for multiplication by a real-valued function g g g on the relevant L 2 L^2 L 2 space.
Since d ρ I = w I d x \mathrm{d}\rho_I=w_I\,\mathrm{d}x d ρ I = w I d x , Lemma 16 yields
s j ( K β , ρ I M g ) ≤ ∥ g ∥ L ∞ ( Ω ) ∥ w I ∥ L ∞ ( Ω ) 1 / 2 s j ( K β ) . (74) s_j(K_{\beta,\rho_I}M_g)\le\lVert g\rVert_{L^\infty(\Omega)}\lVert w_I\rVert_{L^\infty(\Omega)}^{1/2}s_j(K_\beta). \tag*{(74)} s j ( K β , ρ I M g ) ≤ ∥ g ∥ L ∞ ( Ω ) ∥ w I ∥ L ∞ ( Ω ) 1/2 s j ( K β ) . (74)
Let e i e_i e i denote the i i i th coordinate vector. Then
T ~ r = ∑ ∣ β ∣ ≤ r K β K β ∗ . \widetilde{T}_r=\sum_{|\beta|\le r}K_\beta K_\beta^*. T r = ∣ β ∣ ≤ r ∑ K β K β ∗ .
Φ L ∗ = − ∑ i , ℓ = 1 d K e i + e ℓ , ρ I M a i ℓ + ∑ i = 1 d K e i , ρ I M b i + K 0 , ρ I M c , Φ B ∗ f = ∫ ∂ Ω ϕ 0 ( x ) f ( x ) d ρ B ( x ) . \Phi_L^*=-\sum_{i,\ell=1}^dK_{e_i+e_\ell,\rho_I}M_{a_{i\ell}}+\sum_{i=1}^dK_{e_i,\rho_I}M_{b_i}+K_{0,\rho_I}M_c,\qquad\Phi_B^*f=\int_{\partial\Omega}\phi_0(x)f(x)\,\mathrm{d}\rho_B(x). Φ L ∗ = − i , ℓ = 1 ∑ d K e i + e ℓ , ρ I M a i ℓ + i = 1 ∑ d K e i , ρ I M b i + K 0 , ρ I M c , Φ B ∗ f = ∫ ∂ Ω ϕ 0 ( x ) f ( x ) d ρ B ( x ) .
Finite regularity. For ∣ α ∣ = p |\alpha|=p ∣ α ∣ = p and k ≠ 0 k\ne0 k = 0 , the weak chain rule and integration along lines parallel to k k k give
∫ Ω ∣ D x α ϕ ( x , θ ) ∣ 2 d x ≤ C ∣ k ∣ 2 p − 1 ∥ σ ( p ) ∥ L 2 ( − M , M ) 2 . (75) \int_\Omega\left|D_x^\alpha\phi(x,\theta)\right|^2\,\mathrm{d}x\le C|k|^{2p-1}\lVert\sigma^{(p)}\rVert_{L^2(-M,M)}^2. \tag*{(75)} ∫ Ω ∣ D x α ϕ ( x , θ ) ∣ 2 d x ≤ C ∣ k ∣ 2 p − 1 ∥ σ ( p ) ∥ L 2 ( − M , M ) 2 . (75)
For k = 0 k=0 k = 0 , all positive-order spatial derivatives vanish. Compactness of Θ \Theta Θ and continuity of the lower derivatives of σ \sigma σ consequently imply
sup θ ∈ Θ ∥ ϕ ( ⋅ , θ ) ∥ H p ( Ω , d x ) < ∞ . \sup_{\theta\in\Theta}\lVert\phi(\cdot,\theta)\rVert_{H^p(\Omega,\mathrm{d}x)}<\infty. θ ∈ Θ sup ∥ ϕ ( ⋅ , θ ) ∥ H p ( Ω , d x ) < ∞.
Fubini’s theorem, applied also to the weak derivatives, gives
ϕ β ∈ H p − ∣ β ∣ ( Ω ; E ) , ∣ β ∣ ≤ p , \phi_\beta\in H^{p-|\beta|}(\Omega;E),\qquad|\beta|\le p, ϕ β ∈ H p − ∣ β ∣ ( Ω ; E ) , ∣ β ∣ ≤ p ,
where the Sobolev space uses Lebesgue measure. Theorem 17 therefore gives
s j ( K β ) = o ( j − p − ∣ β ∣ d − 1 2 ) . s_j(K_\beta)=o\left(j^{-\frac{p-|\beta|}{d}-\frac12}\right). s j ( K β ) = o ( j − d p − ∣ β ∣ − 2 1 ) .
For ∣ β ∣ = p |\beta|=p ∣ β ∣ = p , this follows instead from ∑ j s j ( K β ) 2 < ∞ \sum_j s_j(K_\beta)^2<\infty ∑ j s j ( K β ) 2 < ∞ and monotonicity. Combining (B.2) with Proposition 15 yields
s j ( Φ L ∗ ) = o ( j − p − 2 d − 1 2 ) . s_j(\Phi_L^*)=o\left(j^{-\frac{p-2}{d}-\frac12}\right). s j ( Φ L ∗ ) = o ( j − d p − 2 − 2 1 ) .
The strict choice of M M M and compactness of Θ \Theta Θ imply that x ↦ ϕ 0 ( x ) x\mapsto\phi_0(x) x ↦ ϕ 0 ( x ) has an E E E -valued C p − 1 C^{p-1} C p − 1 extension to a neighborhood of Ω ‾ \overline{\Omega} Ω . Lemma 25, applied with X = ∂ Ω X=\partial\Omega X = ∂ Ω and ρ = ρ B \rho=\rho_B ρ = ρ B , gives
s j ( Φ B ∗ ) = o ( j − p − 1 d − 1 2 ) . s_j(\Phi_B^*)=o\left(j^{-\frac{p-1}{d}-\frac12}\right). s j ( Φ B ∗ ) = o ( j − d p − 1 − 2 1 ) .
Let J : = # { β ∈ N 0 d : ∣ β ∣ ≤ r } J := \#\{\beta\in\mathbb{N}_{0}^{d} : |\beta| \le r\} J := # { β ∈ N 0 d : ∣ β ∣ ≤ r } . Applying Proposition 15,
μ ~ j ( r ) ≤ ∑ ∣ β ∣ ≤ r s ⌈ j / J ⌉ ( K β ) 2 , (76) \widetilde{\mu}^{(r)}_j \le\sum_{|\beta|\le r} s_{\lceil j/J\rceil}(K_\beta)^2, \tag*{(76)} μ j ( r ) ≤ ∣ β ∣ ≤ r ∑ s ⌈ j / J ⌉ ( K β ) 2 , (76)
μ j P D E ≤ ν I s ⌈ j / 2 ⌉ ( Φ L ∗ ) 2 + η ν B s ⌈ j / 2 ⌉ ( Φ B ∗ ) 2 . \mu^{\mathrm{PDE}}_j \le\nu^{I}s_{\lceil j/2\rceil}(\Phi_L^*)^2+\eta\nu^{B}s_{\lceil j/2\rceil}(\Phi_B^*)^2. μ j PDE ≤ ν I s ⌈ j /2 ⌉ ( Φ L ∗ ) 2 + η ν B s ⌈ j /2 ⌉ ( Φ B ∗ ) 2 .
Thus
μ ~ j ( r ) = o ( j 2 ( p − r ) d − 1 ) , μ j P D E = o ( j 2 ( p − 2 ) d − 1 ) . \widetilde{\mu}^{(r)}_j=o\left(j^{\frac{2(p-r)}{d}-1}\right),\qquad\mu^{\mathrm{PDE}}_j=o\left(j^{\frac{2(p-2)}{d}-1}\right). μ j ( r ) = o ( j d 2 ( p − r ) − 1 ) , μ j PDE = o ( j d 2 ( p − 2 ) − 1 ) .
Analytic regularity. Real analyticity on an open interval containing [ − M , M ] [-M,M] [ − M , M ] provides a holomorphic extension of σ \sigma σ to a complex neighborhood of this interval segment. By compactness of Ω ‾ \overline{\Omega} Ω and Θ \Theta Θ , there is a common complex neighborhood U U U of Ω ‾ \overline{\Omega} Ω such that for every ∣ β ∣ ≤ max { r , 2 } |\beta|\le\max\{r,2\} ∣ β ∣ ≤ max { r , 2 } ,
z ⟼ [ θ ⟼ k β σ ( ∣ β ∣ ) ( k T z + b ) ] z\longmapsto\left[\theta\longmapsto k^\beta\sigma^{(|\beta|)}(k^\mathsf{T}z+b)\right] z ⟼ [ θ ⟼ k β σ ( ∣ β ∣ ) ( k T z + b ) ]
is a norm-holomorphic L 2 ( π ; C ) L^2(\pi;\mathbb{C}) L 2 ( π ; C ) -valued extension of ϕ β \phi_\beta ϕ β . Indeed, after shrinking U U U , all values k T z + b k^\mathsf{T}z+b k T z + b , with z ∈ U z\in U z ∈ U and θ = ( k , b ) ∈ Θ \theta=(k,b)\in\Theta θ = ( k , b ) ∈ Θ , lie in a fixed compact subset of the holomorphic extension domain of σ \sigma σ . Cauchy estimates and compactness of Θ \Theta Θ then provide uniform bounds for the derivatives with respect to z z z , which justify differentiation in L 2 ( π ; C ) L^2(\pi;\mathbb{C}) L 2 ( π ; C ) .
Apply Lemma 26 to X = Ω ‾ X=\overline{\Omega} X = Ω with Lebesgue measure restricted to Ω \Omega Ω , and to X = ∂ Ω X=\partial\Omega X = ∂ Ω with measure ρ B \rho_B ρ B and a = ϕ 0 a=\phi_0 a = ϕ 0 . Taking common constants over the finitely many derivatives gives
s j ( K β ) ≤ C 0 e − c 0 j 1 / d , s j ( Φ B ∗ ) ≤ C 0 e − c 0 j 1 / d . s_j(K_\beta)\le C_0e^{-c_0j^{1/d}},\qquad s_j(\Phi_B^*)\le C_0e^{-c_0j^{1/d}}. s j ( K β ) ≤ C 0 e − c 0 j 1/ d , s j ( Φ B ∗ ) ≤ C 0 e − c 0 j 1/ d .
Equation (B.2) and Proposition 15 then give s j ( Φ L ∗ ) ≤ C 1 e − c 1 j 1 / d s_j(\Phi_L^*)\le C_1e^{-c_1j^{1/d}} s j ( Φ L ∗ ) ≤ C 1 e − c 1 j 1/ d . Using (B.4) and adjusting the constants proves
μ ~ j ( r ) ≤ C τ 2 j 1 / d , μ j P D E ≤ C τ 2 j 1 / d , j ≥ 1 , \widetilde{\mu}^{(r)}_j\le C\tau^{2j^{1/d}},\qquad\mu^{\mathrm{PDE}}_j\le C\tau^{2j^{1/d}},\qquad j\ge1, μ j ( r ) ≤ C τ 2 j 1/ d , μ j PDE ≤ C τ 2 j 1/ d , j ≥ 1 ,
for some C > 0 C>0 C > 0 and τ ∈ ( 0 , 1 ) \tau\in(0,1) τ ∈ ( 0 , 1 ) independent of j j j . □
Proof of Theorem 9
Proof. Let Q 0 Q_0 Q 0 be a cube containing Ω ‾ \overline{\Omega} Ω . For sufficiently small dyadic h h h , a uniform grid of side length h h h meets Q 0 Q_0 Q 0 in O ( h − d ) O(h^{-d}) O ( h − d ) cubes. Let V h ⊂ L 2 ( Ω ) V_h\subset L^2(\Omega) V h ⊂ L 2 ( Ω ) consist of restrictions to Ω \Omega Ω of functions that are polynomials of total degree at most k k k on each grid cube, and let P h P_h P h be the orthogonal projection onto V h V_h V h . Then dim V h ≤ C h − d \dim V_h\le Ch^{-d} dim V h ≤ C h − d .
Fix θ = ( k , b ) \theta=(k,b) θ = ( k , b ) and a multi-index β \beta β with q : = ∣ β ∣ ≤ k q:=|\beta|\le k q := ∣ β ∣ ≤ k . If k ≠ 0 k\ne0 k = 0 , the weak derivative is, almost everywhere,
D x β ϕ ( x , θ ) = k ! ( k − q ) ! k β ( k T x + b ) + k − q , D_x^\beta\phi(x,\theta)=\frac{k!}{(k-q)!}k^\beta(k^\mathsf{T}x+b)_+^{k-q}, D x β ϕ ( x , θ ) = ( k − q )! k ! k β ( k T x + b ) + k − q ,
where t + 0 : = 1 { t > 0 } t_+^0:=\mathbf{1}_{\{t>0\}} t + 0 := 1 { t > 0 } . Call a grid cube cut if its closure meets { x : k T x + b = 0 } \{x:k^\mathsf{T}x+b=0\} { x : k T x + b = 0 } , and let S h , θ S_{h,\theta} S h , θ be the union of the cut cubes meeting Q 0 Q_0 Q 0 . Within a fixed bounded enlargement of Q 0 Q_0 Q 0 , this union lies in a slab of thickness 2 d h 2\sqrt{d}h 2 d h ; hence ∣ S h , θ ∣ ≤ C h |S_{h,\theta}|\le Ch ∣ S h , θ ∣ ≤ C h , uniformly in θ \theta θ . Outside this union the derivative is a polynomial of degree at most k − q k-q k − q . On the cut cubes,
∣ D x β ϕ ( x , θ ) ∣ ≤ C ∣ k ∣ k − q ≤ C h k − q . |D_x^\beta\phi(x,\theta)|\le C|k|^{k-q}\le Ch^{k-q}. ∣ D x β ϕ ( x , θ ) ∣ ≤ C ∣ k ∣ k − q ≤ C h k − q .
Approximate the derivative by itself on the uncut cubes and by zero on the cut cubes. This defines a function in V h V_h V h . The best-approximation property of P h P_h P h gives
∥ ( I − P h ) D x β ϕ ( ⋅ , θ ) ∥ L 2 ( Ω ) 2 ≤ C h 2 ( k − q ) + 1 . (77) \left\|(I-P_h)D_x^\beta\phi(\cdot,\theta)\right\|_{L^2(\Omega)}^2\le Ch^{2(k-q)+1}. \tag*{(77)} ( I − P h ) D x β ϕ ( ⋅ , θ ) L 2 ( Ω ) 2 ≤ C h 2 ( k − q ) + 1 . (77)
If k = 0 k=0 k = 0 , the feature is constant and all its positive-order derivatives vanish, so the same estimate holds. All constants are uniform over the compact set Θ \Theta Θ .
Use the operators K β K_\beta K β defined in the proof of Theorem 8 , then
rank ( K β P h ) ≤ C h − d \operatorname{rank}(K_\beta P_h) \le Ch^{-d} rank ( K β P h ) ≤ C h − d
∥ K β − K β P h ∥ H S 2 = ∫ Θ ∥ ( I − P h ) D x β ϕ ( ⋅ , θ ) ∥ L 2 ( Ω ) 2 d π ( θ ) ≤ C h 2 ( k − q ) + 1 . \begin{aligned}
\lVert K_\beta-K_\beta P_h\rVert_{\mathrm{HS}}^2=\int_{\Theta}\lVert(I-P_h)D_x^\beta\phi(\cdot,\theta)\rVert_{L^2(\Omega)}^2\,\mathrm{d}\pi(\theta) \\
&\le Ch^{2(k-q)+1}.
\end{aligned} ∥ K β − K β P h ∥ HS 2 = ∫ Θ ∥( I − P h ) D x β ϕ ( ⋅ , θ ) ∥ L 2 ( Ω ) 2 d π ( θ ) ≤ C h 2 ( k − q ) + 1 .
Lemma 24 , with s = k − q + 1 2 s=k-q+\frac{1}{2} s = k − q + 2 1 , yields
s j ( K β ) ≤ C j − 1 / 2 − ( k − q + 1 / 2 ) / d . s_j(K_\beta)\le Cj^{-1/2-(k-q+1/2)/d}. s j ( K β ) ≤ C j − 1/2 − ( k − q + 1/2 ) / d .
The first inequality in (B.4) then proves (26) .
Moreover, (B.2) with g ≡ 1 g\equiv1 g ≡ 1 gives
s j ( K β , ρ I ) ≤ ∥ w I ∥ L ∞ ( Ω ) 1 / 2 s j ( K β ) ≤ C j − 1 / 2 − ( k − q + 1 / 2 ) / d . s_j(K_{\beta,\rho_I})\le\lVert w_I\rVert_{L^\infty(\Omega)}^{1/2}s_j(K_\beta)\le Cj^{-1/2-(k-q+1/2)/d}. s j ( K β , ρ I ) ≤ ∥ w I ∥ L ∞ ( Ω ) 1/2 s j ( K β ) ≤ C j − 1/2 − ( k − q + 1/2 ) / d .
For the PDE statement, let k ≥ 2 k\ge2 k ≥ 2 . Lemma 16 and Proposition 15 , applied to the decomposition of Φ L ∗ \Phi_{\mathcal{L}}^* Φ L ∗ , give
s j ( Φ L ∗ ) ≤ C j − 1 / 2 − ( k − 3 / 2 ) / d . s_j(\Phi_{\mathcal{L}}^*)\le Cj^{-1/2-(k-3/2)/d}. s j ( Φ L ∗ ) ≤ C j − 1/2 − ( k − 3/2 ) / d .
Since t + ∈ C k − 1 ( R ) t_+\in C^{k-1}(\mathbb{R}) t + ∈ C k − 1 ( R ) , Lemma 25 gives
s j ( Φ B ∗ ) = o ( j − 1 / 2 − ( k − 1 ) / d ) . s_j(\Phi_B^*)=o\left(j^{-1/2-(k-1)/d}\right). s j ( Φ B ∗ ) = o ( j − 1/2 − ( k − 1 ) / d ) .
The second inequality in (B.4) then proves the claimed PDE eigenvalue bound. □ \square □