betaFeedback

WorksMathematics

A Convergence Analysis of the Random Feature Method for Linear PDEs

Abstract

The Random Feature Method (RFM) distinguishes itself among machine learning approaches to partial differential equations (PDEs) through its superior accuracy and computational efficiency. Nevertheless, the theoretical underpinnings of RFM remain incompletely elucidated. This paper provides a comprehensive analysis of RFM within the framework of kernel methods, focusing on two principal aspects: (i) approximation by finitely many random features in Sobolev norms and the PDE residual seminorm, and (ii) estimation over the induced reproducing kernel Hilbert space (RKHS) from finitely many noiseless collocation observations. We relate both problems to kernel ridge regression (KRR) and establish upper and lower bounds in terms of the eigenvalues of the associated integral operators under the respective assumptions. Combining the approximation and estimation lower bounds, we establish a lower bound under simultaneous restrictions on the numbers of random features and collocation observations. We further quantify how activation regularity affects these eigenvalues, obtaining algebraic upper bounds for finitely smooth activations and stretched-exponential upper bounds for analytic activations.

keywords: 41A25; 46E22; 65N15; 65N35

Introduction

Partial differential equations (PDEs) constitute a cornerstone in the mathematical modeling of physical and engineering systems. In recent years, the advent of deep learning techniques has facilitated the emergence of novel methodologies for solving PDEs [1, 2, 3, 4, 5, 6]. The notable efficacy of these methods stems from their mesh-free nature and their capacity to alleviate the curse of dimensionality. However, their application is constrained by two primary bottlenecks: limited accuracy and high computational expense. These limitations fundamentally originate from the nonconvex optimization landscape encountered during neural network training. Several approaches address this difficulty by fixing the inner-layer parameters [7, 8, 9]. For linear PDEs, optimizing the output-layer coefficients then reduces to a linear least-squares problem, thereby avoiding the complexities associated with nonconvex optimization. A prominent instance is the random feature method (RFM) [8], which uses a random feature model [10] to represent the approximate solution. This model is a two-layer neural network whose hidden-layer parameters are randomly sampled and then held fixed. Specifically, the corresponding function class is

Σmϕ:={uα:uα(x)=1m∑i=1mαiϕ(x,θi), α∈Rm}\Sigma_m^\phi:= \left\{u_\alpha: u_\alpha(x) = \frac{1}{\sqrt{m}}\sum_{i=1}^{m}\alpha_i\phi(x,\theta_i),\ \alpha\in\mathbb{R}^{m}\right\}

where θ1,…,θm\theta_1,\ldots,\theta_m are the sampled feature parameters and α\alpha contains the trainable output coefficients. RFM has exhibited exceptional performance in diverse numerical experiments spanning structural analysis, interface problems, and time-dependent problems [8, 11, 12, 13]. Notably, in many cases, the achieved errors decay at an exponential or near-exponential rate as the number of random features or collocation points increases. Despite these empirical successes, a comprehensive theoretical explanation for such convergence rates remains incomplete. Closing this theoretical gap is essential both to substantiate the empirical efficacy of RFM and to extend its applicability to a wider range of challenging problems.

Throughout this paper, we consider linear boundary value problems of the form

{Lu∗=f∗in Ω,Bu∗=g∗on ∂Ω.\begin{cases} \mathcal{L}u^* = f^* & \text{in }\Omega,\\ \mathcal{B}u^* = g^* & \text{on }\partial\Omega. \end{cases}

where L\mathcal{L} is a linear differential operator, B\mathcal{B} is a linear boundary operator, and u∗u^* is the target solution. Our analysis of RFM distinguishes two sources of error. The approximation error is the distance from u∗u^* to the random feature model. The estimation error arises from recovering u∗u^* from finitely many noiseless observations of f∗f^* and g∗g^*. Since random features were introduced as a computational surrogate for kernel methods [10], we analyze both problems through the induced kernels and their associated integral operators. For finite-feature approximation, we construct kernels on parameter space from the Sobolev inner products of the features and from the weighted inner products of their images under the differential and boundary operators. For each error metric, our duality result reformulates the regularized finite-feature approximation problem as noiseless KRR on the parameter space, where the sampled parameters serve as observation sites for the transformed target. Through this correspondence, the noiseless recovery lower bound and the KRR upper bound in [14] yield the lower and upper bounds for the worst-case approximation error in each metric. For finite collocation estimation, we construct a spatial kernel for which the rescaled interior and boundary observations are point values of the transformed target. The collocation lower bound then follows from the noiseless recovery bound in [14 Theorem 4.1]. For the collocation upper bound, we adapt the concentration argument in the proof of the upper bound in [14] to account for the separate sampling of interior and boundary points. We control each empirical operator under its own sampling distribution and combine the resulting interior and boundary operator deviation bounds to estimate the population residual of the regularized estimator over the full RKHS. Although finite-feature approximation admits a direct analysis, we use duality to study it within the same noiseless KRR framework as recovery from collocation observations. Our approach based on duality yields the same best approximation error as the direct approach (see Remark 5). Both the approximation and collocation bounds depend on the eigenvalues of the corresponding integral operators. To make the bounds explicit, we relate the decay of the associated eigenvalues to the regularity of the activation function. We use Volkov’s singular-value estimates [15] together with our ambient regularity lemmas to derive upper bounds for the eigenvalues governing Sobolev approximation and PDE collocation.

Our main results are summarized below, with precise statements and assumptions given in § 4.

  • Approximation in Sobolev norms and the residual seminorm. We establish upper and lower bounds for the worst-case approximation error over the unit ball of Hk\mathcal{H}_k. Under the respective assumptions of Theorems 1 and 2 for Sobolev approximation and Theorems 3 and 4 for residual approximation, the bounds take the form

μ~m+1(r)≤sup⁡∥u∗∥Hk≤1inf⁡u∈Φ~m∥u−u∗∥Hr(Ω)2≲μ~lm(r)\widetilde{\mu}_{m+1}^{(r)} \le\sup_{\lVert u^*\rVert_{\mathcal{H}_k}\le1}\inf_{u\in\widetilde{\Phi}_m}\lVert u-u^*\rVert_{\mathcal{H}_r(\Omega)}^2 \lesssim\widetilde{\mu}_{l_m}^{(r)}
μ~m+1(Rη)≤sup⁡∥u∗∥Hk≤1inf⁡u∈Φ~m∥u−u∗∥Rη2≲μ~lm(Rη).\widetilde{\mu}_{m+1}^{(\mathcal{R}_{\eta})} \le\sup_{\lVert u^*\rVert_{\mathcal{H}_k}\le1}\inf_{u\in\widetilde{\Phi}_m}\lVert u-u^*\rVert_{\mathcal{R}_{\eta}}^2 \lesssim\widetilde{\mu}_{l_m}^{(\mathcal{R}_{\eta})}.

For each error measure, the upper bound holds with probability at least 1−δ1-\delta under its corresponding feature sampling distribution, whereas the lower bounds hold for every fixed admissible choice of feature parameters. Here, μ~j(r)\widetilde{\mu}_j^{(r)} and μ~j(Rη)\widetilde{\mu}_j^{(\mathcal{R}_{\eta})} denote the jj-th largest positive eigenvalues of the integral operators defined in (41) and (51), respectively. The index lml_m is specified in the respective upper-bound theorem with lm≳log⁡(m/δ)l_m \gtrsim\log(m/\delta).

  • Estimation from finitely many collocation observations. We analyze regularized recovery over the full RKHS using the estimator

u^n,η,λ:=argmin⁡u∈Hk{R^n,η(u)+λ∥u∥Hk2},λ>0,\hat{u}_{n,\eta,\lambda}:=\underset{u\in\mathcal{H}_k}{\operatorname{argmin}}\left\{\widehat{\mathcal{R}}_{n,\eta}(u)+\lambda\|u\|_{\mathcal{H}_k}^{2}\right\},\qquad\lambda>0,

where R^n,η\widehat{\mathcal{R}}_{n,\eta} is the weighted empirical PDE residual defined in (10), constructed from nn noiseless interior and boundary observations. Under the sampling and regularization conditions of Theorem 5, the estimator u^n,η,λ\hat{u}_{n,\eta,\lambda} satisfies, with probability at least 1−δ1-\delta,

sup⁡∥u∗∥Hk≤1∥u^n,η,λ−u∗∥Rη2≲μlnPDE,ln≲nlog⁡(n/δ).\sup_{\|u^*\|_{\mathcal{H}_k}\leq1}\|\hat{u}_{n,\eta,\lambda}-u^*\|_{\mathcal{R}_\eta}^{2}\lesssim\mu_{l_n}^{\mathrm{PDE}},\qquad l_n\lesssim\frac{n}{\log(n/\delta)}.

Here μjPDE\mu_j^{\mathrm{PDE}} denotes the jj-th largest positive eigenvalue of the integral operator defined in (57). The index lnl_n is specified in the theorem. For every fixed admissible collection of collocation observations, we also establish

inf⁡G∈MnPDEA sup⁡∥u∗∥Hk≤1∥G(Sn(u∗))−u∗∥Rη2≳μn+1PDE.\inf_{G\in\mathcal{M}_{n}^{\mathrm{PDE}}\atop{\vphantom{\Large A}}}\ \sup_{\|u^*\|_{\mathcal{H}_k}\leq1}\|G(S_n(u^*))-u^*\|_{\mathcal{R}_\eta}^{2}\gtrsim\mu_{n+1}^{\mathrm{PDE}}.

Here, Sn(u∗)S_n(u^*) denotes the observations, and MnPDE\mathcal{M}_{n}^{\mathrm{PDE}} is the class of reconstruction procedures based only on these observations.

  • A lower bound under joint feature and observation restrictions. We consider reconstruction from nn collocation observations with outputs restricted to a random feature model spanned by mm features. Let Em,nE_{m,n} denote the smallest worst-case squared residual error over the unit ball of Hk\mathcal{H}_k among all such reconstruction procedures. For fixed positive proportions of interior and boundary observations, we establish

Em,n≳max⁡{μm+1PDE,μn+1PDE}.E_{m,n}\gtrsim\max\{\mu_{m+1}^{\mathrm{PDE}},\mu_{n+1}^{\mathrm{PDE}}\}.

for every fixed admissible choice of feature parameters and collocation points, with an implicit constant independent of mm and nn. This bound identifies the separate limitations imposed by the feature space and the available observations and applies, in particular, to the RFM estimator.

  • Dependence of eigenvalue decay on activation regularity. We derive eigenvalue upper bounds for features of the form ϕ(x,θ)=σ(k⊤x+b)\phi(x,\theta)=\sigma(k^{\top}x+b). For the PDE eigenvalues, we specialize to second-order uniformly elliptic equations with Dirichlet boundary conditions. Under the finite-regularity assumptions of Theorem 8, with p≥max⁡{2,r}p\geq\max\{2,r\}, we obtain

μ~j(r)=o(j−1−2(p−r)d),μjPDE=o(j−1−2(p−2)d).\tilde{\mu}_{j}^{(r)}=o\left(j^{-1-\frac{2(p-r)}{d}}\right),\qquad\mu_{j}^{\mathrm{PDE}}=o\left(j^{-1-\frac{2(p-2)}{d}}\right).

Under the analytic assumptions of the same theorem, there exists c>0c>0 such that

μ~j(r)≲e−cj1/d,μjPDE≲e−cj1/d.\tilde{\mu}_{j}^{(r)}\lesssim e^{-cj^{1/d}},\qquad\mu_{j}^{\mathrm{PDE}}\lesssim e^{-cj^{1/d}}.

Related Work

This section reviews related work on random-feature approximation, spectral analysis of kernel methods, and kernel collocation for PDEs.

Random-feature approximation. Early universal approximation results cover networks with randomly generated hidden nodes [16]. For uniformly bounded features, a fixed target admits a high-probability L2L^2 approximation bound of order O(m−1/2)O(m^{-1/2}) if its integral representation has a coefficient function whose ratio to the sampling density is bounded [17]. For ReLUk^k features with well-distributed fixed parameters on SdS^d, the HrH^r approximation error is of order O(m−1/2−(2(k−r)+1)/(2d))O(m^{-1/2-(2(k-r)+1)/(2d)}) for 0≤r≤k0 \le r \le k and targets in H(d+2k+1)/2(Ω)H^{(d+2k+1)/2}(\Omega) on a bounded Lipschitz domain Ω\Omega [18 Theorem 2.2]. For parameters sampled independently and uniformly on SdS^d, the corresponding high-probability HrH^r bound incurs a logarithmic loss in the number of features [18 Theorem 6.2]. [19] studies best LpL^p approximation on an interval by random projection networks with fixed hidden parameters and smooth nonpolynomial activations. [20] analyze RFM for one-dimensional second-order elliptic problems and derive error estimates that depend on the regularity of the solution. In multiple dimensions, they establish high-probability approximation bounds for trigonometric features [21]. Our analysis starts from a general feature map and its induced RKHS. Under the stated spectral and sampling assumptions, we derive upper and lower bounds for the worst-case approximation error over the RKHS unit ball, both in Sobolev norms and in the PDE residual seminorm. For each error metric, the bounds are expressed in terms of the eigenvalues of the corresponding operator on the parameter space. This formulation applies without prescribing a particular activation function. We then use the regularity of the activation function to obtain explicit upper bounds on the eigenvalues.

Spectral analysis of kernel methods. For random-feature approximation and kernel quadrature, the eigenvalues of the associated integral operator yield upper and lower bounds on the number of samples needed to attain a prescribed accuracy. The upper bounds can be attained by sampling from a suitable nonuniform distribution [22]. A duality framework connects random-feature approximation and estimation [23]. In our setting, the sampled feature parameters serve as observation points in a noiseless KRR problem on parameter space. This formulation allows us to bound the finite-feature approximation error using KRR estimates that depend on eigenvalue decay and target regularity [14]. Sobolev and analytic regularity of an integral kernel also yield decay estimates for the singular values of the corresponding operator [15]. We draw on these results to analyze approximation in Sobolev norms and the PDE residual seminorm, recovery from interior and boundary observations, and the dependence of eigenvalue bounds on activation regularity.

Kernel collocation for PDEs. Radial basis function collocation is a classical kernel method for solving PDEs [24, 25]. Native-space interpolation theory relates approximation errors to fill distance and kernel smoothness [26]. For second-order linear elliptic equations with Dirichlet boundary conditions, constrained and weighted least-squares kernel collocation methods admit Sobolev error bounds under suitable smoothness and point-density conditions [27]. For nonlinear and parametric PDEs, error analyses of kernel and Gaussian process methods use RKHS embeddings, PDE stability, and Sobolev interpolation estimates [28]. Our analysis complements these results by deriving upper and lower bounds in terms of operator eigenvalues for two distinct problems: approximation by mm features and recovery over the full RKHS from nn collocation observations. When reconstruction is subject to both restrictions, the worst-case squared residual error is at least the larger of the two lower bounds.

Notations

References

  1. [1]E, W., Han, J., Jentzen, A.: Deep learning-based numerical methods for high-dimensional parabolic partial differential equations. Commun. Math. Stat. 5(4), 349–380 (2017)DOI
  2. [2]Han, J., Jentzen, A., E, W.: Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. 115(34), 8505–8510 (2018)DOI
  3. [3]E, W., Yu, B.: The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat. 6(1), 1–12 (2018)arxiv.org/abs/1710.00211
  4. [4]Sirignano, J., Spiliopoulos, K.: DGM: A deep learning algorithm for solving partial differential equations. J. Comput. Phys. 375, 1339–1364 (2018)DOI
  5. [5]Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019)DOI
  6. [6]Zang, Y., Bao, G., Ye, X., Zhou, H.: Weak adversarial networks for high-dimensional partial differential equations. J. Comput. Phys. 411, 109409 (2020)DOI
  7. [7]Dong, S., Li, Z.: Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. Comput. Methods Appl. Mech. Engrg. 387, 114129 (2021)DOI
  8. [8]Chen, J., Chi, X., E, W., Yang, Z.: Bridging traditional and machine learning-based algorithms for solving PDEs: The random feature method. J. Mach. Learn. 1(3), 268–298 (2022)arxiv.org/abs/2207.13380
  9. [9]Shang, Y., Wang, F., Sun, J.: Randomized neural network with Petrov–Galerkin methods for solving linear and nonlinear partial differential equations. Commun. Nonlinear Sci. Numer. Simul. 127, 107518 (2023)
  10. [10]Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Advances in Neural Information Processing Systems, vol. 20 (2007)
  11. [11]Chen, J., E, W., Luo, Y.: The random feature method for time-dependent problems. East Asian J. Appl. Math. 13(3), 435–463 (2023)arxiv.org/abs/2304.06913
  12. [12]Chi, X., Chen, J., Yang, Z.: The random feature method for solving interface problems. Comput. Methods Appl. Mech. Engrg. 420, 116719 (2024)arxiv.org/abs/2308.04330
  13. [13]Sun, Y., Chen, J.: Two-level random feature methods for elliptic partial differential equations over complex domains. Comput. Methods Appl. Mech. Engrg. 441, 117961 (2025)DOI
  14. [14]Long, J., Peng, X., Wu, L.: Optimal rates and saturation for noiseless kernel ridge regression. ArXiv (2025) arXiv:2402.15718 [stat.ML]arxiv.org/abs/2402.15718
  15. [15]Volkov, D.: Optimal decay rates in Sobolev norms for singular values of integral operators. J. Math. Anal. Appl. 537(2), 128403 (2024)arxiv.org/abs/2309.00153
  16. [16]Huang, G.B., Chen, L., Siew, C.K.: Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Netw. 17(4), 879–892 (2006) https://doi.org/10.1109/TNN.2006.875977
  17. [17]Rahimi, A., Recht, B.: Uniform approximation of functions with random bases. In: 2008 46th Annual Allerton Conference on Communication, Control, And Computing, pp. 555–561 (2008). IEEEDOI
  18. [18]Liu, X., Mao, T., Xu, J.: Integral representations of Sobolev spaces via ReLU^k activation function and optimal error estimates for linearized networks. ArXiv (2025) arXiv:2505.00351arxiv.org/abs/2505.00351
  19. [19]Fabiani, G.: Random projection neural networks of best approximation: Convergence theory and practical applications. SIAM J. Math. Data Sci. 7(2), 385–409 (2025)arxiv.org/abs/2402.11397
  20. [20]Ming, P., Yu, H.: Spectral convergence of random feature method in one dimension. ArXiv (2025) arXiv:2507.07371arxiv.org/abs/2507.07371
  21. [21]Ming, P., Yu, H.: Spectral convergence of random feature method in multiple dimensions. ArXiv (2026) arXiv:2609.03401arxiv.org/abs/2609.03401
  22. [22]Bach, F.: On the equivalence between kernel quadrature rules and random feature expansions. J. Mach. Learn. Res 18(1), 714–751 (2017)
  23. [23]Chen, H., Long, J., Wu, L.: A duality framework for analyzing random feature and two-layer neural networks. Ann. Statist. 53(3), 1044–1067 (2025)arxiv.org/abs/2305.05642
  24. [24]Kansa, E.J.: Multiquadrics—A scattered data approximation scheme with applications to computational fluid-dynamics. I. Surface approximations and partial derivative estimates. Comput. Math. Appl. 19(8-9), 127–145 (1990)DOI
  25. [25]Kansa, E.J.: Multiquadrics—A scattered data approximation scheme with applications to computational fluid-dynamics. II. Solutions to parabolic, hyperbolic and elliptic partial differential equations. Comput. Math. Appl. 19(8-9), 147–161 (1990)DOI
  26. [26]Wendland, H.: Scattered Data Approximation vol. 17. Cambridge Monographs on Applied and Computational Mathematics, Cambridge (2004)DOI
  27. [27]Cheung, K.C., Ling, L., Schaback, R.: H²-convergence of least-squares kernel collocation methods. SIAM J. Numer. Anal. 56(1), 614–633 (2018)arxiv.org/abs/1801.00629
  28. [28]Batlle, P., Chen, Y., Hosseini, B., Owhadi, H., Stuart, A.M.: Error analysis of kernel/GP methods for nonlinear and parametric PDEs. J. Comput. Phys. 520, 113488 (2025)
  29. [29]Aronszajn, N.: Theory of reproducing kernels. Trans. Amer. Math. Soc. 68(3), 337–404 (1950)
  30. [30]Steinwart, I., Christmann, A.: Support Vector Machines. Springer, New York (2008)
  31. [31]Schölkopf, B., Herbrich, R., Smola, A.J.: A generalized representer theorem. In: International Conference on Computational Learning Theory, pp. 416–426 (2001). SpringerDOI
  32. [32]Zhang, T.: Learning bounds for kernel regression using effective data dimensionality. Neural Comput. 17(9), 2077–2098 (2005)pubmed.ncbi.nlm.nih.gov/15992491
  33. [33]Caponnetto, A., De Vito, E.: Optimal rates for the regularized least-squares algorithm. Found. Comput. Math. 7, 331–368 (2007)DOI
  34. [34]Blanchard, G., Mücke, N.: Optimal rates for regularization of statistical inverse learning problems. Found. Comput. Math. 18(4), 971–1013 (2018)arxiv.org/abs/1604.04054
  35. [35]Lin, J., Rudi, A., Rosasco, L., Cevher, V.: Optimal rates for spectral algorithms with least-squares regression over Hilbert spaces. Appl. Comput. Harmon. Anal. 48(3), 868–890 (2020)
  36. [36]Shin, Y., Zhang, Z., Karniadakis, G.E.: Error estimates of residual minimization using neural networks for linear PDEs. J. Mach. Learn. Model. Comput. 4(4), 73–101 (2023) https://doi.org/10.1615/JMachLearnModelComput.2023050411
  37. [37]Kato, T.: Perturbation Theory for Linear Operators vol. 132. Springer, New York (2013)
  38. [38]Gohberg, I., Goldberg, S., Kaashoek, M.A.: Classes of Linear Operators, Vol. I. Operator Theory: Advances and Applications, vol. 49. Birkhäuser, Basel (1990). https://doi.org/10.1007/978-3-0348-7509-7
  39. [39]Minsker, S.: On some extensions of Bernstein’s inequality for self-adjoint operators. Statist. Probab. Lett. 127, 111–119 (2017)arxiv.org/abs/1112.5448
  40. [40]Kruse, K.: Vector-valued holomorphic functions in several variables. Funct. Approx. Comment. Math. 63(2), 247–275 (2020) https://doi.org/10.7169/facm/1861

Paper details

vlnc:W_8mggg5dc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents