betaFeedback

WorksMathematics

Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality

Abstract

In this paper, we study the statistical limits of deep learning techniques for solving elliptic partial differential equations (PDEs) from random samples using the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs). To simplify the problem, we focus on a prototype elliptic PDE: the Schrödinger equation on a hypercube with zero Dirichlet boundary condition, which has wide application in the quantum-mechanical systems. We establish upper and lower bounds for both methods, which improves upon concurrently developed upper bounds for this problem via a fast rate generalization bound. We discover that the current Deep Ritz Methods is sub-optimal and propose a modified version of it. We also prove that PINN and the modified version of DRM can achieve minimax optimal bounds over Sobolev spaces. Empirically, following recent work which has shown that the deep model accuracy will improve with growing training sets according to a power law, we supply computational experiments to show a similar behavior of dimension dependent power law for deep PDE solvers.

Introduction

Partial differential equations (PDEs) play a prominent role in many disciplines of science and engineering. The recent deep learning breakthrough and the rapid development of sensors, computational power, and data storage in the past decade has drawn attention to numerically solving PDEs via machine learning methods [45, 44, 58, 26, 68, 38], especially in high dimensions where conventional methods become impractical. The set of applications that motivate this interest is wide-ranging, including computational physics [26, 45, 58], inverse problem [82, 22, 17] and quantitative finance [30, 21]. The numerical methods generated by the use of deep learning techniques are mesh-less methods, see the discussion in [78]. A natural deep learning technique in the problems that are based on a standard feed-forward type of architecture takes advantage (when available) of a variational formulation, whose solution coincides with the solution of the PDE of interest. Despite the success and popularity of adopting neural networks for solving high-dimensional PDEs, the following question still remains poorly answered.

For a given PDE and a data-driven approximation architecture, how large the sample size and how complex the model are needed to reach a prescribed performance level?

In this paper, we aim to establish the numerical analysis of such deep learning based PDE solvers. Inspired by recent works which showed that the empirical performance of a model is remarkably predictable via a power law of the data number, known as the neural scaling law [37, 31, 65], we aim to explore the neural scaling law for deep PDE solvers and compare its performance to Fourier approximation.

Among the various approaches of using deep learning methods for solving PDEs, in this work, we focus on the Deep Ritz method (DRM) [15, 38] and the Physics-Informed Neural Networks (PINN) approach [68, 58], both of which are based on minimizing neural network parameters according to some loss functional related to the PDEs. To provide theoretical guarantees for DRM and PINN, following [47, 14, 2], we decompose the error into approximation error [80, 70, 66] and generalization error [3, 79, 18, 63, 70]. However, instead of the O(1/n)O(1/\sqrt{n}) (nn is the number of data sampled) slow rate generalization bounds established in prior work [47, 66, 78, 67], we utilize the strongly convex structure of the DRM and PINN objectives and provide an O(1/n)O(1/n) fast rate generalization bound [3, 79] that leads us to a non-parametric estimation bound. Our theory also suggests an optimal selection of network size with respect to the number of sampled data. Moreover, to illustrate the optimality of our upper bound, we also establish an information-theoretic lower bound which matches our upper bound for PINN and a modified version of DRM.

We also test our theory by numerical experiments. Recent works [31, 37, 62, 51] studying a variety of deep learning algorithms all find the same polynomial scaling relation between the testing error and the number of data. As the number of training data nn increases, the population loss L\mathcal{L} of well-trained and well-tuned models scales with nn as a power-law L∝1nα\mathcal{L} \propto\frac{1}{n^\alpha} for some α\alpha. [65] also scans over a large range of α\alpha and problem dimension dd and finds an approximately α∝1d\alpha\propto\frac{1}{d} scaling law. In § 4, we conduct numerical experiments to show that this phenomenon still appears for deep PDE solvers and this neural scaling law tests more idiosyncratic features of the theory.

Related Works

Neural Scaling Law. The starting point of our work is the recent observation across speech, vision and text [31, 37, 62, 61] that the empirical performance of a model satisfies a power law scales as a power-law with model size and dataset size. [65] further finds out that the power of the scaling law depends on the intrinsic dimension of the dataset. Theoretical works [63, 70, 72, 9, 33, 18, 36] explore the optimal power law under the non-parametric curve estimation setting via a plug-in neural network. Our work extends this line of research to solving PDEs.

Deep Network Based PDE Solver. Solving high dimensional partial differential equations (PDEs) has been a long-standing challenge due to the curse of dimensionality. At the same time, deep learning has shown superior flexibility and adaptivity in approximating high dimensional functions, which leads to state-of-the-art performances in a wide range of tasks ranging from computer vision to natural language processing. Recent years, pioneer works [26, 58, 45, 68, 38] try to utilize the deep neural networks to solve different types of PDEs and achieve impressive results in many tasks [46, 42]. Based on the natural idea of representing solutions of PDEs by (deep) neural networks, different loss functions for solving PDEs are proposed. [26, 27] utilize the Feynman-Kac formulation which turns solving PDE to a stochastic control problem and the weak adversarial network [81] solves the weak formulations of PDEs via an adversarial network. In this paper, we focus on the convergence rate of the Deep Ritz Method (DRM) [15, 38] and Physics-Informed neural network (PINN) [58, 68]. DRM [15, 38] utilizes the variational structure of the PDE, which is similar to the Ritz-Galerkin method in classical numerical analysis of PDEs, and trains a neural network to minimize the variational objective. PINN [58, 68] trains a neural network directly to minimize the residual of the PDE, i.e., using the strong form of the PDE.

Theoretical Guarantees For Machine Learning Based PDE Solvers. Theoretical convergence results for deep learning based PDE solvers raises wide interest recently. Specifically, [47, 23, 50, 77, 78, 67, 2] investigate the regularity of PDEs approximated by neural network and [47, 48] further provide a generalization analysis. [54] introduces a prior over the solution of the PDE and considers an equivalent white noise model [6]. [54] provides the rate of convergence of the posterior. Our paper does not need to introduce the prior on the target function and provides a non-asymptotic guarantee for finite number of data. At the same time, [54] can only be applied to linear PDEs while our proof technique can be extend to nonlinear ones. All these papers also fail to answer the question that how to determine the network size corresponding to the sampled data number to achieve a desired statistical convergence rate. [32, 49] consider the similar problem for the optimal transport problem, i.e. Monge-ampere equation. Nevertheless, the variational problem we considered is different from [32, 49] and leads to technical difference. The most related works to ours are two concurrent papers [14, 34, 35]. However, our upper bound is faster than [14, 34, 35]. In this paper, we also show that generalization analysis in [47, 14, 48] are loose due to the lack of a localization technique [12, 3, 39, 78]. With observation of the strong convexity of the loss function, we follow the fast rate results for ERM [63, 79, 18] and provide a near optimal bound for both DRM and PINN.

Contribution

In short, we summarize our contribution as follows

  • In this paper, we first considered the statistical limit of learning a PDE solution from sampled observations. The lower bound shows a non-standard exponent different from non-parametric estimation of a function.

  • Instead of the O(1/n)O(1/\sqrt{n}) slow rate generalization bounds in [47, 14, 34, 36], we utilized the strongly convex nature of the variational form and provided a fast rate generalization bound via the localization methods [75, 3, 39, 69, 79]. We discovered that the current Deep Ritz Methods is sub-optimal and propose a modified version of it. We showed that PINN and the modified version of DRM can achieve nearly min-max optimal convergence rate. Our result is listed in Table 1.

  • We tested the recently discovered neural scaling law [31, 37, 62, 28] for deep PDE solvers numerically. The empirical results verified our theory.

Objective FunctionNeural NetworkPrevious BoundFourier BasisLower Bound
Deep Ritzn−2s−2d+2s−2log⁡nn^{-\frac{2s-2}{d+2s-2}}\log nn−2s−2d+4s−4log⁡nn^{-\frac{2s-2}{d+4s-4}}\log n [14]n−2s−2d+2s−2n^{-\frac{2s-2}{d+2s-2}}n−2s−2d+2s−4n^{-\frac{2s-2}{d+2s-4}}
Modified Deep Ritzn−2s−2d+2s−2log⁡nn^{-\frac{2s-2}{d+2s-2}}\log n/n−2s−2d+2s−4n^{-\frac{2s-2}{d+2s-4}}n−2s−2d+2s−4n^{-\frac{2s-2}{d+2s-4}}
PINNn−2s−4d+2s−4log⁡nn^{-\frac{2s-4}{d+2s-4}}\log nn−2s−4d+4s−8log⁡nn^{-\frac{2s-4}{d+4s-8}}\log n [34]n−2s−4d+2s−4n^{-\frac{2s-4}{d+2s-4}}n−2s−4d+2s−4n^{-\frac{2s-4}{d+2s-4}}

Table 1. Upper bounds and lower bounds we achieve in this paper and previous work. The upper bound colored in red indicates that the convergence rate matches the min-max lower bound.

Set-up

We consider the static Schrödinger equation with zero Dirichlet boundary conditions on the domain Ω\Omega, which we assume to be the unit hypercube in Rd\mathbb{R}^{d}. In order to precisely introduce the problem, we recall some standard notions. We consider our domain as Ω=[0,1]d\Omega= [0,1]^{d} and use L2(Ω)L^{2}(\Omega) to denote the space of square integrable functions on Ω\Omega with respect to the Lebesgue measure. We let L∞(Ω)L^{\infty}(\Omega) be the space of essentially bounded (with respect to the Lebesgue measure) functions on Ω\Omega and C(∂Ω)C(\partial\Omega) denotes the space of continuous functions on ∂Ω\partial\Omega.

Let f∈L2(Ω)f \in L^{2}(\Omega), V∈L∞(Ω)V \in L^{\infty}(\Omega), and g∈L∞(Ω)g \in L^{\infty}(\Omega). Our focus is on the analysis of Deep-

Learning-based numerical methods to solve the elliptic equations

−Δu+Vu=fin Ω,u=gon ∂Ω.(1)\begin{aligned} -\Delta u + Vu &= f &&\text{in }\Omega,\\ u &= g &&\text{on }\partial\Omega. \tag*{(1)} \end{aligned}

Loss Functions for Solving PDEs and Induced Evaluation Metric

In this paper, we mainly focus on analyzing Deep Ritz Methods (DRM) and Physics Informed Neural Network (PINN). In this subsection, we first introduce the objective function and algorithm of the two methods.

Deep Ritz Methods. [15, 68] Recall that the equation 2.1 is equivalent to following variational form

u∗=arg⁡min⁡u∈H01(Ω) EDRM(u):=12∫Ω∣∇u∣2+V∣u∣2 dx−∫Ωfudx,(2)u^*=\underset{u\in H_0^1(\Omega)}{\arg\min}\ \mathcal{E}^{\mathrm{DRM}}(u):=\frac{1}{2}\int_{\Omega}|\nabla u|^2+V|u|^2\,dx-\int_{\Omega}fudx, \tag*{(2)}

where uu is minimized over H01(Ω)H_0^1(\Omega) with boundary condition given by gg on ∂Ω\partial\Omega.

This variational form provides the basis for the DRM type method for solving the static Schrödinger equation based on neural network ansatz. More specifically, the energy functional given in equation 2.2 is viewed as the population risk function to train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F⊂H1(Ω)\mathcal{F}\subset H^1(\Omega). In this paper, we also rely on the strong convexity of the DRM objective respect to the H1H^1 norm.

Proposition 2.1. We further assume 0<Vmin⁡≤V(x)≤Vmax⁡0<V_{\min}\le V(x)\le V_{\max}, then we have

2max⁡{1,Vmax⁡}(EDRM(u)−EDRM(u∗))≤∥u−u∗∥H12≤2max⁡{1,Vmin⁡}(EDRM(u)−EDRM(u∗))\frac{2}{\max\{1,V_{\max}\}}\left(\mathcal{E}^{\mathrm{DRM}}(u)-\mathcal{E}^{\mathrm{DRM}}(u^*)\right)\le\lVert u-u^*\rVert_{H^1}^2\le\frac{2}{\max\{1,V_{\min}\}}\left(\mathcal{E}^{\mathrm{DRM}}(u)-\mathcal{E}^{\mathrm{DRM}}(u^*)\right)

holds for all u∈H01(Ω)u\in H_0^1(\Omega)

Physics-Informed Neural Network. [58, 68]. PINN solves 2.1 via minimizing the following objective function

u∗=arg⁡min⁡u∈H01(Ω) EPINN(u):=arg⁡min⁡u∈H01(Ω)∫Ω∣Δu(x)−V(x)u(x)+f(x)∣2dx.u^*=\underset{u\in H_0^1(\Omega)}{\arg\min}\ \mathcal{E}^{\mathrm{PINN}}(u):=\underset{u\in H_0^1(\Omega)}{\arg\min}\int_{\Omega}|\Delta u(x)-V(x)u(x)+f(x)|^2dx.

The objective function EPINN\mathcal{E}^{\mathrm{PINN}} can also be viewed as the population risk function and we can train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F⊂H01(Ω)\mathcal{F} \subset H^1_0(\Omega). In this paper, we also rely on the strong convexity of the PINN objective with respect to the H2H^2 norm, for which we need some additional assumptions on the potential.

References

  1. [1]R. A. Adams and J. J. Fournier, Sobolev spaces, Elsevier, 2003.
  2. [2]G. Bai, U. Koley, S. Mishra, and R. Molinaro, Physics informed neural networks (PINNs) for approximating nonlinear dispersive pdes, arXiv preprint arXiv:2104.05584, (2021).arxiv.org/abs/2104.05584
  3. [3]P. L. Bartlett, O. Bousquet, S. Mendelson, et al., Local rademacher complexities, The Annals of Statistics, 33 (2005), pp. 1497–1537.
  4. [4]K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart, Model reduction and neural networks for parametric pdes, arXiv preprint arXiv:2005.03180, (2020).arxiv.org/abs/2005.03180
  5. [5]H. Brezis, Functional analysis, Sobolev spaces and partial differential equations, Springer Science & Business Media, 2010.DOI
  6. [6]L. D. Brown and M. G. Low, Asymptotic equivalence of nonparametric regression and white noise, The Annals of Statistics, 24 (1996), pp. 2384–2398.DOI
  7. [7]J. Chen, R. Du, and K. Wu, A comprehensive study of boundary conditions when solving pdes by dnns, arXiv preprint arXiv:2005.04554, (2020).
  8. [8]M. Chen, H. Jiang, W. Liao, and T. Zhao, Efficient approximation of deep relu networks for functions on low dimensional manifolds, Advances in Neural Information Processing Systems, 32 (2019), pp. 8174–8184.
  9. [9]M. Chen, H. Jiang, W. Liao, and T. Zhao, Nonparametric regression on low-dimensional manifolds using deep relu networks, arXiv preprint arXiv:1908.01842, (2019).arxiv.org/abs/1908.01842
  10. [10]Y. Chen, B. Hosseini, H. Owhadi, and A. M. Stuart, Solving and learning nonlinear pdes with gaussian processes, arXiv preprint arXiv:2103.12959, (2021).arxiv.org/abs/2103.12959
  11. [11]L. Chizat, Convergence rates of gradient methods for convex optimization in the space of measures, arXiv preprint arXiv:2105.08368, (2021).arxiv.org/abs/2105.08368
  12. [12]C. De Boor and C. De Boor, A practical guide to splines, vol. 27, springer-verlag New York, 1978.
  13. [13]G. De Palma, B. T. Kiani, and S. Lloyd, Random deep neural networks are biased towards simple functions, arXiv preprint arXiv:1812.10156, (2018).arxiv.org/abs/1812.10156
  14. [14]C. Duan, Y. Jiao, Y. Lai, X. Lu, and Z. Yang, Convergence rate analysis for deep ritz method, arXiv preprint arXiv:2103.13330, (2021).DOI
  15. [15]W. E and B. Yu, The deep ritz method: a deep learning-based numerical algorithm for solving variational problems, Communications in Mathematics and Statistics, 6 (2018), pp. 1–12.arxiv.org/abs/1710.00211
  16. [16]L. C. Evans, Partial differential equations, Graduate studies in mathematics, 19 (1998), p. 7.
  17. [17]Y. Fan and L. Ying, Solving electrical impedance tomography with deep learning, Journal of Computational Physics, 404 (2020), p. 109119.DOI
  18. [18]M. H. Farrell, T. Liang, and S. Misra, Deep neural networks for estimation and inference, Econometrica, 89 (2021), pp. 181–213.
  19. [19]J. Feliu-Faba, Y. Fan, and L. Ying, Meta-learning pseudo-differential operators with deep neural networks, Journal of Computational Physics, 408 (2020), p. 109309.
  20. [20]S. A. Geer and S. van de Geer, Empirical Processes in M-estimation, vol. 6, Cambridge university press, 2000.
  21. [21]M. Germain, H. Pham, and X. Warin, Neural networks-based algorithms for stochastic control and pdes in finance, arXiv preprint arXiv:2101.08068, (2021).arxiv.org/abs/2101.08068
  22. [22]D. Guitton, G. Ongie, and R. Willett, Neural networks for linear inverse problems in imaging, IEEE Transactions on Computational Imaging, 6 (2019), pp. 328–343.
  23. [23]P. Grohs and L. Herrmann, Deep neural network approximation for high-dimensional elliptic pdes with boundary conditions, arXiv preprint arXiv:2007.05384, (2020).arxiv.org/abs/2007.05384
  24. [24]P. Grohs, F. Hornung, A. Jentzen, and P. von Wurstemberger, A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of black-scholes partial differential equations, arXiv preprint arXiv:1809.02362, (2018).DOI
  25. [25]I. Gü​hring, G. Kutyniok, and P. Petersen, Error bounds for approximations with deep relu neural networks in w s, p norms, Analysis and Applications, 18 (2020), pp. 803–859.arxiv.org/abs/1902.07896
  26. [26]J. Han, A. Jentzen, and E. Weinan, Solving high-dimensional partial differential equations using deep learning, Proceedings of the National Academy of Sciences, 115 (2018), pp. 8505–8510.DOI
  27. [27]J. Han, J. Lu, and M. Zhou, Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion monte carlo like approach, Journal of Computational Physics, 423 (2020), p. 109792.
  28. [28]T. Hashimoto, Predicting the impact of dataset composition on model performance, 2021, https://openreview.net/forum?id=butEPeLARP_.
  29. [29]K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026–1034.arxiv.org/abs/1502.01852
  30. [30]J. B. Heaton, N. G. Polson, and J. H. Witte, Deep learning for finance: deep portfolios, Applied Stochastic Models in Business and Industry, 33 (2017), pp. 3–12.DOI
  31. [31]J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou, Deep learning scaling is predictable, empirically, arXiv preprint arXiv:1712.00409, (2017).arxiv.org/abs/1712.00409
  32. [32]J.-C. Hütter and P. Rigollet, Minimax rates of estimation for smooth optimal transport maps, arXiv preprint arXiv:1905.05828, (2019).arxiv.org/abs/1905.05828
  33. [33]M. Imaizumi and K. Fukumizu, Advantage of deep neural networks for estimating functions with singularity on curves, arXiv preprint arXiv:2011.02256, (2020).arxiv.org/abs/2011.02256
  34. [34]Y. Jiao, Y. Lai, D. Li, X. Lu, Y. Wang, and J. Z. Yang, Convergence analysis for the pinns, 2021, https://arxiv.org/abs/2109.01780.
  35. [35]Y. Jiao, Y. Lai, Y. Luo, Y. Wang, and Y. Yang, Error analysis of deep ritz methods for elliptic equations, arXiv preprint arXiv:2107.14478, (2021).arxiv.org/abs/2107.14478
  36. [36]Y. Jiao, G. Shen, Y. Lin, and J. Huang, Deep nonparametric regression on approximately low-dimensional manifolds, arXiv preprint arXiv:2104.06708, (2021).
  37. [37]J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361, (2020).arxiv.org/abs/2001.08361
  38. [38]Y. Khoo, J. Lu, and L. Ying, Solving parametric pde problems with artificial neural networks, arXiv preprint arXiv:1707.03351, (2017).DOI
  39. [39]V. Koltchinskii, Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems: Ecole d’Été de Probabilités de Saint-Flour XXXVIII-2008, vol. 2033, Springer Science & Business Media, 2011.
  40. [40]S. Lanthaler, S. Mishra, and G. E. Karniadakis, Error estimates for deeponets: A deep learning framework in infinite dimensions, arXiv preprint arXiv:2102.09618, (2021).arxiv.org/abs/2102.09618
  41. [41]M. Ledoux and M. Talagrand, Probability in Banach Spaces: isoperimetry and processes, Springer Science & Business Media, 2013.
  42. [42]Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, Fourier neural operator for parametric partial differential equations, arXiv preprint arXiv:2010.08895, (2020).arxiv.org/abs/2010.08895
  43. [43]Y. Liu and H. Wang, Convergence order of wavelet thresholding estimator for differential operators on besov spaces, Applied and Computational Harmonic Analysis, 32 (2012), pp. 342–356.DOI
  44. [44]Z. Long, Y. Lu, and B. Dong, Pde-net 2.0: Learning pdes from data with a numeric-symbolic hybrid deep network, Journal of Computational Physics, 399 (2019), p. 108925.DOI
  45. [45]Z. Long, Y. Lu, X. Ma, and B. Dong, Pde-net: Learning pdes from data, in International Conference on Machine Learning, PMLR, 2018, pp. 3208–3216.arxiv.org/abs/1710.09668
  46. [46]D. Lu, H. Wang, M. Chen, L. Lin, R. Car, E. Weinan, W. Jia, and L. Zhang, 86 flops deep potential molecular dynamics simulation of 100 million atoms with ab initio accuracy, Computer Physics Communications, 259 (2021), p. 107624.DOI
  47. [47]J. Lu, Y. Lu, and M. Wang, A priori generalization analysis of the deep ritz method for solving high dimensional elliptic equations, arXiv preprint arXiv:2101.01708, (2021).arxiv.org/abs/2101.01708
  48. [48]T. Luo and H. Yang, Two-layer neural networks for partial differential equations: Optimization and generalization theory, arXiv preprint arXiv:2006.15733, (2020).arxiv.org/abs/2006.15733
  49. [49]T. Manole, S. Balakrishnan, J. Niles-Weed, and L. Wasserman, Plugin estimation of smooth optimal transport maps, arXiv preprint arXiv:2107.12364, (2021).arxiv.org/abs/2107.12364
  50. [50]T. Marwah, Z. C. Lipton, and A. Risteski, Parametric complexity bounds for approximating pdes with neural networks, arXiv preprint arXiv:2103.02138, (2021).arxiv.org/abs/2103.02138
  51. [51]H. Mikami, K. Fukumizu, S. Murai, S. Suzuki, Y. Kikuchi, T. Suzuki, S.-I. Maeda, and K. Hayashi, A scaling law for synthetic-to-real transfer: A measure of pre-training, arXiv preprint arXiv:2108.11018, (2021).
  52. [52]H. Montanelli and Q. Du, New error bounds for deep relu networks using sparse grids, SIAM Journal on Mathematics of Data Science, 1 (2019), pp. 78–92.arxiv.org/abs/1712.08688
  53. [53]H.-G. Müller and T. Gasser, Optimal convergence properties of kernel estimates of derivatives of a density function, in Smoothing techniques for curve estimation, Springer, 1979, pp. 144–154.DOI
  54. [54]R. Nickl, S. van de Geer, and S. Wang, Convergence rates for penalized least squares estimators in pde constrained regression problems, SIAM/ASA Journal on Uncertainty Quantification, 8 (2020), pp. 374–413.DOI
  55. [55]A. Nitanda and T. Suzuki, Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime, arXiv preprint arXiv:2006.12297, (2020).arxiv.org/abs/2006.12297
  56. [56]G. Ongie, R. Willett, D. Soudry, and N. Srebro, A function space view of bounded norm infinite width relu nets: The multivariate case, arXiv preprint arXiv:1910.01635, (2019).arxiv.org/abs/1910.01635
  57. [57]B. Prakasa Rao, Nonparametric estimation of the derivatives of a density by the method of wavelets, Bulletin of Informatics and Cybernetics, 28 (1996), pp. 91–100.DOI
  58. [58]M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics, 378 (2019), pp. 686–707.DOI
  59. [59]A. Rakhlin, K. Sridharan, and A. B. Tsybakov, Empirical entropy, minimax regret and minimax risk, Bernoulli, 23 (2017), pp. 789–824.DOI
  60. [60]L. Richter, L. Sallandt, and N. Nüsken, Solving high-dimensional parabolic pdes using the tensor train format, arXiv preprint arXiv:2102.11830, (2021).arxiv.org/abs/2102.11830
  61. [61]J. S. Rosenfeld, Scaling laws for deep learning, arXiv preprint arXiv:2108.07686, (2021).arxiv.org/abs/2108.07686
  62. [62]J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit, A constructive prediction of the generalization error across scales, arXiv preprint arXiv:1909.12673, (2019).arxiv.org/abs/1909.12673
  63. [63]J. Schmidt-Hieber et al., Nonparametric regression using deep neural networks with relu activation function, Annals of Statistics, 48 (2020), pp. 1875–1897.
  64. [64]L. Schumaker, Spline functions: basic theory, Cambridge University Press, 2007.
  65. [65]U. Sharma and J. Kaplan, A neural scaling law from the dimension of the data manifold, arXiv preprint arXiv:2004.10802, (2020).arxiv.org/abs/2004.10802
  66. [66]Z. Shen, H. Yang, and S. Zhang, Neural network approximation: Three hidden layers are enough, Neural Networks, (2021).DOI
  67. [67]Y. Shin, Z. Zhang, and G. E. Karniadakis, Error estimates of residual minimization using neural networks for linear pdes, arXiv preprint arXiv:2010.08019, (2020).arxiv.org/abs/2010.08019
  68. [68]J. Sirignano and K. Spiliopoulos, Dgm: A deep learning algorithm for solving partial differential equations, Journal of Computational Physics, 375 (2018), pp. 1339–1364.DOI
  69. [69]N. Srebro, K. Sridharan, and A. Tewari, Smoothness, low noise and fast rates, Advances in neural information processing systems, 23 (2010).
  70. [70]T. Suzuki, Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality, arXiv preprint arXiv:1810.08033, (2018).arxiv.org/abs/1810.08033
  71. [71]T. Suzuki and S. Akiyama, Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methods, arXiv preprint arXiv:2012.03224, (2020).arxiv.org/abs/2012.03224
  72. [72]T. Suzuki and A. Nitanda, Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic besov space, arXiv preprint arXiv:1910.12799, (2019).arxiv.org/abs/1910.12799
  73. [73]J. A. Tropp, An introduction to matrix concentration inequalities, arXiv preprint arXiv:1501.01571, (2015).arxiv.org/abs/1501.01571
  74. [74]A. B. Tsybakov, Introduction to nonparametric estimation, Springer Science & Business Media, 2008.
  75. [75]S. Van De Geer, A new approach to least-squares estimation, with applications, The Annals of Statistics, 15 (1987), pp. 587–602.
  76. [76]Q. Wang et al., Exponential convergence of the deep neural network approximation for analytic functions, arXiv preprint arXiv:1807.00297, (2018).arxiv.org/abs/1807.00297
  77. [77]S. Wojtowytsch et al., Some observations on partial differential equations in barron and multi-layer spaces, arXiv preprint arXiv:2012.01484, (2020).
  78. [78]J. Xu, The finite neuron method and convergence analysis, arXiv preprint arXiv:2010.01458, (2020).DOI
  79. [79]Y. Xu and A. Zeevi, Towards optimal problem dependent generalization error bounds in statistical learning theory, arXiv preprint arXiv:2011.06186, (2020).arxiv.org/abs/2011.06186
  80. [80]D. Yarotsky, Error bounds for approximations with deep relu networks, Neural Networks, 94 (2017), pp. 103–114.DOI
  81. [81]Y. Zang, G. Bao, X. Ye, and H. Zhou, Weak adversarial networks for high-dimensional partial differential equations, Journal of Computational Physics, 411 (2020), p. 109409.DOI
  82. [82]X. Zhang, Y. Lu, J. Liu, and B. Dong, Dynamically unfolding recurrent restorer: A moving endpoint control method for image restoration, arXiv preprint arXiv:1805.07709, (2018).arxiv.org/abs/1805.07709

Paper details

vlnc:W_gdpavy8q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents