Introduction
Partial differential equations (PDEs) play a prominent role in many disciplines of science and engineering. The recent deep learning breakthrough and the rapid development of sensors, computational power, and data storage in the past decade has drawn attention to numerically solving PDEs via machine learning methods [45 , 44 , 58 , 26 , 68 , 38 ] , especially in high dimensions where conventional methods become impractical. The set of applications that motivate this interest is wide-ranging, including computational physics [26 , 45 , 58 ] , inverse problem [82 , 22 , 17 ] and quantitative finance [30 , 21 ] . The numerical methods generated by the use of deep learning techniques are mesh-less methods, see the discussion in [78 ] . A natural deep learning technique in the problems that are based on a standard feed-forward type of architecture takes advantage (when available) of a variational formulation, whose solution coincides with the solution of the PDE of interest. Despite the success and popularity of adopting neural networks for solving high-dimensional PDEs, the following question still remains poorly answered.
For a given PDE and a data-driven approximation architecture, how large the sample size and how complex the model are needed to reach a prescribed performance level?
In this paper, we aim to establish the numerical analysis of such deep learning based PDE solvers. Inspired by recent works which showed that the empirical performance of a model is remarkably predictable via a power law of the data number, known as the neural scaling law [37 , 31 , 65 ] , we aim to explore the neural scaling law for deep PDE solvers and compare its performance to Fourier approximation.
Among the various approaches of using deep learning methods for solving PDEs, in this work, we focus on the Deep Ritz method (DRM) [15 , 38 ] and the Physics-Informed Neural Networks (PINN) approach [68 , 58 ] , both of which are based on minimizing neural network parameters according to some loss functional related to the PDEs. To provide theoretical guarantees for DRM and PINN, following [47 , 14 , 2 ] , we decompose the error into approximation error [80 , 70 , 66 ] and generalization error [3 , 79 , 18 , 63 , 70 ] . However, instead of the O ( 1 / n ) O(1/\sqrt{n}) O ( 1/ n ) (n n n is the number of data sampled) slow rate generalization bounds established in prior work [47 , 66 , 78 , 67 ] , we utilize the strongly convex structure of the DRM and PINN objectives and provide an O ( 1 / n ) O(1/n) O ( 1/ n ) fast rate generalization bound [3 , 79 ] that leads us to a non-parametric estimation bound. Our theory also suggests an optimal selection of network size with respect to the number of sampled data. Moreover, to illustrate the optimality of our upper bound, we also establish an information-theoretic lower bound which matches our upper bound for PINN and a modified version of DRM.
We also test our theory by numerical experiments. Recent works [31 , 37 , 62 , 51 ] studying a variety of deep learning algorithms all find the same polynomial scaling relation between the testing error and the number of data. As the number of training data n n n increases, the population loss L \mathcal{L} L of well-trained and well-tuned models scales with n n n as a power-law L ∝ 1 n α \mathcal{L} \propto\frac{1}{n^\alpha} L ∝ n α 1 for some α \alpha α . [65 ] also scans over a large range of α \alpha α and problem dimension d d d and finds an approximately α ∝ 1 d \alpha\propto\frac{1}{d} α ∝ d 1 scaling law. In § 4 , we conduct numerical experiments to show that this phenomenon still appears for deep PDE solvers and this neural scaling law tests more idiosyncratic features of the theory.
Related Works
Neural Scaling Law. The starting point of our work is the recent observation across speech, vision and text [31 , 37 , 62 , 61 ] that the empirical performance of a model satisfies a power law scales as a power-law with model size and dataset size. [65 ] further finds out that the power of the scaling law depends on the intrinsic dimension of the dataset. Theoretical works [63 , 70 , 72 , 9 , 33 , 18 , 36 ] explore the optimal power law under the non-parametric curve estimation setting via a plug-in neural network. Our work extends this line of research to solving PDEs.
Deep Network Based PDE Solver. Solving high dimensional partial differential equations (PDEs) has been a long-standing challenge due to the curse of dimensionality. At the same time, deep learning has shown superior flexibility and adaptivity in approximating high dimensional functions, which leads to state-of-the-art performances in a wide range of tasks ranging from computer vision to natural language processing. Recent years, pioneer works [26 , 58 , 45 , 68 , 38 ] try to utilize the deep neural networks to solve different types of PDEs and achieve impressive results in many tasks [46 , 42 ] . Based on the natural idea of representing solutions of PDEs by (deep) neural networks, different loss functions for solving PDEs are proposed. [26 , 27 ] utilize the Feynman-Kac formulation which turns solving PDE to a stochastic control problem and the weak adversarial network [81 ] solves the weak formulations of PDEs via an adversarial network. In this paper, we focus on the convergence rate of the Deep Ritz Method (DRM) [15 , 38 ] and Physics-Informed neural network (PINN) [58 , 68 ] . DRM [15 , 38 ] utilizes the variational structure of the PDE, which is similar to the Ritz-Galerkin method in classical numerical analysis of PDEs, and trains a neural network to minimize the variational objective. PINN [58 , 68 ] trains a neural network directly to minimize the residual of the PDE, i.e., using the strong form of the PDE.
Theoretical Guarantees For Machine Learning Based PDE Solvers. Theoretical convergence results for deep learning based PDE solvers raises wide interest recently. Specifically, [47 , 23 , 50 , 77 , 78 , 67 , 2 ] investigate the regularity of PDEs approximated by neural network and [47 , 48 ] further provide a generalization analysis. [54 ] introduces a prior over the solution of the PDE and considers an equivalent white noise model [6 ] . [54 ] provides the rate of convergence of the posterior. Our paper does not need to introduce the prior on the target function and provides a non-asymptotic guarantee for finite number of data. At the same time, [54 ] can only be applied to linear PDEs while our proof technique can be extend to nonlinear ones. All these papers also fail to answer the question that how to determine the network size corresponding to the sampled data number to achieve a desired statistical convergence rate. [32 , 49 ] consider the similar problem for the optimal transport problem, i.e. Monge-ampere equation. Nevertheless, the variational problem we considered is different from [32 , 49 ] and leads to technical difference. The most related works to ours are two concurrent papers [14 , 34 , 35 ] . However, our upper bound is faster than [14 , 34 , 35 ] . In this paper, we also show that generalization analysis in [47 , 14 , 48 ] are loose due to the lack of a localization technique [12 , 3 , 39 , 78 ] . With observation of the strong convexity of the loss function, we follow the fast rate results for ERM [63 , 79 , 18 ] and provide a near optimal bound for both DRM and PINN.
Contribution
In short, we summarize our contribution as follows
In this paper, we first considered the statistical limit of learning a PDE solution from sampled observations. The lower bound shows a non-standard exponent different from non-parametric estimation of a function.
Instead of the O ( 1 / n ) O(1/\sqrt{n}) O ( 1/ n ) slow rate generalization bounds in [47 , 14 , 34 , 36 ] , we utilized the strongly convex nature of the variational form and provided a fast rate generalization bound via the localization methods [75 , 3 , 39 , 69 , 79 ] . We discovered that the current Deep Ritz Methods is sub-optimal and propose a modified version of it. We showed that PINN and the modified version of DRM can achieve nearly min-max optimal convergence rate. Our result is listed in Table 1 .
We tested the recently discovered neural scaling law [31 , 37 , 62 , 28 ] for deep PDE solvers numerically. The empirical results verified our theory.
Table 1. Upper bounds and lower bounds we achieve in this paper and previous work. The upper bound colored in red indicates that the convergence rate matches the min-max lower bound.
Set-up
We consider the static Schrödinger equation with zero Dirichlet boundary conditions on the domain Ω \Omega Ω , which we assume to be the unit hypercube in R d \mathbb{R}^{d} R d . In order to precisely introduce the problem, we recall some standard notions. We consider our domain as Ω = [ 0 , 1 ] d \Omega= [0,1]^{d} Ω = [ 0 , 1 ] d and use L 2 ( Ω ) L^{2}(\Omega) L 2 ( Ω ) to denote the space of square integrable functions on Ω \Omega Ω with respect to the Lebesgue measure. We let L ∞ ( Ω ) L^{\infty}(\Omega) L ∞ ( Ω ) be the space of essentially bounded (with respect to the Lebesgue measure) functions on Ω \Omega Ω and C ( ∂ Ω ) C(\partial\Omega) C ( ∂ Ω ) denotes the space of continuous functions on ∂ Ω \partial\Omega ∂ Ω .
Let f ∈ L 2 ( Ω ) f \in L^{2}(\Omega) f ∈ L 2 ( Ω ) , V ∈ L ∞ ( Ω ) V \in L^{\infty}(\Omega) V ∈ L ∞ ( Ω ) , and g ∈ L ∞ ( Ω ) g \in L^{\infty}(\Omega) g ∈ L ∞ ( Ω ) . Our focus is on the analysis of Deep-
Learning-based numerical methods to solve the elliptic equations
− Δ u + V u = f in Ω , u = g on ∂ Ω . (1) \begin{aligned}
-\Delta u + Vu &= f &&\text{in }\Omega,\\
u &= g &&\text{on }\partial\Omega. \tag*{(1)}
\end{aligned} − Δ u + V u u = f = g in Ω , on ∂ Ω. (1)
Loss Functions for Solving PDEs and Induced Evaluation Metric
In this paper, we mainly focus on analyzing Deep Ritz Methods (DRM) and Physics Informed Neural Network (PINN). In this subsection, we first introduce the objective function and algorithm of the two methods.
Deep Ritz Methods. [15 , 68 ] Recall that the equation 2.1 is equivalent to following variational form
u ∗ = arg min u ∈ H 0 1 ( Ω ) E D R M ( u ) : = 1 2 ∫ Ω ∣ ∇ u ∣ 2 + V ∣ u ∣ 2 d x − ∫ Ω f u d x , (2) u^*=\underset{u\in H_0^1(\Omega)}{\arg\min}\ \mathcal{E}^{\mathrm{DRM}}(u):=\frac{1}{2}\int_{\Omega}|\nabla u|^2+V|u|^2\,dx-\int_{\Omega}fudx, \tag*{(2)} u ∗ = u ∈ H 0 1 ( Ω ) arg min E DRM ( u ) := 2 1 ∫ Ω ∣∇ u ∣ 2 + V ∣ u ∣ 2 d x − ∫ Ω f u d x , (2)
where u u u is minimized over H 0 1 ( Ω ) H_0^1(\Omega) H 0 1 ( Ω ) with boundary condition given by g g g on ∂ Ω \partial\Omega ∂ Ω .
This variational form provides the basis for the DRM type method for solving the static Schrödinger equation based on neural network ansatz. More specifically, the energy functional given in equation 2.2 is viewed as the population risk function to train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F ⊂ H 1 ( Ω ) \mathcal{F}\subset H^1(\Omega) F ⊂ H 1 ( Ω ) . In this paper, we also rely on the strong convexity of the DRM objective respect to the H 1 H^1 H 1 norm.
Proposition 2.1. We further assume 0 < V min ≤ V ( x ) ≤ V max 0<V_{\min}\le V(x)\le V_{\max} 0 < V m i n ≤ V ( x ) ≤ V m a x , then we have
2 max { 1 , V max } ( E D R M ( u ) − E D R M ( u ∗ ) ) ≤ ∥ u − u ∗ ∥ H 1 2 ≤ 2 max { 1 , V min } ( E D R M ( u ) − E D R M ( u ∗ ) ) \frac{2}{\max\{1,V_{\max}\}}\left(\mathcal{E}^{\mathrm{DRM}}(u)-\mathcal{E}^{\mathrm{DRM}}(u^*)\right)\le\lVert u-u^*\rVert_{H^1}^2\le\frac{2}{\max\{1,V_{\min}\}}\left(\mathcal{E}^{\mathrm{DRM}}(u)-\mathcal{E}^{\mathrm{DRM}}(u^*)\right) max { 1 , V m a x } 2 ( E DRM ( u ) − E DRM ( u ∗ ) ) ≤ ∥ u − u ∗ ∥ H 1 2 ≤ max { 1 , V m i n } 2 ( E DRM ( u ) − E DRM ( u ∗ ) ) holds for all u ∈ H 0 1 ( Ω ) u\in H_0^1(\Omega) u ∈ H 0 1 ( Ω )
Physics-Informed Neural Network. [58 , 68 ] . PINN solves 2.1 via minimizing the following objective function
u ∗ = arg min u ∈ H 0 1 ( Ω ) E P I N N ( u ) : = arg min u ∈ H 0 1 ( Ω ) ∫ Ω ∣ Δ u ( x ) − V ( x ) u ( x ) + f ( x ) ∣ 2 d x . u^*=\underset{u\in H_0^1(\Omega)}{\arg\min}\ \mathcal{E}^{\mathrm{PINN}}(u):=\underset{u\in H_0^1(\Omega)}{\arg\min}\int_{\Omega}|\Delta u(x)-V(x)u(x)+f(x)|^2dx. u ∗ = u ∈ H 0 1 ( Ω ) arg min E PINN ( u ) := u ∈ H 0 1 ( Ω ) arg min ∫ Ω ∣Δ u ( x ) − V ( x ) u ( x ) + f ( x ) ∣ 2 d x .
The objective function E P I N N \mathcal{E}^{\mathrm{PINN}} E PINN can also be viewed as the population risk function and we can train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F ⊂ H 0 1 ( Ω ) \mathcal{F} \subset H^1_0(\Omega) F ⊂ H 0 1 ( Ω ) . In this paper, we also rely on the strong convexity of the PINN objective with respect to the H 2 H^2 H 2 norm, for which we need some additional assumptions on the potential.
Proposition 2.2. For PINN, we further assume V ∈ L ∞ ( Ω ) V \in L^\infty(\Omega) V ∈ L ∞ ( Ω ) with 0 < C min < V 2 − Δ V 0 < C_{\min} < V^2 - \Delta V 0 < C m i n < V 2 − Δ V , 0 < C min < V ( x ) ≤ V max 0 < C_{\min} < V(x) \le V_{\max} 0 < C m i n < V ( x ) ≤ V m a x and − Δ V ( x ) ≤ V max -\Delta V(x) \le V_{\max} − Δ V ( x ) ≤ V m a x , then we have for all u ∈ H 0 1 ( Ω ) u \in H^1_0(\Omega) u ∈ H 0 1 ( Ω )
1 2 ( 1 + V max + V max 2 ) ( E P I N N ( u ) − E P I N N ( u ∗ ) ) ≤ ∥ u − u ∗ ∥ H 2 2 \frac{1}{2(1+V_{\max}+V_{\max}^2)}\left(\mathcal{E}^{\mathrm{PINN}}(u)-\mathcal{E}^{\mathrm{PINN}}(u^*)\right) \le\lVert u-u^*\rVert_{H^2}^2 2 ( 1 + V m a x + V m a x 2 ) 1 ( E PINN ( u ) − E PINN ( u ∗ ) ) ≤ ∥ u − u ∗ ∥ H 2 2 ≤ 2 max { 1 , C min } ( E P I N N ( u ) − E P I N N ( u ∗ ) ) . \le\frac{2}{\max\{1,C_{\min}\}}\left(\mathcal{E}^{\mathrm{PINN}}(u)-\mathcal{E}^{\mathrm{PINN}}(u^*)\right). ≤ max { 1 , C m i n } 2 ( E PINN ( u ) − E PINN ( u ∗ ) ) .
Estimator Setting
Empirical Loss Minimization. In order to access the d d d -dimensional integrals, DRM [15 , 38 ] and PINN [58 , 68 ] employ a Monte-Carlo method for computing the high dimensional integrals, which leads to the so-called empirical risk minimization training for neural networks. To define the empirical loss, let { X j } j = 1 n \{X_j\}_{j=1}^n { X j } j = 1 n be an i.i.d. sequence of random variables distributed according to the uniform distribution in domain Ω \Omega Ω . We also have access to f j = f ( X j ) + ξ j f_j=f(X_j)+\xi_j f j = f ( X j ) + ξ j , j = 1 , ⋯ , n j=1,\cdots,n j = 1 , ⋯ , n is the noisy observation of the right hand side of the PDE (1) and ξ \xi ξ is a bounded random variable with mean zero and independent with X j X_j X j . Define the empirical losses E n , S \mathcal{E}_{n,S} E n , S by setting
E n D R M ( u ) = 1 n ∑ j = 1 n [ ∣ Ω ∣ ⋅ ( 1 2 ∣ ∇ u ( X j ) ∣ 2 + 1 2 V ( X j ) ∣ u ( X j ) ∣ 2 − f j u ( X j ) ) ] . (3) \mathcal{E}^{\mathrm{DRM}}_n(u)=\frac{1}{n}\sum_{j=1}^{n}\left[|\Omega|\cdot\left(\frac{1}{2}|\nabla u(X_j)|^2+\frac{1}{2}V(X_j)|u(X_j)|^2-f_j u(X_j)\right)\right]. \tag*{(3)} E n DRM ( u ) = n 1 j = 1 ∑ n [ ∣Ω∣ ⋅ ( 2 1 ∣∇ u ( X j ) ∣ 2 + 2 1 V ( X j ) ∣ u ( X j ) ∣ 2 − f j u ( X j ) ) ] . (3)
E n P I N N ( u ) = 1 n ∑ j = 1 n [ ∣ Ω ∣ ⋅ ( Δ u ( X j ) − V ( X j ) u ( X j ) + f j ) 2 ] . (4) \mathcal{E}^{\mathrm{PINN}}_n(u)=\frac{1}{n}\sum_{j=1}^{n}\left[|\Omega|\cdot\left(\Delta u(X_j)-V(X_j)u(X_j)+f_j\right)^2\right]. \tag*{(4)} E n PINN ( u ) = n 1 j = 1 ∑ n [ ∣Ω∣ ⋅ ( Δ u ( X j ) − V ( X j ) u ( X j ) + f j ) 2 ] . (4)
where ∣ Ω ∣ |\Omega| ∣Ω∣ represent the Lebesgue measure of the sets.
Once given an empirical loss E n ′ \mathcal{E}'_n E n ′ , we apply the empirical loss minimization to seek the estimation u n u_n u n , i.e. u n = a r g m i n u ∈ F E n ( u ) u_n=\mathop{\rm arg\,min}_{u\in\mathcal{F}}\mathcal{E}_n(u) u n = arg min u ∈ F E n ( u ) where F \mathcal{F} F is the parametrized hypothesis function space we consider. For example, reproducing kernel Hilbert space [10 ] and tensor training format [60 ] . In this paper, we consider sparse neural network and truncated fourier basis, which can achieves min-max optimal estimation rate for the non-parametric function estimation [74 , 63 , 18 , 70 , 9 , 36 , 55 ] .
Sparse Neural Network Function Space. In this paper, the hypothesis function space F \mathcal{F} F is expressed by the neural network following [63 , 70 , 18 ] . Let us denote the ReLU3 ^3 3 activation by η 3 ( x ) = max { x 3 , 0 } \eta_3(x)=\max\{x^3,0\} η 3 ( x ) = max { x 3 , 0 } (x ∈ R x\in\mathbb{R} x ∈ R ) which is used in [15 ] , and for a vector x x x , η ( x ) \eta(x) η ( x ) is operated in an element-wise manner. Define the neural network with height L L L , width W W W , sparsity constraint S S S and norm constraint B B B as
Φ ( L , W , S , B ) : = { ( W ( L ) η 3 ( ⋅ ) + b ( L ) ) ∘ ⋯ ∘ ( W ( 2 ) η 3 ( ⋅ ) + b ( 2 ) ) ∘ ( W ( 1 ) x + b ( 1 ) ) | W ( L ) ∈ R 1 × W , b ( L ) ∈ R , W ( 1 ) ∈ R W × d , b ( 1 ) ∈ R W , W ( l ) ∈ R W × W , b ( l ) ∈ R W ( 1 < l < L ) , ∑ l = 1 L ( ∥ W ( l ) ∥ 0 + ∥ b ( l ) ∥ 0 ) ≤ S , max l ∥ W ( l ) ∥ ∞ , ∞ ∨ ∥ b ( l ) ∥ ∞ ≤ B } . (5) \Phi(L,W,S,B):=\left\{\left(\mathcal{W}^{(L)}\eta_{3}(\cdot)+b^{(L)}\right)\circ\cdots\circ\left(\mathcal{W}^{(2)}\eta_{3}(\cdot)+b^{(2)}\right)\circ\left(\mathcal{W}^{(1)}x+b^{(1)}\right)\ \middle|\
\begin{gathered}
\mathcal{W}^{(L)}\in\mathbb{R}^{1\times W},\ b^{(L)}\in\mathbb{R},\ \mathcal{W}^{(1)}\in\mathbb{R}^{W\times d},\ b^{(1)}\in\mathbb{R}^{W},\\
\mathcal{W}^{(l)}\in\mathbb{R}^{W\times W},\ b^{(l)}\in\mathbb{R}^{W}\ (1<l<L),\\
\sum_{l=1}^{L}\left(\lVert\mathcal{W}^{(l)}\rVert_{0}+\lVert b^{(l)}\rVert_{0}\right)\leq S,\ \max_{l}\lVert\mathcal{W}^{(l)}\rVert_{\infty,\infty}\vee\lVert b^{(l)}\rVert_{\infty}\leq B
\end{gathered}
\right\}. \tag*{(5)} Φ ( L , W , S , B ) := ⎩ ⎨ ⎧ ( W ( L ) η 3 ( ⋅ ) + b ( L ) ) ∘ ⋯ ∘ ( W ( 2 ) η 3 ( ⋅ ) + b ( 2 ) ) ∘ ( W ( 1 ) x + b ( 1 ) ) W ( L ) ∈ R 1 × W , b ( L ) ∈ R , W ( 1 ) ∈ R W × d , b ( 1 ) ∈ R W , W ( l ) ∈ R W × W , b ( l ) ∈ R W ( 1 < l < L ) , l = 1 ∑ L ( ∥ W ( l ) ∥ 0 + ∥ b ( l ) ∥ 0 ) ≤ S , l max ∥ W ( l ) ∥ ∞ , ∞ ∨ ∥ b ( l ) ∥ ∞ ≤ B ⎭ ⎬ ⎫ . (5)
where ∘ \circ ∘ denotes the function composition, ∥ ⋅ ∥ 0 \lVert\cdot\rVert_{0} ∥ ⋅ ∥ 0 is the ℓ 0 \ell_{0} ℓ 0 -norm of the matrix (the number of non-zero elements of the matrix) and ∥ ⋅ ∥ ∞ , ∞ \lVert\cdot\rVert_{\infty,\infty} ∥ ⋅ ∥ ∞ , ∞ is the ℓ ∞ \ell_{\infty} ℓ ∞ -norm of the matrix (maximum of the absolute values of the elements).
Truncated Fourier Basis Estimator. We also considered the Truncated Fourier basis as our estimator. Suppose the domain we interested Ω ⊆ [ 0 , 1 ] d \Omega\subseteq[0,1]^{d} Ω ⊆ [ 0 , 1 ] d . For any z ∈ N d z\in\mathbb{N}^{d} z ∈ N d , we consider the corresponding Fourier basis function ϕ z ( x ) : = e 2 π i ⟨ z , x ⟩ \phi_{z}(x):=e^{2\pi i\langle z,x\rangle} ϕ z ( x ) := e 2 π i ⟨ z , x ⟩ ( x ∈ Ω ) (x\in\Omega) ( x ∈ Ω ) . Any function f ∈ L 2 ( Ω ) f\in L^{2}(\Omega) f ∈ L 2 ( Ω ) can be represented as weighted sum of the Fourier basis f ( x ) : = ∑ z ∈ N d f z ϕ z ( x ) f(x):=\sum_{z\in\mathbb{N}^{d}}f_{z}\phi_{z}(x) f ( x ) := ∑ z ∈ N d f z ϕ z ( x ) where f z : = ∫ Ω f ( x ) ϕ z ( x ) ‾ d x f_{z}:=\int_{\Omega}f(x)\overline{\phi_{z}(x)}dx f z := ∫ Ω f ( x ) ϕ z ( x ) d x ( ∀ z ∈ N d ) (\forall z\in\mathbb{N}^{d}) ( ∀ z ∈ N d ) is the Fourier coefficient. This inspired us to use the Fourier Basis whose index lies in a truncated set Z ξ = { z ∈ Z l ∣ ∥ z ∥ ∞ ≤ ξ } Z_{\xi}=\{z\in\mathbb{Z}^{l}\mid\lVert z\rVert_{\infty}\leq\xi\} Z ξ = { z ∈ Z l ∣ ∥ z ∥ ∞ ≤ ξ } to represent the function class F \mathcal{F} F as F ξ = { ∑ ∥ z ∥ ∞ ≤ ξ a z ϕ z ∣ a z ∈ R , ∥ z ∥ ∞ ≤ ξ } \mathcal{F}_{\xi}=\{\sum_{\lVert z\rVert_{\infty}\leq\xi}a_{z}\phi_{z}\mid a_{z}\in\mathbb{R},\ \lVert z\rVert_{\infty}\leq\xi\} F ξ = { ∑ ∥ z ∥ ∞ ≤ ξ a z ϕ z ∣ a z ∈ R , ∥ z ∥ ∞ ≤ ξ } .
Lower Bound
In this section, we aim to consider the statistical limit of learning the solution of a PDE. As discussed in Propositions 2.1 and 2.2 , we directly consider the H 1 H^{1} H 1 norm for DRM and H 2 H^{2} H 2 norm for PINN as the evaluation metric. The lower bound shown as follows.
Theorem 3.1 (Lower bound). We denote u ∗ ( f ) u^{*}(f) u ∗ ( f ) to be the solution of the PDE 2.1 and we can access randomly sampled data { X i , f i } i = 1 , ⋯ , n \{X_{i},f_{i}\}_{i=1,\cdots,n} { X i , f i } i = 1 , ⋯ , n as described in § 2.2 . We further assume u ∗ ( f ) ∈ H s u^{*}(f)\in H^{s} u ∗ ( f ) ∈ H s for a given s ∈ Z + s\in\mathbb{Z}^{+} s ∈ Z + , we have the following lower bounds.
DRM Lower Bound. For all estimators ψ : ( R d ) ⊗ n × R ⊗ n → H s ( Ω ) \psi:(\mathbb{R}^{d})^{\otimes n}\times\mathbb{R}^{\otimes n}\to H^{s}(\Omega) ψ : ( R d ) ⊗ n × R ⊗ n → H s ( Ω ) , we have
inf ψ sup u ∗ ∈ H s ( Ω ) E ∥ ψ ( { X i , f i } i = 1 , ⋯ , n ) − u ∗ ( f ) ∥ H 1 2 ≳ n − 2 s − 2 d + 2 s − 4 . (6) \inf_{\psi}\sup_{u^{*}\in H^{s}(\Omega)}\mathbb{E}\left\lVert\psi\left(\{X_{i},f_{i}\}_{i=1,\cdots,n}\right)-u^{*}(f)\right\rVert_{H^{1}}^{2}\gtrsim n^{-\frac{2s-2}{d+2s-4}}. \tag*{(6)} ψ inf u ∗ ∈ H s ( Ω ) sup E ∥ ψ ( { X i , f i } i = 1 , ⋯ , n ) − u ∗ ( f ) ∥ H 1 2 ≳ n − d + 2 s − 4 2 s − 2 . (6) PINN Lower Bound. For all estimators ψ : ( R d ) ⊗ n × R ⊗ n → H s ( Ω ) \psi:(\mathbb{R}^{d})^{\otimes n}\times\mathbb{R}^{\otimes n}\to H^{s}(\Omega) ψ : ( R d ) ⊗ n × R ⊗ n → H s ( Ω ) , we have
inf ψ sup u ∗ ∈ H s ( Ω ) E ∥ ψ ( { X i , f i } i = 1 , ⋯ , n ) − u ∗ ( f ) ∥ H 2 2 ≳ n − 2 s − 4 d + 2 s − 4 . (7) \inf_{\psi}\sup_{u^{*}\in H^{s}(\Omega)}\mathbb{E}\left\lVert\psi\left(\{X_{i},f_{i}\}_{i=1,\cdots,n}\right)-u^{*}(f)\right\rVert_{H^{2}}^{2}\gtrsim n^{-\frac{2s-4}{d+2s-4}}. \tag*{(7)} ψ inf u ∗ ∈ H s ( Ω ) sup E ∥ ψ ( { X i , f i } i = 1 , ⋯ , n ) − u ∗ ( f ) ∥ H 2 2 ≳ n − d + 2 s − 4 2 s − 4 . (7)
Proof. We construct the following bump function to construct the multiple hypothesis test used for proving the lower bound. Consider a simple C ∞ C^{\infty} C ∞ bump function supported on [ 0 , 1 ] d [0,1]^{d} [ 0 , 1 ] d
g ( x ) = ∏ i = 1 d ξ ( x i ) , x = ( x 1 , ⋯ , x d ) , g(x)=\prod_{i=1}^{d}\xi(x_{i}),\ x=(x_{1},\cdots,x_{d}), g ( x ) = i = 1 ∏ d ξ ( x i ) , x = ( x 1 , ⋯ , x d ) ,
where ξ : R → R \xi:\mathbb{R}\to\mathbb{R} ξ : R → R is a non-zero function in C ∞ ( R ) C^{\infty}(\mathbb{R}) C ∞ ( R ) with support contained in [ 0 , 1 ] [0,1] [ 0 , 1 ] and satisfies ξ ( x ) ≠ 0 , d d x ξ ( x ) ≠ 0 \xi(x)\neq0,\ \frac{d}{dx}\xi(x)\neq0 ξ ( x ) = 0 , d x d ξ ( x ) = 0 . Then ∇ g ( x ) ≠ 0 \nabla g(x)\neq0 ∇ g ( x ) = 0 and the support of function g g g is [ 0 , 1 ] d [0,1]^{d} [ 0 , 1 ] d .
Next, we take m = [ n 1 2 s − 4 + d ] m=\left[n^{\frac{1}{2s-4+d}}\right] m = [ n 2 s − 4 + d 1 ] and consider a regular gird x ( j ) x^{(j)} x ( j ) , j ∈ [ m ] d j\in[m]^{d} j ∈ [ m ] d . According to the Varshamov-Gilbert lemma, there exist 2 m d / 8 2^{m^{d}/8} 2 m d /8 ( 0 , 1 ) (0,1) ( 0 , 1 ) -sequences τ ( 1 ) , ⋯ , τ ( 2 m d / 8 ) ∈ { 0 , 1 } m d \tau^{(1)},\cdots,\tau^{(2^{m^{d}/8})}\in\{0,1\}^{m^{d}} τ ( 1 ) , ⋯ , τ ( 2 m d /8 ) ∈ { 0 , 1 } m d such that ∥ τ ( k ) − τ ( k ′ ) ∥ 2 ≥ m d 8 \lVert\tau^{(k)}-\tau^{(k')}\rVert^{2} \ge\frac{m^{d}}{8} ∥ τ ( k ) − τ ( k ′ ) ∥ 2 ≥ 8 m d for all 0 < k ≠ k ′ ≤ 2 m d / 8 0<k\ne k'\le2^{m^{d}/8} 0 < k = k ′ ≤ 2 m d /8 . Then we construct the multiple hypothesis as
u k ( x ) = ∑ j ∈ [ m ] d τ j ( k ) ω m s + d 2 g ( m ( x − x ( j ) ) ) , k = 1 , 2 , ⋯ , 2 m d / 8 , u_k(x)=\sum_{j\in[m]^d}\tau_j^{(k)}\frac{\omega}{m^{s+\frac{d}{2}}}g\bigl(m(x-x^{(j)})\bigr),\quad k=1,2,\cdots,2^{m^{d}/8}, u k ( x ) = j ∈ [ m ] d ∑ τ j ( k ) m s + 2 d ω g ( m ( x − x ( j ) ) ) , k = 1 , 2 , ⋯ , 2 m d /8 ,
where ω \omega ω is a constant to be determined later. It is easy to find out that u k ∈ C s u_k\in C^s u k ∈ C s .
Then we reduce solving the PDE to a multiple hypothesis testing problem, which considers all mappings from n n n sampled data to the constructed hypothesis Ψ : ( R d ) ⊗ n × R ⊗ n → V : = { u i ∣ i = 1 , 2 , ⋯ , 2 m d / 8 } \Psi:(\mathbb{R}^{d})^{\otimes n}\times\mathbb{R}^{\otimes n}\to\mathcal{V}:=\{u_i|i=1,2,\cdots,2^{m^{d}/8}\} Ψ : ( R d ) ⊗ n × R ⊗ n → V := { u i ∣ i = 1 , 2 , ⋯ , 2 m d /8 } . Then we apply the local Fano method and check that we can obtain a constant lower bound of P ( V ^ ≠ V ) \mathcal{P}(\widehat{V}\ne V) P ( V = V ) for any estimator V ^ \widehat{V} V . From the local Fano method, we know that
I ( V ; X ) ≤ 1 ∣ V ∣ 2 ∑ v ∑ v ≠ v ′ D K L ( P v ∥ P v ′ ) . I(V;X)\le\frac{1}{|\mathcal{V}|^{2}}\sum_{v}\sum_{v\ne v'}D_{KL}(P_v\Vert P_{v'}). I ( V ; X ) ≤ ∣ V ∣ 2 1 v ∑ v = v ′ ∑ D K L ( P v ∥ P v ′ ) .
where P k P_k P k denotes the joint distribution of the sampled data ( x , y ) (x,y) ( x , y ) . In specific, x x x follows a uniform distribution on [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d and y = f ( X ) + ϵ y=f(X)+\epsilon y = f ( X ) + ϵ , where ϵ \epsilon ϵ is independently sampled from a standard Gaussian distribution N ( 0 , 1 ) N(0,1) N ( 0 , 1 ) . Then we have
K L ( P k ∥ P k ′ ) = E log ( d P k d P k ′ ) = ∥ Δ u k + V u k ∥ L 2 2 ≤ C ω m 2 s − 4 . KL(P_k\Vert P_{k'})=\mathbb{E}\log\left(\frac{dP_k}{dP_{k'}}\right)=\lVert\Delta u_k+Vu_k\rVert_{L^2}^{2}\le\frac{C\omega}{m^{2s-4}}. K L ( P k ∥ P k ′ ) = E log ( d P k ′ d P k ) = ∥ Δ u k + V u k ∥ L 2 2 ≤ m 2 s − 4 C ω .
Using Fano inequality, if we select m ∝ [ n 1 2 s − 4 + d ] m\propto\left[n^{\frac{1}{2s-4+d}}\right] m ∝ [ n 2 s − 4 + d 1 ] then we have the following lower bound when ω \omega ω is taken to be sufficiently large:
P ( V ^ ≠ V ) ≥ 1 − I ( V ; X ) + log 2 log ( ∣ V ∣ ) ≥ 1 − 8 C ω m 2 s − 4 m d log 2 ≥ 1 / 2. \mathcal{P}(\widehat{V}\ne V)\ge1-\frac{I(V;X)+\log2}{\log(|\mathcal{V}|)}\ge1-\frac{8C\omega}{m^{2s-4}m^d\log2}\ge1/2. P ( V = V ) ≥ 1 − log ( ∣ V ∣ ) I ( V ; X ) + log 2 ≥ 1 − m 2 s − 4 m d log 2 8 C ω ≥ 1/2.
At the same time, we can estimate the separation of the hypotheses in two different norms:
∫ [ 0 , 1 ] d ∥ ∇ u k − ∇ u k ′ ∥ 2 d x = κ 2 m 2 s − 2 + d ∑ j ∈ [ m ] d ∥ τ j ( k ) − τ j ( k ′ ) ∥ 1 ∫ R d ∥ ∇ g ( x ) ∥ 2 d x ≳ 1 m 2 s − 2 . \int_{[0,1]^d}\lVert\nabla u_k-\nabla u_{k'}\rVert^2dx=\frac{\kappa^2}{m^{2s-2+d}}\sum_{j\in[m]^d}\lVert\tau_j^{(k)}-\tau_j^{(k')}\rVert_1\int_{\mathbb{R}^d}\lVert\nabla g(x)\rVert^2dx\gtrsim\frac{1}{m^{2s-2}}. ∫ [ 0 , 1 ] d ∥ ∇ u k − ∇ u k ′ ∥ 2 d x = m 2 s − 2 + d κ 2 j ∈ [ m ] d ∑ ∥ τ j ( k ) − τ j ( k ′ ) ∥ 1 ∫ R d ∥ ∇ g ( x ) ∥ 2 d x ≳ m 2 s − 2 1 .
∫ [ 0 , 1 ] d ∥ Δ u k − Δ u k ′ ∥ 2 d x = κ 2 m 2 s − 4 + d ∑ j ∈ [ m ] d ∥ τ j ( k ) − τ j ( k ′ ) ∥ 1 ∫ R d Δ g ( x ) 2 d x ≳ 1 m 2 s − 4 . \int_{[0,1]^d}\lVert\Delta u_k-\Delta u_{k'}\rVert^2dx=\frac{\kappa^2}{m^{2s-4+d}}\sum_{j\in[m]^d}\lVert\tau_j^{(k)}-\tau_j^{(k')}\rVert_1\int_{\mathbb{R}^d}\Delta g(x)^2dx\gtrsim\frac{1}{m^{2s-4}}. ∫ [ 0 , 1 ] d ∥ Δ u k − Δ u k ′ ∥ 2 d x = m 2 s − 4 + d κ 2 j ∈ [ m ] d ∑ ∥ τ j ( k ) − τ j ( k ′ ) ∥ 1 ∫ R d Δ g ( x ) 2 d x ≳ m 2 s − 4 1 .
Plugging in m ∝ [ n 1 2 s − 4 + d ] m\propto\left[n^{\frac{1}{2s-4+d}}\right] m ∝ [ n 2 s − 4 + d 1 ] , we know that with constant probability we have
inf ψ sup u ∗ ∈ H s ( Ω ) E ∥ ψ ( { X i , Y i } i = 1 , ⋯ , n ) − u ∗ ( f ) ∥ H 1 2 ≳ n − 2 s − 2 d + 2 s − 4 . (8) \inf_{\psi}\sup_{u^*\in H^s(\Omega)}\mathbb{E}\lVert\psi(\{X_i,Y_i\}_{i=1,\cdots,n})-u^*(f)\rVert_{H^1}^{2}\gtrsim n^{-\frac{2s-2}{d+2s-4}}. \tag*{(8)} ψ inf u ∗ ∈ H s ( Ω ) sup E ∥ ψ ({ X i , Y i } i = 1 , ⋯ , n ) − u ∗ ( f ) ∥ H 1 2 ≳ n − d + 2 s − 4 2 s − 2 . (8)
inf ψ sup u ∗ ∈ H s ( Ω ) E ∥ ψ ( { X i , Y i } i = 1 , ⋯ , n ) − u ∗ ( f ) ∥ H 2 2 ≳ n − 2 s − 4 d + 2 s − 4 . (9) \inf_{\psi}\sup_{u^*\in H^s(\Omega)}\mathbb{E}\lVert\psi(\{X_i,Y_i\}_{i=1,\cdots,n})-u^*(f)\rVert_{H^2}^{2}\gtrsim n^{-\frac{2s-4}{d+2s-4}}. \tag*{(9)} ψ inf u ∗ ∈ H s ( Ω ) sup E ∥ ψ ({ X i , Y i } i = 1 , ⋯ , n ) − u ∗ ( f ) ∥ H 2 2 ≳ n − d + 2 s − 4 2 s − 4 . (9)
Given that n − 2 ( β − k ) d + 2 β n^{-\frac{2(\beta-k)}{d+2\beta}} n − d + 2 β 2 ( β − k ) is the minimax rate of estimation of the k k k -th derivative of a β \beta β -smooth density in L 2 L_2 L 2 [43 , 57 , 53 ] , the lower bound obtained here is the rate of estimating the right hand side function f f f in terms of the H − 1 H^{-1} H − 1 norm. Given the H − 1 H^{-1} H − 1 norm error estimate on f f f , we can achieve estimate of u u u , which provides an alternative way to understand our upper bound. The lower bound is non-standard, for the 2 s − 2 2s-2 2 s − 2 in the numerator is different from the 2 s − 4 2s-4 2 s − 4 in the denominator.
Upper Bound
To theoretically understand the empirical success of Physics Informed Neural Networks and the Deep Ritz solver, in this section, we aim to prove that the excess risk Δ E n : = E ( u n ) − E ( u ∗ ) \Delta\mathcal{E}_n := \mathcal{E}(u_n)-\mathcal{E}(u^*) Δ E n := E ( u n ) − E ( u ∗ ) of a well-trained neural networks on the PINN/DRM loss function will follow a precise power-law scaling relations with the size of the training dataset. Similar to [78 , 47 , 14 , 34 , 35 ] , we decompose the excess risk into approximation error and generalization error. Different from the concurrent bound [14 , 34 ] , we provided a fast rate O ( 1 / n ) O(1/n) O ( 1/ n ) by utilizing the strong convexity of the objective function established in § 2.1 and achieved a faster and near optimal upper bound.
Proof Sketch
Error Decomposition. We first decompose the excess risk Δ E n : = E ( u n ) − E ( u ∗ ) \Delta\mathcal{E}_n := \mathcal{E}(u_n)-\mathcal{E}(u^*) Δ E n := E ( u n ) − E ( u ∗ ) of a well-trained neural network on the PINN/DRM loss function into approximation error and generalization error, similar to [78 , 47 , 14 , 34 , 35 ] . The regularity results used in the decomposition are proved in Appendix A.§ 2 . Explicitly, for any u F ∈ F ( Ω ) u_F \in\mathcal{F}(\Omega) u F ∈ F ( Ω ) , we can decompose the excess risk as
Δ E ( n ) ( u ^ ) = E ( u ^ ) − E ( u ∗ ) = [ E ( u ^ ) − E n ( u ^ ) ] + [ E n ( u ^ ) − E n ( u F ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u ^ ) − E n ( u ^ ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] , (10) \begin{aligned}
\Delta\mathcal{E}^{(n)}(\hat{u}) &= \mathcal{E}(\hat{u})-\mathcal{E}(u^*) \\
&= [\mathcal{E}(\hat{u})-\mathcal{E}_n(\hat{u})]+[\mathcal{E}_n(\hat{u})-\mathcal{E}_n(u_F)]+[\mathcal{E}_n(u_F)-\mathcal{E}(u_F)]+[\mathcal{E}(u_F)-\mathcal{E}(u^*)] \\
&\leq[\mathcal{E}(\hat{u})-\mathcal{E}_n(\hat{u})]+[\mathcal{E}_n(u_F)-\mathcal{E}(u_F)]+[\mathcal{E}(u_F)-\mathcal{E}(u^*)], \tag*{(10)}
\end{aligned} Δ E ( n ) ( u ^ ) = E ( u ^ ) − E ( u ∗ ) = [ E ( u ^ ) − E n ( u ^ )] + [ E n ( u ^ ) − E n ( u F )] + [ E n ( u F ) − E ( u F )] + [ E ( u F ) − E ( u ∗ )] ≤ [ E ( u ^ ) − E n ( u ^ )] + [ E n ( u F ) − E ( u F )] + [ E ( u F ) − E ( u ∗ )] , (10)
where the expectation is on uniformly sampled data, F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) is the space of parametrized estimators we used like truncated Fourier series or sparse neural networks, u ^ \hat{u} u ^ is the minimizer of the empirical loss E n \mathcal{E}_n E n in F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) and u ∗ u^* u ∗ is the minimizer of the population loss E \mathcal{E} E (i.e., ground truth solution). The inequality in the third line follows from the fact that u ^ \hat{u} u ^ is the minimizer of the empirical loss E n \mathcal{E}_n E n in the space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , which implies E n ( u ^ ) ≤ E n ( u F ) \mathcal{E}_n(\hat{u}) \leq\mathcal{E}_n(u_F) E n ( u ^ ) ≤ E n ( u F ) . We call the first term generalization error as it’s measuring the difference between E n \mathcal{E}_n E n and E \mathcal{E} E . We call the second term approximation error as it seeks for a parametrized estimator u F u_F u F that approximates the ground truth solution u ∗ u^* u ∗ well in F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) . The upper bounds on generalization and approximation error that we achieved in this paper are listed in Table 2 .
Table 2. Approximation and generalization results we achieved in this paper.
Let n n n denote the number of sampled datapoints. For the generalization error, different from the concurrent upper bound O ( 1 n ) O\left(\frac{1}{\sqrt{n}}\right) O ( n 1 ) [14 , 34 ] , we provide a faster and near optimal upper bound O ( 1 n ) O\left(\frac{1}{n}\right) O ( n 1 ) by utilizing the strong convexity of the objective function established in Appendix A.§ 2 . Via using the Peeling Lemma (Lemma A.7, for completeness, we also provide a proof), we show that the generalization error can be bounded by the fixed point of the local Rademacher complexity
ϕ ( r ) = R n ( { I ( u ) ∣ ∥ u − u ∗ ∥ A 2 ≤ r } ) \phi(r) = R_n\left(\{\mathcal{I}(u)\mid\|u-u^*\|_A^2 \le r\}\right) ϕ ( r ) = R n ( { I ( u ) ∣ ∥ u − u ∗ ∥ A 2 ≤ r } )
where R n R_n R n is the Rademacher complexity, I ( u ) = Δ u + V u \mathcal{I}(u)=\Delta u+Vu I ( u ) = Δ u + V u , ∥ ⋅ ∥ A = ∥ ⋅ ∥ H 2 \|\cdot\|_A=\|\cdot\|_{H^2} ∥ ⋅ ∥ A = ∥ ⋅ ∥ H 2 for PINN and I ( u ) = ∥ ∇ u ∥ 2 + V u \mathcal{I}(u)=\|\nabla u\|^2+Vu I ( u ) = ∥∇ u ∥ 2 + V u , ∥ ⋅ ∥ A = ∥ ⋅ ∥ H 1 \|\cdot\|_A=\|\cdot\|_{H^1} ∥ ⋅ ∥ A = ∥ ⋅ ∥ H 1 for DRM. Once we show that ϕ ( r ) \phi(r) ϕ ( r ) is of magnitude O ( r n ) O\left(\sqrt{\frac{r}{n}}\right) O ( n r ) , we can achieve the O ( 1 n ) O\left(\frac{1}{n}\right) O ( n 1 ) convergence rate via solving the fixed point equation ϕ ( r ) = O ( r n ) = r ⇒ r = O ( 1 n ) \phi(r)=O\left(\sqrt{\frac{r}{n}}\right)=r\Rightarrow r=O\left(\frac{1}{n}\right) ϕ ( r ) = O ( n r ) = r ⇒ r = O ( n 1 ) . Using the solution of the fixed point equation of the local Rademacher complexity to bound the generalization error is a standard result in empirical process [3 , 69 , 39 , 79 , 18 ] . The difference is that we used the H 1 / H 2 H^1/H^2 H 1 / H 2 norm to define the localized set, while the previous papers used the ℓ 2 \ell_2 ℓ 2 distance. The way to obtain the fast rate generalization bound is using the Peeling Lemma.
We present the error decomposition results as a meta theorem, which is shown in Theorem 4.14 for PINN (proof in Appendix A.§ 4 ), Theorem 4.11 for DRM and Theorem 5.1 for MDRM (proof in Appendix A.§ 5 ), respectively. To make the final rate depend on the data number only, we need bounds of the approximation error in § 4.2 and bounds of the local Rademacher complexity in Appendix A.§ 3 .
Approximation Error. The proof of the approximation results of truncated Fourier series is easy and intuitive. For completeness, we provide it in § 4.2.1 . The proof of the approximation results of neural networks follows from the fact that a B-spline approximation can be formulated as a ReLU3 neural network efficiently. Our proof basically follows [14 , 34 ] , while the only difference is the activation function. Our proof is also very similar to [80 , 70 ] , but the depth of our network is of constant magnitude instead of O ( 1 log ε ) O\left(\frac{1}{\log\varepsilon}\right) O ( l o g ε 1 ) magnitude, where ε \varepsilon ε denotes the desired approximation error. Such improvement of depth results from the fact that ReLU3 activations can approximate B-splines more easily than the ReLU activations, which is useful in our generalization analysis. Although the proof of the approximation results of neural networks in the Sobolev space is standard, we still list it in Appendix § 4.2.2 .
Generalization Error. As we discussed above, the generalization error can be bounded by the fix point of the local Rademacher complexity, i.e. the solution of ϕ ( r ) = r \phi(r)=r ϕ ( r ) = r . Once we have a O ( r n ) O\left(\sqrt{\frac{r}{n}}\right) O ( n r ) bound of ϕ ( r ) \phi(r) ϕ ( r ) , we can achieve the O ( 1 n ) O\left(\frac{1}{n}\right) O ( n 1 ) fast rate generalization bound we want. It remains to upper bound the the local Rademacher complexity ϕ ( r ) \phi(r) ϕ ( r ) .
For the upper bound on the local Rademacher complexity of truncated Fourier series estimators, our proof technique is similar to that of the kernel estimators, whose Rademacher complexity can be bounded by the trace of the Gram matrix (i.e. the effective number of basis). One interesting thing we showed is that the final upper bound of the Rademacher complexity localized by H 1 H^1 H 1 norm is ξ d − 2 r n \sqrt{\frac{\xi^{d-2}r}{n}} n ξ d − 2 r . The term ξ d − 2 \xi^{d-2} ξ d − 2 in the numerator is smaller than ξ d \xi^d ξ d , which is the exact number of Fourier basis. This improvement results from the H 1 H^1 H 1 norm localization. The detailed proof is given in Lemma 4.6 , Lemma 4.7 and Lemma 4.8 .
For the upper bound on the local Rademacher complexity bound for neural network, we follow [63 , 70 , 18 ] to use a Dudley integral theorem and a covering number argument. The covering number arguments are shown in Theorem A.18 , Theorem A.22 and Theorem A.25 . The final local Rademacher complexity bounds are given in Lemma 4.9 and Lemma 4.10 . The difference is that the complexity of gradient of ReLU3 activation function makes the covering number depend exponentially on the neural network’s depth. However, the improvement of neural network’s depth to constant magnitude mentioned above in the approximation results saves this problem. One drawback of our proof is that the H 1 H^1 H 1 norm localization wouldn’t improve the bound for Rademacher complexity and leads to sub-optimal upper bounds. We hypothesize that our bound is tight for sparse neural network and put seeking a right complexity measure of neural network for solving PDEs as a future work.
Bounding the Approximation Error
Approximation using Truncated Fourier Basis
Lemma 4.1. Given α > 0 \alpha> 0 α > 0 and a fixed integer ξ ∈ Z + \xi\in\mathbb{Z}^{+} ξ ∈ Z + . For any function f ∈ H α ( Ω ) f \in H^\alpha(\Omega) f ∈ H α ( Ω ) , we let f ξ = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z f_\xi= \sum_{\lVert z\rVert_\infty\leq\xi} f_z\phi_z f ξ = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z be the best approximation of f f f in the space F ξ ( Ω ) F_\xi(\Omega) F ξ ( Ω ) . Then for any 0 < β ≤ α 0 < \beta\leq\alpha 0 < β ≤ α , we have the following inequality:
∥ f − f ξ ∥ H β ( Ω ) 2 ≤ ξ − 2 ( α − β ) ∥ f ∥ H α 2 . \lVert f-f_\xi\rVert_{H^\beta(\Omega)}^2 \leq\xi^{-2(\alpha-\beta)}\lVert f\rVert_{H^\alpha}^2. ∥ f − f ξ ∥ H β ( Ω ) 2 ≤ ξ − 2 ( α − β ) ∥ f ∥ H α 2 .
Proof. For f ∈ H α ( Ω ) f \in H^\alpha(\Omega) f ∈ H α ( Ω ) , we know the Fourier coefficient satisfies
∑ ∥ z ∥ ∞ ≥ ξ ∣ f z ∣ 2 ∥ z ∥ 2 α ≲ ∥ f ∥ H α 2 . \sum_{\lVert z\rVert_\infty\geq\xi} |f_z|^2\lVert z\rVert^{2\alpha} \lesssim\lVert f\rVert_{H^\alpha}^2. ∥ z ∥ ∞ ≥ ξ ∑ ∣ f z ∣ 2 ∥ z ∥ 2 α ≲ ∥ f ∥ H α 2 .
We directly construct f ξ = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z f_\xi= \sum_{\lVert z\rVert_\infty\leq\xi} f_z\phi_z f ξ = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z to be the truncated Fourier series of the function f f f , then we have
∥ f − f ξ ∥ H β ( Ω ) 2 ≲ ∑ ∥ z ∥ ∞ ≥ ξ ∣ f z ∣ 2 ∥ z ∥ 2 β ≤ ξ − 2 ( α − β ) ∑ ∥ z ∥ ∞ ≥ ξ ∣ f z ∣ 2 ∥ z ∥ 2 α ≤ ξ − 2 ( α − β ) ∥ f ∥ H α 2 . \lVert f-f_\xi\rVert_{H^\beta(\Omega)}^2 \lesssim\sum_{\lVert z\rVert_\infty\geq\xi} |f_z|^2\lVert z\rVert^{2\beta} \leq\xi^{-2(\alpha-\beta)}\sum_{\lVert z\rVert_\infty\geq\xi} |f_z|^2\lVert z\rVert^{2\alpha} \leq\xi^{-2(\alpha-\beta)}\lVert f\rVert_{H^\alpha}^2. ∥ f − f ξ ∥ H β ( Ω ) 2 ≲ ∥ z ∥ ∞ ≥ ξ ∑ ∣ f z ∣ 2 ∥ z ∥ 2 β ≤ ξ − 2 ( α − β ) ∥ z ∥ ∞ ≥ ξ ∑ ∣ f z ∣ 2 ∥ z ∥ 2 α ≤ ξ − 2 ( α − β ) ∥ f ∥ H α 2 .
Approximation using Neural Network
In this section, we aim to provide approximation bound for deep neural network. Our proof of the approximation upper bound is based on the observation that the B-spline approximation[12 , 64 ] can be formulated as a ReLU3 neural network efficiently[70 , 25 , 14 , 34 ] . Although the proof of the approximation of the neural network to the Sobolev spaces is a standard approach, we still demonstrate the proof sketch here.
Definition 4.2. (Univariate and Multivariate B-splines ) Fix an arbitrary integer l ∈ Z + l \in\mathbb{Z}^{+} l ∈ Z + . Consider a corresponding uniform partition π l \pi_l π l of [ 0 , 1 ] [0,1] [ 0 , 1 ] :
π l : 0 = t 0 ( l ) < t 1 ( l ) < ⋯ < t l − 1 ( l ) < t l ( l ) = 1 , \pi_l : 0 = t_0^{(l)} < t_1^{(l)} < \cdots< t_{l-1}^{(l)} < t_l^{(l)} = 1, π l : 0 = t 0 ( l ) < t 1 ( l ) < ⋯ < t l − 1 ( l ) < t l ( l ) = 1 , where t i ( l ) = i l t_i^{(l)} = \frac{i}{l} t i ( l ) = l i (∀ 0 ≤ i ≤ l \forall\ 0 \le i \le l ∀ 0 ≤ i ≤ l ). Now for any k ∈ Z + k \in\mathbb{Z}^{+} k ∈ Z + , we can define an extended partition π l , k \pi_{l,k} π l , k as:
π l , k : t − k + 1 ( l ) = ⋯ = t − 1 ( l ) = 0 = t 0 ( l ) < t 1 ( l ) < ⋯ < t l − 1 ( l ) < t l ( l ) = 1 = t l + 1 ( l ) = ⋯ = t l + k − 1 ( l ) . \pi_{l,k} : t_{-k+1}^{(l)} = \cdots= t_{-1}^{(l)} = 0 = t_0^{(l)} < t_1^{(l)} < \cdots< t_{l-1}^{(l)} < t_l^{(l)} = 1 = t_{l+1}^{(l)} = \cdots= t_{l+k-1}^{(l)}. π l , k : t − k + 1 ( l ) = ⋯ = t − 1 ( l ) = 0 = t 0 ( l ) < t 1 ( l ) < ⋯ < t l − 1 ( l ) < t l ( l ) = 1 = t l + 1 ( l ) = ⋯ = t l + k − 1 ( l ) . Based on the extended partition π l , k \pi_{l,k} π l , k , the univariate B B B -splines of order k k k with respect to partition π l \pi_l π l are defined by:
N l , i ( k ) ( x ) : = ( − 1 ) k ( t i + k ( l ) − t i ( l ) ) [ t i ( l ) , … , t i + k ( l ) ] max { ( x − t ) , 0 } k − 1 , x ∈ [ 0 , 1 ] , i ∈ I l , k (11) N_{l,i}^{(k)}(x) := (-1)^k(t_{i+k}^{(l)}-t_i^{(l)})\left[t_i^{(l)},\ldots,t_{i+k}^{(l)}\right]\max\{(x-t),0\}^{k-1},\quad x\in[0,1],\ i\in I_{l,k} \tag*{(11)} N l , i ( k ) ( x ) := ( − 1 ) k ( t i + k ( l ) − t i ( l ) ) [ t i ( l ) , … , t i + k ( l ) ] max {( x − t ) , 0 } k − 1 , x ∈ [ 0 , 1 ] , i ∈ I l , k (11) where I l , k = { − k + 1 , − k + 2 , ⋯ , l − 1 } I_{l,k}=\{-k+1,-k+2,\cdots,l-1\} I l , k = { − k + 1 , − k + 2 , ⋯ , l − 1 } and [ t i ( l ) , … , t i + k ( l ) ] \left[t_i^{(l)},\ldots,t_{i+k}^{(l)}\right] [ t i ( l ) , … , t i + k ( l ) ] denotes the divided difference operator.
Equivalently, for any x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] , we can rewrite the univariate B B B -splines N l , i ( k ) ( x ) N_{l,i}^{(k)}(x) N l , i ( k ) ( x ) in an explicit form:
N l , i ( k ) ( x ) = { l k − 1 ( k − 1 ) ! ∑ j = 0 k ( − 1 ) j ( k j ) max { x − i + j l , 0 } k − 1 , ( 0 ≤ i ≤ l − k + 1 ) ∑ j = 0 k − 1 a i j max { x − j l , 0 } k − 1 + ∑ n = 1 k − 2 b i n x n + b i 0 , ( − k + 1 ≤ i ≤ 0 ) ∑ j = l − k + 1 l c i j max { x − j l , 0 } k − 1 , ( l − k + 1 ≤ i ≤ l − 1 ) (12) N_{l,i}^{(k)}(x)=
\begin{cases}
\frac{l^{k-1}}{(k-1)!}\sum_{j=0}^{k}(-1)^j\binom{k}{j}\max\left\{x-\frac{i+j}{l},0\right\}^{k-1}, & (0\le i\le l-k+1)\\
\sum_{j=0}^{k-1}a_{ij}\max\left\{x-\frac{j}{l},0\right\}^{k-1}+\sum_{n=1}^{k-2}b_{in}x^n+b_{i0}, & (-k+1\le i\le0)\\
\sum_{j=l-k+1}^{l}c_{ij}\max\left\{x-\frac{j}{l},0\right\}^{k-1}, & (l-k+1\le i\le l-1)
\end{cases} \tag*{(12)} N l , i ( k ) ( x ) = ⎩ ⎨ ⎧ ( k − 1 )! l k − 1 ∑ j = 0 k ( − 1 ) j ( j k ) max { x − l i + j , 0 } k − 1 , ∑ j = 0 k − 1 a ij max { x − l j , 0 } k − 1 + ∑ n = 1 k − 2 b in x n + b i 0 , ∑ j = l − k + 1 l c ij max { x − l j , 0 } k − 1 , ( 0 ≤ i ≤ l − k + 1 ) ( − k + 1 ≤ i ≤ 0 ) ( l − k + 1 ≤ i ≤ l − 1 ) (12) where { a i j ∣ − k + 1 ≤ i ≤ 0 , 0 ≤ j ≤ k − 1 } \{a_{ij}\mid-k+1\le i\le0,\ 0\le j\le k-1\} { a ij ∣ − k + 1 ≤ i ≤ 0 , 0 ≤ j ≤ k − 1 } , { b i n ∣ − k + 1 ≤ i ≤ 0 , 1 ≤ n ≤ k − 2 } \{b_{in}\mid-k+1\le i\le0,\ 1\le n\le k-2\} { b in ∣ − k + 1 ≤ i ≤ 0 , 1 ≤ n ≤ k − 2 } and { c i j ∣ l − k + 1 ≤ i ≤ l − 1 , l − k + 1 ≤ j ≤ l − 1 } \{c_{ij}\mid l-k+1\le i\le l-1,\ l-k+1\le j\le l-1\} { c ij ∣ l − k + 1 ≤ i ≤ l − 1 , l − k + 1 ≤ j ≤ l − 1 } are some fixed constants.
For any index vector i = ( i 1 , i 2 , ⋯ , i d ) ∈ I l , k d i=(i_1,i_2,\cdots,i_d)\in I_{l,k}^{d} i = ( i 1 , i 2 , ⋯ , i d ) ∈ I l , k d , we can define a corresponding multivariate B B B -spline as a product of univariate B B B -splines:
N l , i ( k ) ( x ) : = ∏ j = 1 d N l , i j ( k ) ( x j ) . (13) N_{l,i}^{(k)}(x):=\prod_{j=1}^{d}N_{l,i_j}^{(k)}(x_j). \tag*{(13)} N l , i ( k ) ( x ) := j = 1 ∏ d N l , i j ( k ) ( x j ) . (13)
Definition 4.3. (Interpolation Operator [64 ] ) Take some domain Ω ⊂ [ 0 , 1 ] d \Omega\subset[0,1]^d Ω ⊂ [ 0 , 1 ] d and two arbitrary integers k , l ∈ Z + k,l\in\mathbb{Z}^{+} k , l ∈ Z + . Consider the extended partition π l , k \pi_{l,k} π l , k and the corresponding set of multivariate B B B -splines { N l , i ( k ) ( x ) } i ∈ I k , l d \{N_{l,i}^{(k)}(x)\}_{i\in I_{k,l}^{d}} { N l , i ( k ) ( x ) } i ∈ I k , l d defined in Definition 4.2 . For any i ∈ I k , l d i\in I_{k,l}^{d} i ∈ I k , l d , we define the domain Ω i : = { x ∈ Ω : x j ∈ [ t i j , t i j + k ] , 1 ≤ j ≤ d } \Omega_i:=\{x\in\Omega:x_j\in[t_{i_j},t_{i_j+k}],\ 1\le j\le d\} Ω i := { x ∈ Ω : x j ∈ [ t i j , t i j + k ] , 1 ≤ j ≤ d } . There exists a set of linear functionals { λ i } i ∈ I k , l d \{\lambda_i\}_{i\in I_{k,l}^{d}} { λ i } i ∈ I k , l d , where λ i : L 1 ( Ω ) → R \lambda_i:L^1(\Omega)\to\mathbb{R} λ i : L 1 ( Ω ) → R (∀ i ∈ I k , l d \forall\ i\in I_{k,l}^{d} ∀ i ∈ I k , l d ), such that for any i ∈ I k , l d i\in I_{k,l}^{d} i ∈ I k , l d and p ∈ [ 1 , ∞ ] p\in[1,\infty] p ∈ [ 1 , ∞ ] , we have:
λ i ( N l , j ( k ) ) = δ i , j and ∣ λ i ( f ) ∣ ≤ 9 d ( k − 1 ) ( 2 k + 1 ) d ( k l ) − d p ∥ f ∥ L p ( Ω i ) , ∀ f ∈ L p ( Ω ) . (14) \lambda_i(N_{l,j}^{(k)})=\delta_{i,j}\quad\text{and}\quad|\lambda_i(f)|\le9^{d(k-1)}(2k+1)^d\left(\frac{k}{l}\right)^{-\frac{d}{p}}\|f\|_{L^p(\Omega_i)},\quad\forall\ f\in L^p(\Omega). \tag*{(14)} λ i ( N l , j ( k ) ) = δ i , j and ∣ λ i ( f ) ∣ ≤ 9 d ( k − 1 ) ( 2 k + 1 ) d ( l k ) − p d ∥ f ∥ L p ( Ω i ) , ∀ f ∈ L p ( Ω ) . (14) The corresponding interpolation operator Q k , l Q_{k,l} Q k , l is defined as:
Q k , l f : = ∑ i ∈ I k , l d λ i ( f ) N l , i ( k ) , ∀ f ∈ L 1 ( Ω ) . Q_{k,l}f:=\sum_{i\in I_{k,l}^{d}}\lambda_i(f)N_{l,i}^{(k)},\quad\forall\ f\in L^1(\Omega). Q k , l f := i ∈ I k , l d ∑ λ i ( f ) N l , i ( k ) , ∀ f ∈ L 1 ( Ω ) .
Theorem 4.4. [64 ] Fix f ∈ W s ( Ω ) f \in W^s(\Omega) f ∈ W s ( Ω ) with Ω ⊂ [ 0 , 1 ] d \Omega\subset[0,1]^d Ω ⊂ [ 0 , 1 ] d , s ∈ Z + s \in\mathbb{Z}^{+} s ∈ Z + and p ∈ [ 1 , ∞ ) p \in[1,\infty) p ∈ [ 1 , ∞ ) . Then for any k , l , r ∈ Z + k,l,r \in\mathbb{Z}^{+} k , l , r ∈ Z + with k ≥ s k \ge s k ≥ s and 0 ≤ r ≤ s 0 \le r \le s 0 ≤ r ≤ s , we have that there exists some constant C = C ( k , s , r , p , d ) C=C(k,s,r,p,d) C = C ( k , s , r , p , d ) , such that:
∥ f − Q k , l f ∥ H r ( Ω ) ≤ C ( 1 l ) s − r ∥ f ∥ H s ( Ω ) . \lVert f-Q_{k,l}f\rVert_{H^r(\Omega)} \le C\left(\frac{1}{l}\right)^{s-r}\lVert f\rVert_{H^s(\Omega)}. ∥ f − Q k , l f ∥ H r ( Ω ) ≤ C ( l 1 ) s − r ∥ f ∥ H s ( Ω ) .
Theorem 4.5. (Approximation result of Deep Neural Network) Fix some dimension d ∈ Z + d \in\mathbb{Z}^{+} d ∈ Z + , some domain Ω ⊂ [ 0 , 1 ] d \Omega\subset[0,1]^d Ω ⊂ [ 0 , 1 ] d . We pick some l = N 1 d ≥ 2 l=N^{\frac{1}{d}}\ge2 l = N d 1 ≥ 2 , for any s , r ∈ Z + s,r \in\mathbb{Z}^{+} s , r ∈ Z + with 0 ≤ r ≤ s 0\le r\le s 0 ≤ r ≤ s and any function u ∗ ∈ H s ( Ω ) u^* \in H^s(\Omega) u ∗ ∈ H s ( Ω ) , there exists some sparse Deep Neural Network u D N N ∈ Φ ( L , W , S , B ) u_{\mathrm{DNN}}\in\Phi(L,W,S,B) u DNN ∈ Φ ( L , W , S , B ) with L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( N ) W=O(N) W = O ( N ) , S = O ( N ) S=O(N) S = O ( N ) , B = O ( N ) B=O(N) B = O ( N ) , such that:
∥ u D N N − u ∗ ∥ H r ( Ω ) ≲ N − s − r d ∥ u ∗ ∥ H s ( Ω ) . (15) \lVert u_{\mathrm{DNN}}-u^*\rVert_{H^r(\Omega)}\lesssim N^{-\frac{s-r}{d}}\lVert u^*\rVert_{H^s(\Omega)}. \tag*{(15)} ∥ u DNN − u ∗ ∥ H r ( Ω ) ≲ N − d s − r ∥ u ∗ ∥ H s ( Ω ) . (15)
Proof. We firstly show that the given function u ∗ u^* u ∗ can be approximated well by some linear combination of multivariate splines, which is denoted by u s p u_{\mathrm{sp}} u sp . Note that N N N is assumed to be sufficiently large. Hence, we may pick l = ⌈ N 1 d ⌉ = Θ ( N 1 d ) ∈ Z + l=\lceil N^{\frac{1}{d}}\rceil=\Theta(N^{\frac{1}{d}})\in\mathbb{Z}^{+} l = ⌈ N d 1 ⌉ = Θ ( N d 1 ) ∈ Z + to be the partition size of the B-splines. Moreover, by picking k = 4 k=4 k = 4 and p = 2 p=2 p = 2 in Theorem 4.4 , we have that the linear combination u s p : = Q 4 , l u ∗ = ∑ i ∈ I 4 , l d λ i ( u ∗ ) N l , i ( 4 ) u_{\mathrm{sp}}:=Q_{4,l}u^*=\sum_{i\in I_{4,l}^d}\lambda_i(u^*)N_{l,i}^{(4)} u sp := Q 4 , l u ∗ = ∑ i ∈ I 4 , l d λ i ( u ∗ ) N l , i ( 4 ) satisfies:
∥ u ∗ − u s p ∥ H r ( Ω ) = ∥ u ∗ − Q 4 , l u ∗ ∥ H r ( Ω ) ≤ C ( 1 l ) s − r ∥ u ∗ ∥ H s ( Ω ) = C N − s − r d ∥ u ∗ ∥ H s ( Ω ) . \lVert u^*-u_{\mathrm{sp}}\rVert_{H^r(\Omega)}=\lVert u^*-Q_{4,l}u^*\rVert_{H^r(\Omega)}\le C\left(\frac{1}{l}\right)^{s-r}\lVert u^*\rVert_{H^s(\Omega)}=CN^{-\frac{s-r}{d}}\lVert u^*\rVert_{H^s(\Omega)}. ∥ u ∗ − u sp ∥ H r ( Ω ) = ∥ u ∗ − Q 4 , l u ∗ ∥ H r ( Ω ) ≤ C ( l 1 ) s − r ∥ u ∗ ∥ H s ( Ω ) = C N − d s − r ∥ u ∗ ∥ H s ( Ω ) .
We will then show that the linear combination u s p = ∑ i ∈ I 4 , l d λ i ( f ) N l , i ( 4 ) u_{\mathrm{sp}}=\sum_{i\in I_{4,l}^d}\lambda_i(f)N_{l,i}^{(4)} u sp = ∑ i ∈ I 4 , l d λ i ( f ) N l , i ( 4 ) can be implemented by some Deep Neural Network u D N N ∈ Φ ( L , W , S , B ) u_{\mathrm{DNN}}\in\Phi(L,W,S,B) u DNN ∈ Φ ( L , W , S , B ) with L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( N ) W=O(N) W = O ( N ) , S = O ( N ) S=O(N) S = O ( N ) and B = O ( log N ) B=O(\log N) B = O ( log N ) . Firstly, note that for x ≥ 0 x\ge0 x ≥ 0 , both x x x and x 2 x^2 x 2 can be expressed in terms of the ReLU3 activation function η 3 \eta_3 η 3 with no error:
x = − 1 12 [ η 3 ( x + 3 ) − 5 η 3 ( x + 2 ) + 7 η 3 ( x + 1 ) − 3 η 3 ( x ) + 6 ] x=-\frac{1}{12}\left[\eta_3(x+3)-5\eta_3(x+2)+7\eta_3(x+1)-3\eta_3(x)+6\right] x = − 12 1 [ η 3 ( x + 3 ) − 5 η 3 ( x + 2 ) + 7 η 3 ( x + 1 ) − 3 η 3 ( x ) + 6 ]
x 2 = − 1 6 [ η 3 ( x + 2 ) − 4 η 3 ( x + 1 ) + 3 η 3 ( x ) − 4 ] x^2=-\frac{1}{6}\left[\eta_3(x+2)-4\eta_3(x+1)+3\eta_3(x)-4\right] x 2 = − 6 1 [ η 3 ( x + 2 ) − 4 η 3 ( x + 1 ) + 3 η 3 ( x ) − 4 ]
Applying the explicit formula listed in equation 4.3 implies that for any − 3 ≤ i ≤ l − 1 -3\le i\le l-1 − 3 ≤ i ≤ l − 1 , the univariate B-spline function N l , i ( 4 ) ( x ) N_{l,i}^{(4)}(x) N l , i ( 4 ) ( x ) (x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] ) can be implemented by some ReLU3 Deep Neural Network v D N N v_{\mathrm{DNN}} v DNN with both scalar input and scalar output. We have that for v D N N v_{\mathrm{DNN}} v DNN , the depth L v L_v L v is 2 and the maximum width W v W_v W v is upper bounded by 11. Secondly, for any x , y ≥ 0 x,y\ge0 x , y ≥ 0 , we have that the product operation x ⋅ y x\cdot y x ⋅ y can be expressed in terms of the ReLU3 activation function η 3 \eta_3 η 3 with no error:
x ⋅ y = 1 2 [ ( x + y ) 2 − x 2 − y 2 ] = − 1 12 [ η 3 ( x + y + 2 ) − 4 η 3 ( x + y + 1 ) + 3 η 3 ( x + y ) − η 3 ( x + 2 ) + 4 η 3 ( x + 1 ) − 3 η 3 ( x ) − η 3 ( y + 2 ) + 4 η 3 ( y + 1 ) − 3 η 3 ( y ) + 4 ] \begin{aligned}
x\cdot y=\frac{1}{2}\left[(x+y)^2-x^2-y^2\right] \\
&=-\frac{1}{12}\left[\eta_3(x+y+2)-4\eta_3(x+y+1)+3\eta_3(x+y)-\eta_3(x+2)+4\eta_3(x+1)-3\eta_3(x)-\eta_3(y+2)+4\eta_3(y+1)-3\eta_3(y)+4\right]
\end{aligned} x ⋅ y = 2 1 [ ( x + y ) 2 − x 2 − y 2 ] = − 12 1 [ η 3 ( x + y + 2 ) − 4 η 3 ( x + y + 1 ) + 3 η 3 ( x + y ) − η 3 ( x + 2 ) + 4 η 3 ( x + 1 ) − 3 η 3 ( x ) − η 3 ( y + 2 ) + 4 η 3 ( y + 1 ) − 3 η 3 ( y ) + 4 ]
In [64 ] , it has been proved that the B-splines are always non-negative, i.e. N l , i ( 4 ) ( x ) ≥ 0 N_{l,i}^{(4)}(x) \ge0 N l , i ( 4 ) ( x ) ≥ 0 , ∀ x ∈ [ 0 , 1 ] \forall\ x \in[0,1] ∀ x ∈ [ 0 , 1 ] . Therefore, by multiplying the non-negative univariate B-splines, we can implement any multivariate B-spline N l , i ( 4 ) = ∏ j = 1 d N l , i j ( 4 ) ( x j ) N_{l,\boldsymbol{i}}^{(4)} = \prod_{j=1}^{d}N_{l,i_j}^{(4)}(x_j) N l , i ( 4 ) = ∏ j = 1 d N l , i j ( 4 ) ( x j ) with some ReLU3 Deep Neural Network p D N N p_{\mathrm{DNN}} p DNN . We have that for p D N N p_{\mathrm{DNN}} p DNN , the depth L p = ⌈ log 2 d ⌉ + 2 L_p = \lceil\log_2 d\rceil+ 2 L p = ⌈ log 2 d ⌉ + 2 and the maximum width W p = max { 11 d , 9 2 d } W_p = \max\{11d,\frac{9}{2}d\} W p = max { 11 d , 2 9 d } . Hence, we can further claim that u ∗ = ∑ i ∈ I k , l d λ i ( u ∗ ) N l , i ( 4 ) u^* = \sum_{\boldsymbol{i}\in I_{k,l}^{d}}\lambda_i(u^*)N_{l,\boldsymbol{i}}^{(4)} u ∗ = ∑ i ∈ I k , l d λ i ( u ∗ ) N l , i ( 4 ) , which is a linear combination of the multivariate B-splines N l , i ( 4 ) N_{l,\boldsymbol{i}}^{(4)} N l , i ( 4 ) , can be implemented by some ReLU3 Deep Neural Network u D N N u_{\mathrm{DNN}} u DNN . It remains to check that u D N N ∈ Φ ( L , W , S , B ) u_{\mathrm{DNN}} \in\Phi(L,W,S,B) u DNN ∈ Φ ( L , W , S , B ) with L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( N ) W=O(N) W = O ( N ) , S = O ( N ) S=O(N) S = O ( N ) and B = O ( N ) B=O(N) B = O ( N ) . Note that we can ensure that the hidden layers of u D N N u_{\mathrm{DNN}} u DNN are of the same dimension W W W by adding inactive neurons.
For the depth L L L of u D N N u_{\mathrm{DNN}} u DNN , we have that L L L is equal to L p + 1 L_p+1 L p + 1 , where L p L_p L p denotes the depth of the ReLU3 Deep Neural Network p D N N p_{\mathrm{DNN}} p DNN . Thus, we have L = L p + 1 = ⌈ log 2 d ⌉ + 3 L=L_p+1=\lceil\log_2 d\rceil+3 L = L p + 1 = ⌈ log 2 d ⌉ + 3 , which implies that L = O ( 1 ) L=O(1) L = O ( 1 ) .
For the width W W W of u D N N u_{\mathrm{DNN}} u DNN , we have that W ≤ ∣ I k , l d ∣ W p W\leq|I_{k,l}^{d}|W_p W ≤ ∣ I k , l d ∣ W p , where W p W_p W p denotes the width of the ReLU3 Deep Neural Network p D N N p_{\mathrm{DNN}} p DNN . This implies:
W ≤ ∣ I k , l d ∣ × 11 d = 11 d ( l + k ) d = 11 d ( l + 4 ) d = O ( l d ) ⇒ W = O ( N ) W \leq|I_{k,l}^{d}| \times11d = 11d(l+k)^d = 11d(l+4)^d = O(l^d) \Rightarrow W=O(N) W ≤ ∣ I k , l d ∣ × 11 d = 11 d ( l + k ) d = 11 d ( l + 4 ) d = O ( l d ) ⇒ W = O ( N )
For the sparsity constraint S S S of u D N N u_{\mathrm{DNN}} u DNN , starting from the third layer, the number of activated neurons is half of the number of activated neurons at previous layer. This yields the following upper bound on S S S :
S ≤ 2 ( W + W + ∑ j = 0 L − 2 W 2 j ) ≤ 8 W ⇒ S = O ( W ) = O ( N ) S \leq2\left(W+W+\sum_{j=0}^{L-2}\frac{W}{2^j}\right)\leq8W \Rightarrow S=O(W)=O(N) S ≤ 2 ( W + W + j = 0 ∑ L − 2 2 j W ) ≤ 8 W ⇒ S = O ( W ) = O ( N )
For the norm constraint B B B of u D N N u_{\mathrm{DNN}} u DNN , we have the following upper bound on B B B from equation 4.3 and equation 4.5:
B = O ( max { l k − 1 , sup i ∈ I k , l d λ i ( u ∗ ) } ) = O ( max { l 3 , l d } ) = O ( N ) B=O\left(\max\left\{l^{k-1},\sup_{i\in I_{k,l}^{d}}\lambda_i(u^*)\right\}\right)=O\left(\max\{l^3,l^d\}\right)=O(N) B = O ( max { l k − 1 , i ∈ I k , l d sup λ i ( u ∗ ) } ) = O ( max { l 3 , l d } ) = O ( N )
Now we have shown that parameters L , W , S , B L,W,S,B L , W , S , B of the Deep Neural Network u D N N u_{\mathrm{DNN}} u DNN are of the desired magnitude, which completes our proof.
Bounding the Local Rademacher Complexity
Local Rademacher Complexity of Truncated Fourier Basis
In this subsection, we aim to bound the local Rademacher complexity of the Truncated Fourier Basis estimator. The proof is standard and we put the proof in the appendix.
Lemma 4.6. (Local Rademacher Complexity of Localized Truncated Fourier Series ) For a fixed ξ ∈ Z + \xi\in\mathbb{Z}^{+} ξ ∈ Z + , we consider a localized class of functions F ρ , ξ ( Ω ) = { f ∈ F ξ ( Ω ) ∣ ∥ f ∥ H 1 ( Ω ) 2 ≤ ρ } F_{\rho,\xi}(\Omega)=\{f\in F_{\xi}(\Omega)\mid\|f\|_{H^1(\Omega)}^2\leq\rho\} F ρ , ξ ( Ω ) = { f ∈ F ξ ( Ω ) ∣ ∥ f ∥ H 1 ( Ω ) 2 ≤ ρ } , where ρ > 0 \rho>0 ρ > 0 is fixed. Then we have the following upper bound on the local Rademacher complexity:
R n ( F ρ , ξ ( Ω ) ) = E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i f ( X i ) | X 1 , ⋯ , X n ] ] ≲ ρ n ξ d − 2 2 . (16) R_n(F_{\rho,\xi}(\Omega))=\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in F_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(X_i)\,\middle|\,X_1,\cdots,X_n\right]\right]\lesssim\sqrt{\frac{\rho}{n}\xi^{\frac{d-2}{2}}}. \tag*{(16)} R n ( F ρ , ξ ( Ω )) = E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i f ( X i ) X 1 , ⋯ , X n ] ] ≲ n ρ ξ 2 d − 2 . (16)
Lemma 4.7. (Local Rademacher Complexity of Localized Truncated Fourier Series’ Gradient ) For a fixed ξ ∈ Z + \xi\in\mathbb{Z}^{+} ξ ∈ Z + , we consider a localized class of functions G ρ , ξ ( Ω ) = { ∥ ∇ f ∥ ∣ f ∈ F ρ , ξ ( Ω ) } \mathcal{G}_{\rho,\xi}(\Omega)=\{\|\nabla f\| \mid f\in F_{\rho,\xi}(\Omega)\} G ρ , ξ ( Ω ) = { ∥∇ f ∥ ∣ f ∈ F ρ , ξ ( Ω )} , where ρ > 0 \rho>0 ρ > 0 is fixed. Then for any sample { X i } i = 1 n ⊂ Ω \{X_i\}_{i=1}^{n}\subset\Omega { X i } i = 1 n ⊂ Ω , we have the following upper bound on the local Rademacher complexity:
R n ( G ρ , ξ ( Ω ) ) = E X 1 , … , X n [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ∥ ∇ f ( X i ) ∥ ∣ X 1 , … , X n ] ] ≲ ρ n ξ d 2 . (17) R_n(\mathcal{G}_{\rho,\xi}(\Omega))=\mathbb{E}_{X_1,\ldots,X_n}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in F_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla f(X_i)\|\mathrel{\bigg|}X_1,\ldots,X_n\right]\right]\lesssim\sqrt{\frac{\rho}{n}}\xi^{\frac{d}{2}}. \tag*{(17)} R n ( G ρ , ξ ( Ω )) = E X 1 , … , X n [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ∥∇ f ( X i ) ∥ X 1 , … , X n ] ] ≲ n ρ ξ 2 d . (17)
Lemma 4.8. (Local Rademacher Complexity of Localized Truncated Fourier Series’ Laplacian ) For a fixed ξ ∈ Z + \xi\in\mathbb{Z}^{+} ξ ∈ Z + , we consider a localized class of functions J ρ , ξ ( Ω ) : = { f ∈ F ξ ( Ω ) ∣ ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ } \mathcal{J}_{\rho,\xi}(\Omega):=\left\{f\in F_{\xi}(\Omega)\mid\|f\|_{H^{2}(\Omega)}^{2}\leq\rho\right\} J ρ , ξ ( Ω ) := { f ∈ F ξ ( Ω ) ∣ ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ } , where ρ > 0 \rho>0 ρ > 0 is fixed. Correspondingly, we define a localized class of Laplacians K ρ , ξ ( Ω ) : = { Δ f ∣ f ∈ J ρ , ξ ( Ω ) } \mathcal{K}_{\rho,\xi}(\Omega):=\{\Delta f\mid f\in\mathcal{J}_{\rho,\xi}(\Omega)\} K ρ , ξ ( Ω ) := { Δ f ∣ f ∈ J ρ , ξ ( Ω )} . Then for any sample { X i } i = 1 n ⊂ Ω \{X_i\}_{i=1}^{n}\subset\Omega { X i } i = 1 n ⊂ Ω , we have the following upper bound on the local Rademacher complexity:
R n ( K ρ , ξ ( Ω ) ) = E X 1 , … , X n [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i Δ f ( X i ) ∣ X 1 , … , X n ] ] ≲ ρ n ξ d 2 . (18) R_n(\mathcal{K}_{\rho,\xi}(\Omega))=\mathbb{E}_{X_1,\ldots,X_n}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in F_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\Delta f(X_i)\mathrel{\bigg|}X_1,\ldots,X_n\right]\right]\lesssim\sqrt{\frac{\rho}{n}}\xi^{\frac{d}{2}}. \tag*{(18)} R n ( K ρ , ξ ( Ω )) = E X 1 , … , X n [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i Δ f ( X i ) X 1 , … , X n ] ] ≲ n ρ ξ 2 d . (18)
Local Rademacher Complexity of Deep Neural Networks
In this subsection, we aim to bound the local Rademacher complexity of the Neural Network estimator. Informally, we showed that the local Rademacher complexity is at the scale of N n \sqrt{\frac{N}{n}} n N , where N N N is the number of neuron of a neural network. For simplicity, we put the proof in the appendix.
Lemma 4.9 (Local Rademacher Complexity Bound for Deep Ritz Method). Consider a Deep Neural Network space F ( Ω ) = Φ ( L , W , S , B ) F(\Omega)=\Phi(L,W,S,B) F ( Ω ) = Φ ( L , W , S , B ) with L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( N ) W=O(N) W = O ( N ) , S = O ( N ) S=O(N) S = O ( N ) and B = O ( N ) B=O(N) B = O ( N ) , where N ∈ Z + N\in\mathbb{Z}^{+} N ∈ Z + is fixed to be sufficiently large. Moreover, assume that the gradients and function value of F ( Ω ) F(\Omega) F ( Ω ) , V V V and f f f are uniformly bounded
max { sup u ∈ F ( Ω ) ∥ u ∥ L ∞ ( Ω ) , sup u ∈ F ( Ω ) ∥ ∇ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ ∇ u ∗ ∥ L ∞ ( Ω ) , V max , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (19) \max\left\{\sup_{u\in F(\Omega)}\|u\|_{L^{\infty}(\Omega)},\sup_{u\in F(\Omega)}\|\nabla u\|_{L^{\infty}(\Omega)},\|u^{*}\|_{L^{\infty}(\Omega)},\|\nabla u^{*}\|_{L^{\infty}(\Omega)},V_{\max},\|f\|_{L^{\infty}(\Omega)}\right\}\leq C. \tag*{(19)} max { u ∈ F ( Ω ) sup ∥ u ∥ L ∞ ( Ω ) , u ∈ F ( Ω ) sup ∥∇ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥∇ u ∗ ∥ L ∞ ( Ω ) , V m a x , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (19) For any ρ > 0 \rho>0 ρ > 0 , we consider a localized set L ρ L_{\rho} L ρ defined by:
L ρ ( Ω ) : = { u : u ∈ F ( Ω ) , ∥ u − u ∗ ∥ H 1 2 ≤ ρ } . L_{\rho}(\Omega):=\{u:u\in F(\Omega),\|u-u^{*}\|_{H^{1}}^{2}\leq\rho\}. L ρ ( Ω ) := { u : u ∈ F ( Ω ) , ∥ u − u ∗ ∥ H 1 2 ≤ ρ } . Then for any ρ ≳ n − 2 \rho\gtrsim n^{-2} ρ ≳ n − 2 , the Rademacher complexity of a localized function space S ρ ( Ω ) : = { h : = ∣ Ω ∣ ⋅ [ 1 2 ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) + 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] ∣ u ∈ L ρ ( Ω ) } S_{\rho}(\Omega):=\left\{h:=|\Omega|\cdot\left[\frac{1}{2}\left(\|\nabla u\|^{2}-\|\nabla u^{*}\|^{2}\right)+\frac{1}{2}V\left(|u|^{2}-|u^{*}|^{2}\right)-f(u-u^{*})\right]\mid u\in L_{\rho}(\Omega)\right\} S ρ ( Ω ) := { h := ∣Ω∣ ⋅ [ 2 1 ( ∥∇ u ∥ 2 − ∥∇ u ∗ ∥ 2 ) + 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] ∣ u ∈ L ρ ( Ω ) } can be upper bounded by a sub-root function
ϕ ( ρ ) : = O ( S 3 L ρ n log ( B W n ) ) . \phi(\rho):=O\left(\sqrt{\frac{S^{3}L\rho}{n}\log(BWn)}\right). ϕ ( ρ ) := O ( n S 3 L ρ log ( B W n ) ) . i.e. we have
ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) a n d R n ( S ρ ( Ω ) ) ≤ ϕ ( ρ ) . (20) \phi(4\rho)\leq2\phi(\rho)\ \mathit{and}\ R_n(S_{\rho}(\Omega))\leq\phi(\rho). \tag*{(20)} ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( S ρ ( Ω )) ≤ ϕ ( ρ ) . (20) holds for all ρ ≳ n − 2 \rho\gtrsim n^{-2} ρ ≳ n − 2 .
Lemma 4.10 (Local Rademacher Complexity Bound for Physics Informed Neural Network). Consider a Deep Neural Network space F ( Ω ) = Φ ( L , W , S , B ) \mathcal{F}(\Omega)=\Phi(L,W,S,B) F ( Ω ) = Φ ( L , W , S , B ) with L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( N ) W=O(N) W = O ( N ) , S = O ( N ) S=O(N) S = O ( N ) and B = O ( N ) B=O(N) B = O ( N ) , where N ∈ Z + N\in\mathbb{Z}^{+} N ∈ Z + is fixed to be sufficiently large. Moreover, assume that the gradients and function value of F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , V V V and f f f are uniformly bounded
max { sup u ∈ F ( Ω ) ∥ u ∥ L ∞ ( Ω ) , sup u ∈ F ( Ω ) ∥ Δ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ Δ u ∗ ∥ L ∞ ( Ω ) , V max , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (21) \max\left\{\sup_{u\in\mathcal{F}(\Omega)}\|u\|_{L^\infty(\Omega)},\sup_{u\in\mathcal{F}(\Omega)}\|\Delta u\|_{L^\infty(\Omega)},\|u^*\|_{L^\infty(\Omega)},\|\Delta u^*\|_{L^\infty(\Omega)},V_{\max},\|f\|_{L^\infty(\Omega)}\right\}\le C. \tag*{(21)} max { u ∈ F ( Ω ) sup ∥ u ∥ L ∞ ( Ω ) , u ∈ F ( Ω ) sup ∥Δ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥Δ u ∗ ∥ L ∞ ( Ω ) , V m a x , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (21) For any ρ > 0 \rho>0 ρ > 0 , we consider a localized set M ρ \mathcal{M}_\rho M ρ defined by:
M ρ ( Ω ) : = { u : u ∈ F ( Ω ) , ∥ u − u ∗ ∥ H 2 ≤ ρ } . \mathcal{M}_\rho(\Omega):=\{u:u\in\mathcal{F}(\Omega),\|u-u^*\|_{H^2}\le\rho\}. M ρ ( Ω ) := { u : u ∈ F ( Ω ) , ∥ u − u ∗ ∥ H 2 ≤ ρ } . Then for any ρ ≳ n − 2 \rho\gtrsim n^{-2} ρ ≳ n − 2 , the Rademacher complexity of a localized function space T ρ ( Ω ) : = { h : = ∣ Ω ∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] ∣ u ∈ M ρ ( Ω ) } \mathcal{T}_\rho(\Omega):=\left\{h:=|\Omega|\cdot\left[(\Delta u-Vu+f)^2-(\Delta u^*-Vu^*+f)^2\right]\mid u\in\mathcal{M}_\rho(\Omega)\right\} T ρ ( Ω ) := { h := ∣Ω∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] ∣ u ∈ M ρ ( Ω ) } can be upper bounded by a sub-root function
ϕ ( ρ ) : = O ( S 3 L ρ n log ( B W n ) ) . \phi(\rho):=O\left(\sqrt{\frac{S^3L\rho}{n}}\log(BWn)\right). ϕ ( ρ ) := O ( n S 3 L ρ log ( B W n ) ) . i.e. we have
ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( T ρ ( Ω ) ) ≤ ϕ ( ρ ) . (22) \phi(4\rho)\le2\phi(\rho)\text{ and }R_n(\mathcal{T}_\rho(\Omega))\le\phi(\rho). \tag*{(22)} ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( T ρ ( Ω )) ≤ ϕ ( ρ ) . (22) holds for all ρ ≳ n − 2 \rho\gtrsim n^{-2} ρ ≳ n − 2 .
Final Upper Bound
Deep Ritz Methods. In this subsection, we provide the proof of upper bounds for DRM. We first provide a meta-theorem to illustrate the approximation and generalization decomposition with a O ( 1 / n ) O(1/n) O ( 1/ n ) fast rate generalization bound[3 , 78 ] . Then we use truncated fourier basis estimator and neural network estimator as example to obtain the final rate.
Theorem 4.11 (Meta-theorem for Upper Bounds of Deep Ritz Methods). Let u ∗ ∈ H s ( Ω ) u^*\in H^s(\Omega) u ∗ ∈ H s ( Ω ) denote the true solution to the PDE model with Dirichlet boundary condition:
− Δ u + V u = f on Ω , u = 0 on ∂ Ω , (23) \begin{aligned}
-\Delta u+Vu&=f\text{ on }\Omega,\\
u&=0\text{ on }\partial\Omega, \tag*{(23)}
\end{aligned} − Δ u + V u u = f on Ω , = 0 on ∂ Ω , (23) where f ∈ L 2 ( Ω ) f\in L^2(\Omega) f ∈ L 2 ( Ω ) and V ∈ L ∞ ( Ω ) V\in L^\infty(\Omega) V ∈ L ∞ ( Ω ) with 0 < V min ≤ V ( x ) ≤ V max > 0 0<V_{\min}\le V(x)\le V_{\max}>0 0 < V m i n ≤ V ( x ) ≤ V m a x > 0 . For a fixed function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , consider the empirical loss induced by the Deep Ritz Method:
E n ( u ) = 1 n ∑ j = 1 n [ ∣ Ω ∣ ⋅ ( 1 2 ∣ ∇ u ( X j ) ∣ 2 + 1 2 V ( X j ) ∣ u ( X j ) ∣ 2 − f ( X j ) u ( X j ) ) ] . (24) \mathcal{E}_n(u)=\frac{1}{n}\sum_{j=1}^{n}\left[|\Omega|\cdot\left(\frac{1}{2}|\nabla u(X_j)|^2+\frac{1}{2}V(X_j)|u(X_j)|^2-f(X_j)u(X_j)\right)\right]. \tag*{(24)} E n ( u ) = n 1 j = 1 ∑ n [ ∣Ω∣ ⋅ ( 2 1 ∣∇ u ( X j ) ∣ 2 + 2 1 V ( X j ) ∣ u ( X j ) ∣ 2 − f ( X j ) u ( X j ) ) ] . (24) where { X j } j = 1 n \{X_j\}_{j=1}^n { X j } j = 1 n are datapoints uniformly sampled from the domain Ω \Omega Ω . Then the Deep Ritz estimator associated with function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) is defined as the minimizer of E n ( u ) \mathcal{E}_n(u) E n ( u ) over the function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) :
u ^ D R M = min u ∈ F ( Ω ) E n ( u ) \hat{u}_{DRM}=\min_{u\in\mathcal{F}(\Omega)}\mathcal{E}_n(u) u ^ D R M = u ∈ F ( Ω ) min E n ( u ) Moreover, we assume that there exists some constant C > 0 C>0 C > 0 such that all function u u u in the function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , the real solution u ∗ u^* u ∗ and f f f , V V V satisfy the following two conditions.
max { sup u ∈ F ( Ω ) ∥ u ∥ L ∞ ( Ω ) , sup u ∈ F ( Ω ) ∥ ∇ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ ∇ u ∗ ∥ L ∞ ( Ω ) , V max , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (25) \max\left\{ \sup_{u\in\mathcal{F}(\Omega)} \lVert u\rVert_{L^\infty(\Omega)},\ \sup_{u\in\mathcal{F}(\Omega)} \lVert\nabla u\rVert_{L^\infty(\Omega)},\ \lVert u^*\rVert_{L^\infty(\Omega)},\ \lVert\nabla u^*\rVert_{L^\infty(\Omega)},\ V_{\max},\ \lVert f\rVert_{L^\infty(\Omega)} \right\} \le C. \tag*{(25)} max { u ∈ F ( Ω ) sup ∥ u ∥ L ∞ ( Ω ) , u ∈ F ( Ω ) sup ∥ ∇ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ ∇ u ∗ ∥ L ∞ ( Ω ) , V m a x , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (25)
u = 0 on ∂ Ω . u=0\ \text{on }\partial\Omega. u = 0 on ∂ Ω.
At the the same time, for any ρ > 0 \rho>0 ρ > 0 , we assume the Rademacher complexity of a localized function space S ρ ( Ω ) : = { h : = ∣ Ω ∣ ⋅ [ 1 2 ( ∣ ∇ u ∣ 2 − ∣ ∇ u ∗ ∣ 2 ) + 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] | ∥ u − u ∗ ∥ H 1 2 ≤ ρ } S_\rho(\Omega):=\left\{h:=|\Omega|\cdot\left[\frac{1}{2}\left(|\nabla u|^2-|\nabla u^*|^2\right)+\frac{1}{2}V(|u|^2-|u^*|^2)-f(u-u^*)\right]\ \middle|\ \lVert u-u^*\rVert_{H^1}^2\le\rho\right\} S ρ ( Ω ) := { h := ∣Ω∣ ⋅ [ 2 1 ( ∣∇ u ∣ 2 − ∣∇ u ∗ ∣ 2 ) + 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] ∥ u − u ∗ ∥ H 1 2 ≤ ρ } can be upper bounded by a sub-root function ϕ = ϕ ( ρ ) : [ 0 , ∞ ) → [ 0 , ∞ ) \phi=\phi(\rho):[0,\infty)\to[0,\infty) ϕ = ϕ ( ρ ) : [ 0 , ∞ ) → [ 0 , ∞ ) , i.e.
ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( S ρ ( Ω ) ) ≤ ϕ ( ρ ) ( ∀ ρ > 0 ) . (26) \phi(4\rho)\le2\phi(\rho)\ \text{and}\ R_n(S_\rho(\Omega))\le\phi(\rho)\quad(\forall\ \rho>0). \tag*{(26)} ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( S ρ ( Ω )) ≤ ϕ ( ρ ) ( ∀ ρ > 0 ) . (26)
For all constant t > 0 t>0 t > 0 . We denote r ∗ r^* r ∗ to be the solution of the fix point equation of local Rademacher complexity r = ϕ ( r ) r=\phi(r) r = ϕ ( r ) . There exists a constant C p C_p C p such that for probability 1 − C p exp ( − t ) 1-C_p\exp(-t) 1 − C p exp ( − t ) , we have the following upper bound for the Deep Ritz Estimator
∥ u ^ D R M − u ∗ ∥ H 1 2 ≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \lVert\widehat{u}_{DRM}-u^*\rVert_{H^1}^2\lesssim\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right)+\max\left\{r^*,\frac{t}{n}\right\}. ∥ u D R M − u ∗ ∥ H 1 2 ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Proof. To upper bound the excess risk Δ E ( n ) : = E ( u ^ D R M ) − E ( u ∗ ) \Delta\mathcal{E}^{(n)}:=\mathcal{E}(\widehat{u}_{DRM})-\mathcal{E}(u^*) Δ E ( n ) := E ( u D R M ) − E ( u ∗ ) , following [78 , 47 , 14 ] , we decompose the excess risk into approximation error and generalization error with probability 1 − e − t 1-e^{-t} 1 − e − t .
Δ E ( n ) ( u ^ D R M ) = E ( u ^ D R M ) − E ( u ∗ ) = [ E ( u ^ D R M ) − E n ( u ^ D R M ) ] + [ E n ( u ^ D R M ) − E n ( u F ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u ^ D R M ) − E n ( u ^ D R M ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u ^ D R M ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ D R M ) ] + 3 2 [ E ( u F ) − E ( u ∗ ) ] + t 2 n . (27) \begin{aligned}
\Delta\mathcal{E}^{(n)}(\widehat{u}_{DRM})&=\mathcal{E}(\widehat{u}_{DRM})-\mathcal{E}(u^*)\\
&=\left[\mathcal{E}(\widehat{u}_{DRM})-\mathcal{E}_n(\widehat{u}_{DRM})\right]+\left[\mathcal{E}_n(\widehat{u}_{DRM})-\mathcal{E}_n(u_{\mathcal{F}})\right]\\
&\quad+\left[\mathcal{E}_n(u_{\mathcal{F}})-\mathcal{E}(u_{\mathcal{F}})\right]+\left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right]\\
&\le\left[\mathcal{E}(\widehat{u}_{DRM})-\mathcal{E}_n(\widehat{u}_{DRM})\right]+\left[\mathcal{E}_n(u_{\mathcal{F}})-\mathcal{E}(u_{\mathcal{F}})\right]+\left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right]\\
&\le\left[\mathcal{E}(\widehat{u}_{DRM})-\mathcal{E}(u^*)+\mathcal{E}_n(u^*)-\mathcal{E}_n(\widehat{u}_{DRM})\right]\\
&\quad+\frac{3}{2}\left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right]+\frac{t}{2n}. \tag*{(27)}
\end{aligned} Δ E ( n ) ( u D R M ) = E ( u D R M ) − E ( u ∗ ) = [ E ( u D R M ) − E n ( u D R M ) ] + [ E n ( u D R M ) − E n ( u F ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u D R M ) − E n ( u D R M ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u D R M ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u D R M ) ] + 2 3 [ E ( u F ) − E ( u ∗ ) ] + 2 n t . (27)
where the expectation is on all sampled data. The inequality of the third line is because the u u u is the minimizer of the empirical loss E n \mathcal{E}_n E n in the solution set F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , so we have E n ( u ) ≤ E n ( u F ) \mathcal{E}_n(u)\le\mathcal{E}_n(u_{\mathcal{F}}) E n ( u ) ≤ E n ( u F ) . The last inequality is based on the Bernstein inequality. The variance of h = ∣ Ω ∣ ⋅ [ 1 2 ( ∣ ∇ u ∣ 2 − ∣ ∇ u ∗ ∣ 2 ) + 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] h=|\Omega|\cdot\left[\frac{1}{2}\left(|\nabla u|^2-|\nabla u^*|^2\right)+\frac{1}{2}V(|u|^2-|u^*|^2)-f(u-u^*)\right] h = ∣Ω∣ ⋅ [ 2 1 ( ∣∇ u ∣ 2 − ∣∇ u ∗ ∣ 2 ) + 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] can be bounded by [ E ( u F ) − E ( u ∗ ) ] \left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right] [ E ( u F ) − E ( u ∗ ) ] due to the strong convexity of the variation objective (29) . According to the Bernstein inequality, we know with probability 1 − e − t 1-e^{-t} 1 − e − t we have
E n ( u F ) − E n ( u ∗ ) − E ( u F ) + E ( u ∗ ) ≤ t [ E ( u F ) − E ( u ∗ ) ] n ≤ 1 2 [ E ( u F ) − E ( u ∗ ) ] + t 2 n . \mathcal{E}_n(u_{\mathcal{F}})-\mathcal{E}_n(u^*)-\mathcal{E}(u_{\mathcal{F}})+\mathcal{E}(u^*)\le\sqrt{\frac{t\left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right]}{n}}\le\frac{1}{2}\left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right]+\frac{t}{2n}. E n ( u F ) − E n ( u ∗ ) − E ( u F ) + E ( u ∗ ) ≤ n t [ E ( u F ) − E ( u ∗ ) ] ≤ 2 1 [ E ( u F ) − E ( u ∗ ) ] + 2 n t .
Note that 4.18 holds for all function lies in the function space F \mathcal{F} F . Thus, we can take u F : = arg min u 0 ∈ F ( Ω ) ( E ( u 0 ) − E ( u ∗ ) ) u_{\mathcal{F}} := \operatorname*{arg\,min}_{u_0\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_0)-\mathcal{E}(u^\ast)\right) u F := arg min u 0 ∈ F ( Ω ) ( E ( u 0 ) − E ( u ∗ ) ) and finally get
Δ E ( n ) ≤ E ( u ^ D R M ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ ) ⏟ Δ E g e n + 3 2 inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) ⏟ Δ E a p p + t 2 m . \Delta\mathcal{E}^{(n)} \le\underbrace{\mathcal{E}(\hat{u}_{\mathrm{DRM}})-\mathcal{E}(u^\ast)+\mathcal{E}_n(u^\ast)-\mathcal{E}_n(\hat{u})}_{\Delta\mathcal{E}_{\mathrm{gen}}}+\frac{3}{2}\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\underbrace{\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^\ast)\right)}_{\Delta\mathcal{E}_{\mathrm{app}}}+\frac{t}{2m}. Δ E ( n ) ≤ Δ E gen E ( u ^ DRM ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ ) + 2 3 u F ∈ F ( Ω ) inf Δ E app ( E ( u F ) − E ( u ∗ ) ) + 2 m t .
This inequality decompose the excess risk to the generalization error Δ E g e n : = E ( u ^ D R M ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ D R M ) \Delta\mathcal{E}_{\mathrm{gen}} := \mathcal{E}(\hat{u}_{\mathrm{DRM}})-\mathcal{E}(u^\ast)+\mathcal{E}_n(u^\ast)-\mathcal{E}_n(\hat{u}_{\mathrm{DRM}}) Δ E gen := E ( u ^ DRM ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ DRM ) and the approximation error Δ E a p p = inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) \Delta\mathcal{E}_{\mathrm{app}}=\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^\ast)\right) Δ E app = inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) . We’ll focus on providing fast rate upper bounds of the generalization error for the two estimators using the localization technique[3 , 78 ] . To achieve the fast generalization bound, we focus on the following normalized empirical process
S ~ r ( Ω ) : = { h ~ ( x ) : = E [ h ] − h ( x ) E [ h ] + r | h ∈ S ( Ω ) } ( r > 0 ) . \widetilde{S}_r(\Omega):=\left\{\widetilde{h}(x):=\frac{\mathbb{E}[h]-h(x)}{\mathbb{E}[h]+r}\ \middle|\ h\in S(\Omega)\right\}\ (r>0). S r ( Ω ) := { h ( x ) := E [ h ] + r E [ h ] − h ( x ) h ∈ S ( Ω ) } ( r > 0 ) .
First, we try to bound the expectation of the normalized empirical process. Applying the Symmetrization Lemma A.3, we can first bound the expectation as
sup h ~ ∈ S ~ r ( Ω ) E x ′ [ 1 n ∑ i = 1 n h ~ ( x i ′ ) ] ≤ E x ′ [ sup h ∈ S ( Ω ) ∣ 1 n ∑ i = 1 n h ( x i ′ ) − E [ h ] E [ h ] + r ∣ ] ≤ 2 R n ( S ^ r ( Ω ) ) . \sup_{\widetilde{h}\in\widetilde{S}_r(\Omega)}\mathbb{E}_{x'}\left[\frac{1}{n}\sum_{i=1}^{n}\widetilde{h}(x'_i)\right]\leq\mathbb{E}_{x'}\left[\sup_{h\in S(\Omega)}\left|\frac{1}{n}\sum_{i=1}^{n}\frac{h(x'_i)-\mathbb{E}[h]}{\mathbb{E}[h]+r}\right|\right]\leq2R_n(\widehat{S}_r(\Omega)). h ∈ S r ( Ω ) sup E x ′ [ n 1 i = 1 ∑ n h ( x i ′ ) ] ≤ E x ′ [ h ∈ S ( Ω ) sup n 1 i = 1 ∑ n E [ h ] + r h ( x i ′ ) − E [ h ] ] ≤ 2 R n ( S r ( Ω )) .
where the function class S ^ r ( Ω ) \widehat{S}_r(\Omega) S r ( Ω ) is defined as:
S ^ r ( Ω ) : = { h ^ ( x ) : = h ( x ) E [ h ] + r | h ∈ S ( Ω ) } , \widehat{S}_r(\Omega):=\left\{\widehat{h}(x):=\frac{h(x)}{\mathbb{E}[h]+r}\ \middle|\ h\in S(\Omega)\right\}, S r ( Ω ) := { h ( x ) := E [ h ] + r h ( x ) h ∈ S ( Ω ) } ,
where S ( Ω ) = { h : = ∣ Ω ∣ ⋅ [ 1 2 ( ∣ ∇ u ∣ 2 − ∣ ∇ u ∗ ∣ 2 ) + 1 2 ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] } S(\Omega)=\left\{h:=|\Omega|\cdot\left[\frac{1}{2}\left(|\nabla u|^2-|\nabla u^\ast|^2\right)+\frac{1}{2}\left(|u|^2-|u^\ast|^2\right)-f(u-u^\ast)\right]\right\} S ( Ω ) = { h := ∣Ω∣ ⋅ [ 2 1 ( ∣∇ u ∣ 2 − ∣∇ u ∗ ∣ 2 ) + 2 1 ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] } . Then Applying the Peeling Lemma to any function h ∈ S ( Ω ) h\in S(\Omega) h ∈ S ( Ω ) helps us upper bound the local Rademacher complexity R n ( S ^ r ( Ω ) ) R_n(\widehat{S}_r(\Omega)) R n ( S r ( Ω )) with the function ϕ \phi ϕ defined in equation 4.17:
R n ( S ^ r ( Ω ) ) = E σ [ E x [ sup h ∈ S ( Ω ) 1 n ∑ i = 1 n σ i h ( x i ) E [ h ] + r ] ] ≤ 4 ϕ ( r ) r . R_n(\widehat{S}_r(\Omega))=\mathbb{E}_{\sigma}\left[\mathbb{E}_{x}\left[\sup_{h\in S(\Omega)}\frac{\frac{1}{n}\sum_{i=1}^{n}\sigma_i h(x_i)}{\mathbb{E}[h]+r}\right]\right]\leq\frac{4\phi(r)}{r}. R n ( S r ( Ω )) = E σ [ E x [ h ∈ S ( Ω ) sup E [ h ] + r n 1 ∑ i = 1 n σ i h ( x i ) ] ] ≤ r 4 ϕ ( r ) .
Combining all inequalities derived above yields:
sup h ~ ∈ S ~ r ( Ω ) E x ′ [ 1 n ∑ i = 1 n h ~ ( x i ′ ) ] ≤ 2 R n ( S ^ r ( Ω ) ) ≤ 8 ϕ ( r ) r ( r > 0 ) . (28) \sup_{\widetilde{h}\in\widetilde{S}_r(\Omega)}\mathbb{E}_{x'}\left[\frac{1}{n}\sum_{i=1}^{n}\widetilde{h}(x'_i)\right]\leq2R_n(\widehat{S}_r(\Omega))\leq\frac{8\phi(r)}{r}\quad(r>0). \tag*{(28)} h ∈ S r ( Ω ) sup E x ′ [ n 1 i = 1 ∑ n h ( x i ′ ) ] ≤ 2 R n ( S r ( Ω )) ≤ r 8 ϕ ( r ) ( r > 0 ) . (28)
Secondly we’ll apply the Talagrand concentration inequality, which requires us to verify the condition needed. We will first check that the expectation value E [ h ] \mathbb{E}[h] E [ h ] is always non-negative for any h ∈ S ( Ω ) h \in\mathcal{S}(\Omega) h ∈ S ( Ω ) :
E [ h ] = 1 ∣ Ω ∣ ∫ Ω ( 1 2 ∣ ∇ u ( x ) ∣ 2 + 1 2 V ( x ) ∣ u ( x ) ∣ 2 − f ( x ) u ( x ) ) d x − 1 ∣ Ω ∣ ∫ Ω ( 1 2 ∣ ∇ u ∗ ( x ) ∣ 2 + 1 2 V ( x ) ∣ u ∗ ( x ) ∣ 2 − f ( x ) u ∗ ( x ) ) d x = E ( u ) − E ( u ∗ ) ≥ 0 ⇒ E [ h ] ≥ 0. \begin{aligned}
\mathbb{E}[h] &= \frac{1}{|\Omega|}\int_{\Omega}\left(\frac{1}{2}|\nabla u(x)|^2+\frac{1}{2}V(x)|u(x)|^2-f(x)u(x)\right)\,dx \\
&\quad-\frac{1}{|\Omega|}\int_{\Omega}\left(\frac{1}{2}|\nabla u^*(x)|^2+\frac{1}{2}V(x)|u^*(x)|^2-f(x)u^*(x)\right)\,dx \\
&= \mathcal{E}(u)-\mathcal{E}(u^*)\geq0 \Rightarrow\mathbb{E}[h]\geq0.
\end{aligned} E [ h ] = ∣Ω∣ 1 ∫ Ω ( 2 1 ∣∇ u ( x ) ∣ 2 + 2 1 V ( x ) ∣ u ( x ) ∣ 2 − f ( x ) u ( x ) ) d x − ∣Ω∣ 1 ∫ Ω ( 2 1 ∣∇ u ∗ ( x ) ∣ 2 + 2 1 V ( x ) ∣ u ∗ ( x ) ∣ 2 − f ( x ) u ∗ ( x ) ) d x = E ( u ) − E ( u ∗ ) ≥ 0 ⇒ E [ h ] ≥ 0.
We will proceed to verify that any h ~ = E [ h ] − h E [ h ] + r ∈ S ~ r ( Ω ) \tilde{h}=\frac{\mathbb{E}[h]-h}{\mathbb{E}[h]+r}\in\widetilde{\mathcal{S}}_r(\Omega) h ~ = E [ h ] + r E [ h ] − h ∈ S r ( Ω ) is of bounded inf-norm. We need to prove that any h ∈ S ( Ω ) h\in\mathcal{S}(\Omega) h ∈ S ( Ω ) is of bounded inf-norm beforehand. Using boundedness condition listed in equation (25) implies:
∥ h ∥ ∞ = ∥ ∣ Ω ∣ 2 ( ∣ ∇ u ∣ 2 − ∣ ∇ u ∗ ∣ 2 ) + ∣ Ω ∣ 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ∥ ∞ ≤ ∣ Ω ∣ 2 ( ∥ ∇ u ∥ ∞ 2 + ∥ ∇ u ∗ ∥ ∞ 2 ) + ∣ Ω ∣ 2 V max ( ∥ u ∥ ∞ 2 + ∥ u ∗ ∥ ∞ 2 ) + ∣ Ω ∣ ∥ f ∥ ∞ ( ∥ u ∥ ∞ + ∥ u ∗ ∥ ∞ ) ≤ ∣ Ω ∣ 2 × 2 C 2 + ∣ Ω ∣ 2 V max × 2 C 2 + 2 ∣ Ω ∣ C 2 = ∣ Ω ∣ ( V max + 3 ) C 2 \begin{aligned}
\|h\|_\infty
&=\left\|\frac{|\Omega|}{2}\left(|\nabla u|^2-|\nabla u^*|^2\right)+\frac{|\Omega|}{2}V\left(|u|^2-|u^*|^2\right)-f(u-u^*)\right\|_\infty\\
&\leq\frac{|\Omega|}{2}\left(\|\nabla u\|_\infty^2+\|\nabla u^*\|_\infty^2\right)+\frac{|\Omega|}{2}V_{\max}\left(\|u\|_\infty^2+\|u^*\|_\infty^2\right)+|\Omega|\|f\|_\infty\left(\|u\|_\infty+\|u^*\|_\infty\right) \\
&\leq\frac{|\Omega|}{2}\times2\mathcal{C}^2+\frac{|\Omega|}{2}V_{\max}\times2\mathcal{C}^2+2|\Omega|\mathcal{C}^2=|\Omega|(V_{\max}+3)\mathcal{C}^2
\end{aligned} ∥ h ∥ ∞ = 2 ∣Ω∣ ( ∣∇ u ∣ 2 − ∣∇ u ∗ ∣ 2 ) + 2 ∣Ω∣ V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ∞ ≤ 2 ∣Ω∣ ( ∥∇ u ∥ ∞ 2 + ∥∇ u ∗ ∥ ∞ 2 ) + 2 ∣Ω∣ V m a x ( ∥ u ∥ ∞ 2 + ∥ u ∗ ∥ ∞ 2 ) + ∣Ω∣∥ f ∥ ∞ ( ∥ u ∥ ∞ + ∥ u ∗ ∥ ∞ ) ≤ 2 ∣Ω∣ × 2 C 2 + 2 ∣Ω∣ V m a x × 2 C 2 + 2∣Ω∣ C 2 = ∣Ω∣ ( V m a x + 3 ) C 2
By taking M : = ∣ Ω ∣ ( V max + 3 ) C 2 M:=|\Omega|(V_{\max}+3)\mathcal{C}^2 M := ∣Ω∣ ( V m a x + 3 ) C 2 , we then have ∥ h ∥ ∞ ≤ M \|h\|_\infty\leq M ∥ h ∥ ∞ ≤ M for all h ∈ S ( Ω ) h\in\mathcal{S}(\Omega) h ∈ S ( Ω ) . Note that the denominator can be lower bounded by ∣ E [ h ] + r ∣ ≥ r > 0 |\mathbb{E}[h]+r|\geq r>0 ∣ E [ h ] + r ∣ ≥ r > 0 . Combining these two inequalities help us upper bound the inf-norm ∥ h ~ ∥ ∞ = sup x ∈ Ω ∣ h ~ ( x ) ∣ \|\tilde{h}\|_\infty=\sup_{x\in\Omega}|\tilde{h}(x)| ∥ h ~ ∥ ∞ = sup x ∈ Ω ∣ h ~ ( x ) ∣ as follows:
∥ h ~ ∥ ∞ = ∥ E [ h ] − h ∥ ∞ ∣ E [ h ] + r ∣ ≤ 2 ∥ h ∥ ∞ r ≤ 2 M r = : β . \|\tilde{h}\|_\infty=\frac{\|\mathbb{E}[h]-h\|_\infty}{|\mathbb{E}[h]+r|}\leq\frac{2\|h\|_\infty}{r}\leq\frac{2M}{r}=:\beta. ∥ h ~ ∥ ∞ = ∣ E [ h ] + r ∣ ∥ E [ h ] − h ∥ ∞ ≤ r 2∥ h ∥ ∞ ≤ r 2 M =: β .
We will then check the normalized functions E [ h ] − h ( x ) E [ h ] + r \frac{\mathbb{E}[h]-h(x)}{\mathbb{E}[h]+r} E [ h ] + r E [ h ] − h ( x ) in S ~ r ( Ω ) \widetilde{\mathcal{S}}_r(\Omega) S r ( Ω ) have bounded second moment, which is satisfied because of the regularity results of the PDE. We aim to show that there exist some constants α , α ′ > 0 \alpha,\alpha'>0 α , α ′ > 0 , such that for any h ∈ S ( Ω ) h\in\mathcal{S}(\Omega) h ∈ S ( Ω ) , the following inequality holds:
α E [ h 2 ] ≤ ∥ u − u ∗ ∥ H 1 ( Ω ) 2 ≤ α ′ E [ h ] . (29) \alpha\mathbb{E}[h^2]\leq\|u-u^*\|_{H^1(\Omega)}^2\leq\alpha'\mathbb{E}[h]. \tag*{(29)} α E [ h 2 ] ≤ ∥ u − u ∗ ∥ H 1 ( Ω ) 2 ≤ α ′ E [ h ] . (29)
The RHS of the inequality follows from strong convexity of the DRM objective function proved in Theorem A.1 :
E [ h ] = E ( u ) − E ( u ∗ ) ≥ min { 1 , V min } 4 ∥ u − u ∗ ∥ H 1 ( Ω ) 2 \mathbb{E}[h]=\mathcal{E}(u)-\mathcal{E}(u^*)\geq\frac{\min\{1,V_{\min}\}}{4}\|u-u^*\|_{H^1(\Omega)}^2 E [ h ] = E ( u ) − E ( u ∗ ) ≥ 4 min { 1 , V m i n } ∥ u − u ∗ ∥ H 1 ( Ω ) 2
The LHS of the inequality follows from boundedness condition listed in equation (25) and the QM-AM inequality:
E [ h 2 ] = ∫ Ω [ 1 2 ( ∣ ∇ u ∣ 2 − ∣ ∇ u ∗ ∣ 2 ) + 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] 2 d x ≤ 3 4 ∫ Ω ( ∣ ∇ u ∣ 2 − ∣ ∇ u ∗ ∣ 2 ) 2 d x + 3 4 ∫ Ω V 2 ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) 2 d x + 3 ∫ Ω f 2 ( u − u ∗ ) 2 d x ≤ 3 4 ∫ Ω ∣ ∇ u − ∇ u ∗ ∣ 2 ( ∣ ∇ u ∣ + ∣ ∇ u ∗ ∣ ) 2 d x + 3 4 V max 2 ∫ Ω ∣ u − u ∗ ∣ 2 ( ∣ u ∣ + ∣ u ∗ ∣ ) 2 d x + 3 C 2 ∫ Ω ( u − u ∗ ) 2 d x ≤ 3 C 2 ∫ Ω ∣ ∇ u − ∇ u ∗ ∣ 2 d x + 3 C 2 ( 1 + V max 2 ) ∫ Ω ∣ u − u ∗ ∣ 2 d x ≤ 3 C 2 ( 1 + V max 2 ) ∥ u − u ∗ ∥ H 1 ( Ω ) 2 \begin{aligned}
\mathbb{E}[h^2]
&=\int_{\Omega}\left[\frac{1}{2}\left(|\nabla u|^2-|\nabla u^*|^2\right)+\frac{1}{2}V\left(|u|^2-|u^*|^2\right)-f(u-u^*)\right]^2dx \\
&\leq\frac{3}{4}\int_{\Omega}\left(|\nabla u|^2-|\nabla u^*|^2\right)^2dx+\frac{3}{4}\int_{\Omega}V^2\left(|u|^2-|u^*|^2\right)^2dx+3\int_{\Omega}f^2(u-u^*)^2dx \\
&\leq\frac{3}{4}\int_{\Omega}|\nabla u-\nabla u^*|^2(|\nabla u|+|\nabla u^*|)^2dx+\frac{3}{4}V_{\max}^2\int_{\Omega}|u-u^*|^2(|u|+|u^*|)^2dx \\
&\quad+3\mathcal{C}^2\int_{\Omega}(u-u^*)^2dx\leq3\mathcal{C}^2\int_{\Omega}|\nabla u-\nabla u^*|^2dx+3\mathcal{C}^2(1+V_{\max}^2)\int_{\Omega}|u-u^*|^2dx \\
&\leq3\mathcal{C}^2(1+V_{\max}^2)\|u-u^*\|_{H^1(\Omega)}^2
\end{aligned} E [ h 2 ] = ∫ Ω [ 2 1 ( ∣∇ u ∣ 2 − ∣∇ u ∗ ∣ 2 ) + 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] 2 d x ≤ 4 3 ∫ Ω ( ∣∇ u ∣ 2 − ∣∇ u ∗ ∣ 2 ) 2 d x + 4 3 ∫ Ω V 2 ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) 2 d x + 3 ∫ Ω f 2 ( u − u ∗ ) 2 d x ≤ 4 3 ∫ Ω ∣∇ u − ∇ u ∗ ∣ 2 ( ∣∇ u ∣ + ∣∇ u ∗ ∣ ) 2 d x + 4 3 V m a x 2 ∫ Ω ∣ u − u ∗ ∣ 2 ( ∣ u ∣ + ∣ u ∗ ∣ ) 2 d x + 3 C 2 ∫ Ω ( u − u ∗ ) 2 d x ≤ 3 C 2 ∫ Ω ∣∇ u − ∇ u ∗ ∣ 2 d x + 3 C 2 ( 1 + V m a x 2 ) ∫ Ω ∣ u − u ∗ ∣ 2 d x ≤ 3 C 2 ( 1 + V m a x 2 ) ∥ u − u ∗ ∥ H 1 ( Ω ) 2
By picking α ′ = 4 min { 1 , V min } \alpha' = \frac{4}{\min\{1,V_{\min}\}} α ′ = m i n { 1 , V m i n } 4 and α = 1 3 C 2 ( 1 + V max 2 ) \alpha= \frac{1}{3C^2(1+V_{\max}^2)} α = 3 C 2 ( 1 + V m a x 2 ) 1 , we have finished proving inequality 4.20. Then we can can upper bound the expectation E [ h ~ 2 ] \mathbb{E}[\tilde{h}^2] E [ h ~ 2 ] as:
E [ h ~ 2 ] = E [ ( h − E [ h ] ) 2 ] ∣ E [ h ] + r ∣ 2 = E [ h 2 ] − E [ h ] 2 ∣ E [ h ] + r ∣ 2 ≤ E [ h 2 ] ∣ E [ h ] + r ∣ 2 . \mathbb{E}[\tilde{h}^2] = \frac{\mathbb{E}[(h-\mathbb{E}[h])^2]}{\lvert\mathbb{E}[h]+r\rvert^2} = \frac{\mathbb{E}[h^2]-\mathbb{E}[h]^2}{\lvert\mathbb{E}[h]+r\rvert^2} \le\frac{\mathbb{E}[h^2]}{\lvert\mathbb{E}[h]+r\rvert^2}. E [ h ~ 2 ] = ∣ E [ h ] + r ∣ 2 E [( h − E [ h ] ) 2 ] = ∣ E [ h ] + r ∣ 2 E [ h 2 ] − E [ h ] 2 ≤ ∣ E [ h ] + r ∣ 2 E [ h 2 ] .
Using the fact that E [ h ] ≥ 0 \mathbb{E}[h] \ge0 E [ h ] ≥ 0 and inequality 4.20, we can lower bound the denominator ∣ E [ h ] + r ∣ 2 \lvert\mathbb{E}[h]+r\rvert^2 ∣ E [ h ] + r ∣ 2 as follows:
∣ E [ h ] + r ∣ 2 ≥ 2 E [ h ] r ≥ 2 r α α ′ E [ h 2 ] . \lvert\mathbb{E}[h]+r\rvert^2 \ge2\mathbb{E}[h]r \ge\frac{2r\alpha}{\alpha'}\mathbb{E}[h^2]. ∣ E [ h ] + r ∣ 2 ≥ 2 E [ h ] r ≥ α ′ 2 r α E [ h 2 ] .
Therefore, we can deduce that:
E [ h ~ 2 ] ≤ E [ h 2 ] ∣ E [ h ] + r ∣ 2 ≤ E [ h 2 ] 2 r α α ′ E [ h 2 ] = α ′ 2 r α = : σ 2 . \mathbb{E}[\tilde{h}^2] \le\frac{\mathbb{E}[h^2]}{\lvert\mathbb{E}[h]+r\rvert^2} \le\frac{\mathbb{E}[h^2]}{\frac{2r\alpha}{\alpha'}\mathbb{E}[h^2]} = \frac{\alpha'}{2r\alpha} =: \sigma^2. E [ h ~ 2 ] ≤ ∣ E [ h ] + r ∣ 2 E [ h 2 ] ≤ α ′ 2 r α E [ h 2 ] E [ h 2 ] = 2 r α α ′ =: σ 2 .
Hence, any function in the localized class S ~ r ( Ω ) \tilde{S}_r(\Omega) S ~ r ( Ω ) is of bounded second moment.
It is easy to check that for any h ~ ∈ S ~ r ( Ω ) \tilde{h} \in\tilde{S}_r(\Omega) h ~ ∈ S ~ r ( Ω ) , we have
E [ h ~ ] = E [ h ] − E [ h ] E [ h ] + r = 0 , \mathbb{E}[\tilde{h}] = \frac{\mathbb{E}[h]-\mathbb{E}[h]}{\mathbb{E}[h]+r} = 0, E [ h ~ ] = E [ h ] + r E [ h ] − E [ h ] = 0 ,
i.e. any function in the localized class S ~ r ( Ω ) \tilde{S}_r(\Omega) S ~ r ( Ω ) is of zero mean.
Now we have verified that any function h ~ ∈ S ~ r ( Ω ) \tilde{h} \in\tilde{S}_r(\Omega) h ~ ∈ S ~ r ( Ω ) satisfies all the required conditions. By taking μ \mu μ to be the uniform distribution on the domain Ω \Omega Ω and applying Talagrand’s Concentration inequality given in Lemma A.6, we have:
P x [ sup h ~ ∈ S ~ r ( Ω ) 1 n ∑ i = 1 n h ~ ( x i ) ≥ 2 sup h ~ ∈ S ~ r ( Ω ) E x ′ [ 1 n ∑ i = 1 n h ~ ( x i ′ ) ] + 2 t σ 2 n + 2 t β n ] ≤ e − t . \mathbb{P}_x\left[\sup_{\tilde{h}\in\tilde{S}_r(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x_i)\ge2\sup_{\tilde{h}\in\tilde{S}_r(\Omega)}\mathbb{E}_{x'}\left[\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x_i')\right]+\sqrt{\frac{2t\sigma^2}{n}+\frac{2t\beta}{n}}\right]\le e^{-t}. P x [ h ~ ∈ S ~ r ( Ω ) sup n 1 i = 1 ∑ n h ~ ( x i ) ≥ 2 h ~ ∈ S ~ r ( Ω ) sup E x ′ [ n 1 i = 1 ∑ n h ~ ( x i ′ ) ] + n 2 t σ 2 + n 2 tβ ] ≤ e − t .
By using the upper bound deduced above and plugging in the expressions of β \beta β and σ \sigma σ , we can rewrite Talagrand’s Concentration Inequality in the following way. With probability at least 1 − e − t 1-e^{-t} 1 − e − t , the inequality below holds:
1 n ∑ i = 1 n h ~ ( x i ) ≤ sup h ~ ∈ S ~ r ( Ω ) 1 n ∑ i = 1 n h ~ ( x i ) ≤ 2 sup h ~ ∈ S ~ r ( Ω ) E x ′ [ 1 n ∑ i = 1 n h ~ ( x i ′ ) ] + 2 t σ 2 n + 2 t β n ≤ 16 ϕ ( r ) r + t α ′ n α r + 4 M t n r = : ψ ( r ) . \begin{aligned}
\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x_i)\le\sup_{\tilde{h}\in\tilde{S}_r(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x_i)\le2\sup_{\tilde{h}\in\tilde{S}_r(\Omega)}\mathbb{E}_{x'}\left[\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x_i')\right]+\sqrt{\frac{2t\sigma^2}{n}+\frac{2t\beta}{n}} \\
&\le\frac{16\phi(r)}{r}+\sqrt{\frac{t\alpha'}{n\alpha r}+\frac{4Mt}{nr}}=:\psi(r).
\end{aligned} n 1 i = 1 ∑ n h ~ ( x i ) ≤ h ~ ∈ S ~ r ( Ω ) sup n 1 i = 1 ∑ n h ~ ( x i ) ≤ 2 h ~ ∈ S ~ r ( Ω ) sup E x ′ [ n 1 i = 1 ∑ n h ~ ( x i ′ ) ] + n 2 t σ 2 + n 2 tβ ≤ r 16 ϕ ( r ) + n α r t α ′ + n r 4 M t =: ψ ( r ) .
Let’s pick the critical radius r 0 r_0 r 0 to be:
r 0 = max { 2 14 r ∗ , 24 M t n , 36 α ′ t α n } . (30) r_0=\max\left\{2^{14}r_*,\frac{24Mt}{n},\frac{36\alpha't}{\alpha n}\right\}. \tag*{(30)} r 0 = max { 2 14 r ∗ , n 24 M t , α n 36 α ′ t } . (30)
Note that concavity of the function ϕ \phi ϕ implies that ϕ ( r ) ≤ r \phi(r) \le r ϕ ( r ) ≤ r for any r ≥ r ∗ r \ge r^* r ≥ r ∗ . Combining this with the first inequality listed in 4.17 yields:
16 ϕ ( r ) r ≤ 2 11 ϕ ( r 0 2 14 ) 2 14 r 0 2 14 = 1 8 × ϕ ( r 0 2 14 ) r 2 14 ≤ 1 8 . \frac{16\phi(r)}{r} \le\frac{2^{11}\phi\left(\frac{r_0}{2^{14}}\right)}{2^{14}\frac{r_0}{2^{14}}} = \frac{1}{8} \times\frac{\phi\left(\frac{r_0}{2^{14}}\right)}{\frac{r}{2^{14}}} \le\frac{1}{8}. r 16 ϕ ( r ) ≤ 2 14 2 14 r 0 2 11 ϕ ( 2 14 r 0 ) = 8 1 × 2 14 r ϕ ( 2 14 r 0 ) ≤ 8 1 .
On the other hand, applying equation 4.21 yields:
α t ′ n α r 0 ≤ α t n α 36 α ′ t = 1 6 , \sqrt{\frac{\alpha t'}{n\alpha r_0}} \le\sqrt{\frac{\alpha t}{n\alpha36\alpha't}} = \frac{1}{6}, n α r 0 α t ′ ≤ n α 36 α ′ t α t = 6 1 ,
4 M t n r 0 ≤ 4 M t n × n 24 M t = 1 6 . \frac{4Mt}{nr_0} \le\frac{4Mt}{n} \times\frac{n}{24Mt} = \frac{1}{6}. n r 0 4 M t ≤ n 4 M t × 24 M t n = 6 1 .
Summing the three inequalities above implies:
ψ ( r 0 ) = 16 ϕ ( r 0 ) r 0 + t α ′ n α r 0 + 4 M t n r 0 ≤ 1 8 + 1 6 + 1 6 < 1 2 . \psi(r_0) = \frac{16\phi(r_0)}{r_0} + \sqrt{\frac{t\alpha'}{n\alpha r_0}} + \frac{4Mt}{nr_0} \le\frac{1}{8} + \frac{1}{6} + \frac{1}{6} < \frac{1}{2}. ψ ( r 0 ) = r 0 16 ϕ ( r 0 ) + n α r 0 t α ′ + n r 0 4 M t ≤ 8 1 + 6 1 + 6 1 < 2 1 .
By picking r = r 0 r=r_0 r = r 0 , we can further deduce that for any function u ∈ F ( Ω ) u \in\mathcal{F}(\Omega) u ∈ F ( Ω ) , the following inequality holds with probability 1 − e − t 1-e^{-t} 1 − e − t :
E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) E ( u ) − E ( u ∗ ) + r 0 = 1 n ∑ i = 1 n h ~ ( x i ) ≤ ψ ( r 0 ) < 1 2 . \frac{\mathcal{E}(u)-\mathcal{E}(u^*)-\mathcal{E}_n(u)+\mathcal{E}_n(u^*)}{\mathcal{E}(u)-\mathcal{E}(u^*)+r_0}
= \frac{1}{n}\sum_{i=1}^{n}\widetilde{h}(x_i) \le\psi(r_0) < \frac{1}{2}. E ( u ) − E ( u ∗ ) + r 0 E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) = n 1 i = 1 ∑ n h ( x i ) ≤ ψ ( r 0 ) < 2 1 .
Multiplying the denominator on both sides indicates:
Δ E g e n = E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) ≤ 1 2 [ E ( u ) − E ( u ∗ ) ] + 1 2 r 0 = 1 2 Δ E ( n ) + 1 2 r 0 . \Delta\mathcal{E}_{\mathrm{gen}} = \mathcal{E}(u)-\mathcal{E}(u^*)-\mathcal{E}_n(u)+\mathcal{E}_n(u^*) \le\frac{1}{2}\left[\mathcal{E}(u)-\mathcal{E}(u^*)\right]+\frac{1}{2}r_0 = \frac{1}{2}\Delta\mathcal{E}^{(n)}+\frac{1}{2}r_0. Δ E gen = E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) ≤ 2 1 [ E ( u ) − E ( u ∗ ) ] + 2 1 r 0 = 2 1 Δ E ( n ) + 2 1 r 0 .
Substituting the upper bound above into the decomposition Δ E ( n ) ≤ Δ E g e n + 3 2 Δ E a p p + t 2 n \Delta\mathcal{E}^{(n)} \le\Delta\mathcal{E}_{\mathrm{gen}}+\frac{3}{2}\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{2n} Δ E ( n ) ≤ Δ E gen + 2 3 Δ E app + 2 n t yields that with probability 1 − e − t 1-e^{-t} 1 − e − t , we have:
Δ E ( n ) ≤ Δ E g e n + 3 2 Δ E a p p + t 2 n ≤ 1 2 Δ E ( n ) + 1 2 r 0 + 3 2 Δ E a p p + t 2 n . \Delta\mathcal{E}^{(n)} \le\Delta\mathcal{E}_{\mathrm{gen}}+\frac{3}{2}\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{2n} \le\frac{1}{2}\Delta\mathcal{E}^{(n)}+\frac{1}{2}r_0+\frac{3}{2}\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{2n}. Δ E ( n ) ≤ Δ E gen + 2 3 Δ E app + 2 n t ≤ 2 1 Δ E ( n ) + 2 1 r 0 + 2 3 Δ E app + 2 n t .
Simplifying the inequality above yields that with probability 1 − e − t 1-e^{-t} 1 − e − t , we have:
Δ E ( n ) ≤ r 0 + 3 Δ E a p p + t n = 3 inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { 2 14 r ∗ , 24 M t n , 36 α ′ t α n } + t n \Delta\mathcal{E}^{(n)} \le r_0+3\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{n}
= 3\inf_{u_F\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_F)-\mathcal{E}(u^*)\right)+\max\left\{2^{14}r^*,\frac{24Mt}{n},\frac{36\alpha't}{\alpha n}\right\}+\frac{t}{n} Δ E ( n ) ≤ r 0 + 3Δ E app + n t = 3 u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { 2 14 r ∗ , n 24 M t , α n 36 α ′ t } + n t
≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \lesssim\inf_{u_F\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_F)-\mathcal{E}(u^*)\right)+\max\left\{r^*,\frac{t}{n}\right\}. ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Moreover, using strong convexity of the DRM objective function proved in Theorem A.1 implies:
Δ E ( n ) = E ( u ^ D R M ) − E ( u ∗ ) ≥ min { 1 , V min } 4 ∥ u ^ D R M − u ∗ ∥ H 1 ( Ω ) 2 . \Delta\mathcal{E}^{(n)}=\mathcal{E}(\hat{u}_{\mathrm{DRM}})-\mathcal{E}(u^*)\ge\frac{\min\{1,V_{\min}\}}{4}\left\|\hat{u}_{\mathrm{DRM}}-u^*\right\|_{H^1(\Omega)}^2. Δ E ( n ) = E ( u ^ DRM ) − E ( u ∗ ) ≥ 4 min { 1 , V m i n } ∥ u ^ DRM − u ∗ ∥ H 1 ( Ω ) 2 .
Combining the two bounds above yields that with probability 1 − e − t 1-e^{-t} 1 − e − t , we have:
∥ u ^ D R M − u ∗ ∥ H 1 ( Ω ) 2 ≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \left\|\hat{u}_{\mathrm{DRM}}-u^*\right\|_{H^1(\Omega)}^2 \lesssim\inf_{u_F\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_F)-\mathcal{E}(u^*)\right)+\max\left\{r^*,\frac{t}{n}\right\}. ∥ u ^ DRM − u ∗ ∥ H 1 ( Ω ) 2 ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Theorem 4.12. (Final Upper Bound of DRM with Deep Neural Network Estimator ) With proper assumptions, consider the sparse Deep Neural Network function space Φ ( L , W , S , B ) \Phi(L,W,S,B) Φ ( L , W , S , B ) with parameters L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( n d d + 2 s − 2 ) W=O(n^{\frac{d}{d+2s-2}}) W = O ( n d + 2 s − 2 d ) , S = O ( n d d + 2 s − 2 ) S=O(n^{\frac{d}{d+2s-2}}) S = O ( n d + 2 s − 2 d ) , B = O ( 1 ) B=O(1) B = O ( 1 ) , then the Deep ritz estimator u ^ D R M D N N = min u ∈ Φ ( L , W , S , B ) E n D R M ( u ) \hat{u}^{DNN}_{DRM}=\min_{u\in\Phi(L,W,S,B)}\mathcal{E}^{DRM}_{n}(u) u ^ D R M D N N = min u ∈ Φ ( L , W , S , B ) E n D R M ( u ) satisfies the following upper bound with high probability:
∥ u ^ D R M D N N − u ∗ ∥ H 1 2 ≲ n − 2 s − 2 d + 2 s − 2 log n . \left\|\hat{u}^{DNN}_{DRM}-u^*\right\|_{H^1}^2\lesssim n^{-\frac{2s-2}{d+2s-2}}\log n. u ^ D R M D N N − u ∗ H 1 2 ≲ n − d + 2 s − 2 2 s − 2 log n .
Proof. On the one hand, by taking s = 1 s=1 s = 1 and p = 2 p=2 p = 2 in Theorem 4.5 proved above, we have that there exists some Deep Neural Network u D N N ∈ Φ ( L , W , S , B ) u_{DNN}\in\Phi(L,W,S,B) u D N N ∈ Φ ( L , W , S , B ) with L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( N ) W=O(N) W = O ( N ) , S = O ( N ) S=O(N) S = O ( N ) , B = O ( N ) B=O(N) B = O ( N ) , such that.
∥ u D N N − u ∗ ∥ H 1 ( Ω ) 2 ≤ N − 2 s − 2 d ∥ u ∗ ∥ H s ( Ω ) 2 . \left\|u_{DNN}-u^*\right\|_{H^1(\Omega)}^2\leq N^{-\frac{2s-2}{d}}\left\|u^*\right\|_{H^s(\Omega)}^2. ∥ u D N N − u ∗ ∥ H 1 ( Ω ) 2 ≤ N − d 2 s − 2 ∥ u ∗ ∥ H s ( Ω ) 2 .
Applying strong convexity of the DRM objective function proved in § 2.1 further implies:
Δ E a p p ≲ ∥ u D N N − u ∗ ∥ H 1 ( Ω ) 2 ≤ N − 2 s − 2 d . \Delta\mathcal{E}_{app}\lesssim\left\|u_{DNN}-u^*\right\|_{H^1(\Omega)}^2\leq N^{-\frac{2s-2}{d}}. Δ E a pp ≲ ∥ u D N N − u ∗ ∥ H 1 ( Ω ) 2 ≤ N − d 2 s − 2 .
On the other hand, from Lemma 4.9 proved above, we know that the function ϕ ( ρ ) \phi(\rho) ϕ ( ρ ) that upper bounds the local Rademacher complexity of the Deep Neural Network space is of the same magnitude as S 3 L ρ n log ( B W n ) \sqrt{\frac{S^3L\rho}{n}}\log(BWn) n S 3 L ρ log ( B W n ) . By plugging in the magnitudes of L , W , S , B L,W,S,B L , W , S , B , we can determine the critical radius r ∗ r^* r ∗ :
r ∗ 3 L S n log ( B W n ) ≃ r ∗ N n ( 2 log N + log n ) ≃ r ∗ ⇒ r ∗ ≃ N ( log N + log n ) n . \sqrt{\frac{r^*3LS}{n}\log(BWn)}\simeq\sqrt{\frac{r^*N}{n}(2\log N+\log n)}\simeq r^*\Rightarrow r^*\simeq\frac{N(\log N+\log n)}{n}. n r ∗ 3 L S log ( B W n ) ≃ n r ∗ N ( 2 log N + log n ) ≃ r ∗ ⇒ r ∗ ≃ n N ( log N + log n ) .
Combining the two bounds above with Theorem 4.11 yields that with high probability, we have:
∥ u ^ D R M D N N − u ∗ ∥ H 1 2 ≲ Δ E a p p + r ^ ≲ N − 2 ( s − 1 ) d + N ( log N + log n ) n . \left\|\hat{u}^{DNN}_{DRM}-u^*\right\|_{H^1}^2\lesssim\Delta\mathcal{E}_{app}+\hat{r}\lesssim N^{-\frac{2(s-1)}{d}}+\frac{N(\log N+\log n)}{n}. u ^ D R M D N N − u ∗ H 1 2 ≲ Δ E a pp + r ^ ≲ N − d 2 ( s − 1 ) + n N ( log N + log n ) .
By equating the two terms above, we can solve for the optimal N N N that yields the desired bound:
N − 2 ( s − 1 ) d ≃ N n ⇒ N ≃ n d d + 2 s − 2 . N^{-\frac{2(s-1)}{d}}\simeq\frac{N}{n}\Rightarrow N\simeq n^{\frac{d}{d+2s-2}}. N − d 2 ( s − 1 ) ≃ n N ⇒ N ≃ n d + 2 s − 2 d .
Plugging in the optimal N N N gives us the magnitudes of the four parameters L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( n d d + 2 s − 2 ) W=O(n^{\frac{d}{d+2s-2}}) W = O ( n d + 2 s − 2 d ) , S = O ( n d d + 2 s − 2 ) S=O(n^{\frac{d}{d+2s-2}}) S = O ( n d + 2 s − 2 d ) , B = O ( n d d + 2 s − 2 ) B=O(n^{\frac{d}{d+2s-2}}) B = O ( n d + 2 s − 2 d ) , as well as the final rate:
∥ u ^ D R M D N N − u ∗ ∥ H 1 2 ≲ N − 2 ( s − 1 ) d + N log N n ≲ n − 2 ( s − 1 ) d + 2 ( s − 1 ) log n . \left\|\hat{u}^{DNN}_{DRM}-u^*\right\|_{H^1}^2\lesssim N^{-\frac{2(s-1)}{d}}+\frac{N\log N}{n}\lesssim n^{-\frac{2(s-1)}{d+2(s-1)}}\log n. u ^ D R M D N N − u ∗ H 1 2 ≲ N − d 2 ( s − 1 ) + n N log N ≲ n − d + 2 ( s − 1 ) 2 ( s − 1 ) log n .
Theorem 4.13. (Final Upper Bound of DRM with Truncated Fourier Series Estimator ) With proper assumptions, consider the Deep Ritz objective with a plug in Fourier Series estimator u ^ D R M F o u r i e r = min u ∈ F ξ ( Ω ) E n D R M ( u ) \hat{u}^{Fourier}_{DRM}=\min_{u\in\mathcal{F}_{\xi}(\Omega)}\mathcal{E}^{DRM}_{n}(u) u ^ D R M F o u r i er = min u ∈ F ξ ( Ω ) E n D R M ( u ) with ξ = Θ ( n 1 d + 2 s − 2 ) \xi=\Theta(n^{\frac{1}{d+2s-2}}) ξ = Θ ( n d + 2 s − 2 1 ) , then with high probability we have
∥ u ^ D R M F o u r i e r − u ∗ ∥ H 1 2 ≲ n − 2 s − 2 d + 2 s − 2 . \left\|\hat{u}^{Fourier}_{DRM}-u^*\right\|_{H^1}^2\lesssim n^{-\frac{2s-2}{d+2s-2}}. u ^ D R M F o u r i er − u ∗ H 1 2 ≲ n − d + 2 s − 2 2 s − 2 .
Proof. On the one hand, from Lemma 4.6 and Lemma 4.7 proved above, we know that the function ϕ ( ρ ) \phi(\rho) ϕ ( ρ ) that upper bounds the local Rademacher complexity for Truncated Fourier Series can be dominated by the term
R n ( S ρ ( Ω ) ) ≲ R n ( { u − u ∗ : u ∈ F ρ , ξ ( Ω ) , ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) + R n ( { ∥ ∇ u − ∇ u ∗ ∥ : u ∈ F ρ , ξ ( Ω ) , ∥ ∇ u − ∇ u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) ≲ E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ( u ( X i ) − u ∗ ( X i ) ) ] ∥ u − Π ξ u ∗ ∥ H 1 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ∥ ∇ u ( X i ) − ∇ u ∗ ( X i ) ∥ ] ∥ u − Π ξ u ∗ ∥ H 1 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ 1 n ∑ i = 1 n σ i ∥ Π > ξ u ∗ ( X i ) ∥ ] ] + E X [ E σ [ 1 n ∑ i = 1 n σ i Π > ξ u ∗ ( X i ) ] ] ≲ E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i f ( X i ) ] ∥ f ∥ H 1 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ∥ ∇ f ( X i ) ∥ ] ∥ f ∥ H 1 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ 1 n ∑ i = 1 n σ i ∥ ∇ Π ξ u ∗ ( X i ) ∥ ] ] + E X [ E σ [ 1 n ∑ i = 1 n σ i Π > ξ u ∗ ( X i ) ] ] ≤ ρ n ξ d + ∥ Π ξ f ∥ H 1 2 n ≤ ρ n ξ d + ξ − 2 ( s − 1 ) n ≲ ρ n ξ d + 1 n + ξ − 2 ( s − 1 ) , (31) \begin{aligned}
R_n(\mathcal{S}_{\rho}(\Omega)) &\lesssim R_n\left(\left\{u-u^{*}:u\in\mathcal{F}_{\rho,\xi}(\Omega),\|u-u^{*}\|_{L_2(\Omega)}\le\sqrt{\rho}\right\}\right)\\
&\quad+R_n\left(\left\{\|\nabla u-\nabla u^{*}\|:u\in\mathcal{F}_{\rho,\xi}(\Omega),\|\nabla u-\nabla u^{*}\|_{L_2(\Omega)}\le\sqrt{\rho}\right\}\right)\\
&\lesssim\mathbb{E}_X\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i(u(X_i)-u^{*}(X_i))\right]\|u-\Pi_{\xi}u^{*}\|_{H^1(\Omega)}^2\le\rho\right]\\
&\quad+\mathbb{E}_X\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla u(X_i)-\nabla u^{*}(X_i)\|\right]\|u-\Pi_{\xi}u^{*}\|_{H^1(\Omega)}^2\le\rho\right]\\
&\quad+\mathbb{E}_X\left[\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\Pi_{>\xi}u^{*}(X_i)\|\right]\right]+\mathbb{E}_X\left[\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\Pi_{>\xi}u^{*}(X_i)\right]\right]\\
&\lesssim\mathbb{E}_X\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(X_i)\right]\|f\|_{H^1(\Omega)}^2\le\rho\right]\\
&\quad+\mathbb{E}_X\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla f(X_i)\|\right]\|f\|_{H^1(\Omega)}^2\le\rho\right]\\
&\quad+\mathbb{E}_X\left[\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla\Pi_{\xi}u^{*}(X_i)\|\right]\right]+\mathbb{E}_X\left[\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\Pi_{>\xi}u^{*}(X_i)\right]\right]\\
&\le\sqrt{\frac{\rho}{n}\xi^d}+\sqrt{\frac{\|\Pi_{\xi}f\|_{H^1}^2}{n}}\le\sqrt{\frac{\rho}{n}\xi^d}+\sqrt{\frac{\xi^{-2(s-1)}}{n}}\lesssim\sqrt{\frac{\rho}{n}\xi^d}+\frac{1}{n}+\xi^{-2(s-1)}, \tag*{(31)}
\end{aligned} R n ( S ρ ( Ω )) ≲ R n ( { u − u ∗ : u ∈ F ρ , ξ ( Ω ) , ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) + R n ( { ∥∇ u − ∇ u ∗ ∥ : u ∈ F ρ , ξ ( Ω ) , ∥∇ u − ∇ u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) ≲ E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ( u ( X i ) − u ∗ ( X i )) ] ∥ u − Π ξ u ∗ ∥ H 1 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ∥∇ u ( X i ) − ∇ u ∗ ( X i ) ∥ ] ∥ u − Π ξ u ∗ ∥ H 1 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ n 1 i = 1 ∑ n σ i ∥ Π > ξ u ∗ ( X i ) ∥ ] ] + E X [ E σ [ n 1 i = 1 ∑ n σ i Π > ξ u ∗ ( X i ) ] ] ≲ E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i f ( X i ) ] ∥ f ∥ H 1 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ∥∇ f ( X i ) ∥ ] ∥ f ∥ H 1 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ n 1 i = 1 ∑ n σ i ∥∇ Π ξ u ∗ ( X i ) ∥ ] ] + E X [ E σ [ n 1 i = 1 ∑ n σ i Π > ξ u ∗ ( X i ) ] ] ≤ n ρ ξ d + n ∥ Π ξ f ∥ H 1 2 ≤ n ρ ξ d + n ξ − 2 ( s − 1 ) ≲ n ρ ξ d + n 1 + ξ − 2 ( s − 1 ) , (31)
where Π ξ u : = ∑ ∥ z ∥ ∞ ≤ ξ u z φ z ( x ) \Pi_{\xi}u:=\sum_{\|z\|_{\infty}\le\xi}u_z\varphi_z(x) Π ξ u := ∑ ∥ z ∥ ∞ ≤ ξ u z φ z ( x ) is the projection to the Fourier basis whose frequency is smaller than ξ \xi ξ and Π > ξ u : = ∑ ∥ z ∥ ∞ > ξ u z φ z ( x ) \Pi_{>\xi}u:=\sum_{\|z\|_{\infty}>\xi}u_z\varphi_z(x) Π > ξ u := ∑ ∥ z ∥ ∞ > ξ u z φ z ( x ) is the projection to the Fourier basis whose frequency is larger than ξ \xi ξ . Then, the critical radius r ^ \hat r r ^ can be determined as follows:
ρ n ξ d + 1 n + ξ − 2 ( s − 1 ) ≃ ρ ⇒ r ^ ≃ ξ d n + 1 n + ξ − 2 ( s − 1 ) . \sqrt{\frac{\rho}{n}\xi^d+\frac{1}{n}+\xi^{-2(s-1)}}\simeq\rho\Rightarrow\hat r\simeq\frac{\xi^d}{n}+\frac{1}{n}+\xi^{-2(s-1)}. n ρ ξ d + n 1 + ξ − 2 ( s − 1 ) ≃ ρ ⇒ r ^ ≃ n ξ d + n 1 + ξ − 2 ( s − 1 ) .
On the other hand, by taking α = s \alpha=s α = s and β = 1 \beta=1 β = 1 in Lemma 4.1 and applying strong convexity of the DRM objective function proved in Theorem A.1 , we can upper bound the approximation error Δ E a p p \Delta\mathcal{E}_{\mathrm{app}} Δ E app as below:
Δ E a p p ≲ ξ − 2 ( s − 1 ) . \Delta\mathcal{E}_{\mathrm{app}}\lesssim\xi^{-2(s-1)}. Δ E app ≲ ξ − 2 ( s − 1 ) .
By equating the two terms above, we can solve for ξ \xi ξ that yields the desired bound:
ξ d n ≃ ξ − 2 ( s − 1 ) ⇒ ξ ≃ n 1 d + 2 s − 2 . \frac{\xi^d}{n}\simeq\xi^{-2(s-1)}\Rightarrow\xi\simeq n^{\frac{1}{d+2s-2}}. n ξ d ≃ ξ − 2 ( s − 1 ) ⇒ ξ ≃ n d + 2 s − 2 1 .
Plugging in the expression of ξ \xi ξ gives the final upper bound:
E x ∼ μ [ Δ E n ] ≲ r ^ + Δ E a p p ≲ ξ d n + ξ − 2 ( s − 1 ) ∼ n − 2 s − 2 d + 2 s − 2 . \mathbb{E}_{x\sim\mu}[\Delta\mathcal{E}_n] \lesssim\hat{r}+\Delta\mathcal{E}_{\mathrm{app}} \lesssim\frac{\xi^d}{n}+\xi^{-2(s-1)}\sim n^{-\frac{2s-2}{d+2s-2}}. E x ∼ μ [ Δ E n ] ≲ r ^ + Δ E app ≲ n ξ d + ξ − 2 ( s − 1 ) ∼ n − d + 2 s − 2 2 s − 2 .
Physics Informed Neural Network. Then we aim to provide the upper bound for the Physics Informed Neural Network with a Similar meta-theorem followed by upper bounds of DNN and truncated Fourier Estimators. For simplicity, we drop the proof of the meta-theorem to appendix for all the proof follows the similar idea of the DRM one.
Theorem 4.14 (Meta-theorem for Upper Bounds of Physics Informed Neural Network). Let u ∗ ∈ H s ( Ω ) u^* \in H^s(\Omega) u ∗ ∈ H s ( Ω ) denote the true solution to the PDE model with Dirichlet boundary condition:
− Δ u + V u = f on Ω , u = 0 on ∂ Ω , (32) \begin{aligned}
-\Delta u + Vu &= f \text{ on } \Omega,\\
u &= 0 \text{ on } \partial\Omega, \tag*{(32)}
\end{aligned} − Δ u + V u u = f on Ω , = 0 on ∂ Ω , (32) where f ∈ L 2 ( Ω ) f \in L^2(\Omega) f ∈ L 2 ( Ω ) and V ∈ L ∞ ( Ω ) V \in L^\infty(\Omega) V ∈ L ∞ ( Ω ) with 0 < V min ≤ V ( x ) ≤ V max > 0 0 < V_{\min} \le V(x) \le V_{\max} > 0 0 < V m i n ≤ V ( x ) ≤ V m a x > 0 . For a fixed function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , consider the empirical loss induced by the Physics Informed Neural Network:
E n ( u ) = 1 n ∑ j = 1 n [ ∣ Ω ∣ ⋅ ( Δ u ( X j ) − V ( X j ) u ( X j ) + f ( X j ) ) 2 ] , (33) \mathcal{E}_n(u)=\frac{1}{n}\sum_{j=1}^{n}\left[|\Omega|\cdot\left(\Delta u(X_j)-V(X_j)u(X_j)+f(X_j)\right)^2\right], \tag*{(33)} E n ( u ) = n 1 j = 1 ∑ n [ ∣Ω∣ ⋅ ( Δ u ( X j ) − V ( X j ) u ( X j ) + f ( X j ) ) 2 ] , (33) where { X j } j = 1 n \{X_j\}_{j=1}^{n} { X j } j = 1 n are datapoints uniformly sampled from the domain Ω \Omega Ω . Then the Physics Informed Neural Network estimator associated with function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) is defined as the minimizer of E n ( u ) \mathcal{E}_n(u) E n ( u ) over the function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) :
u ^ P I N N = min u ∈ F ( Ω ) E n ( u ) \hat{u}_{\mathrm{PINN}}=\min_{u\in\mathcal{F}(\Omega)}\mathcal{E}_n(u) u ^ PINN = u ∈ F ( Ω ) min E n ( u ) Moreover, we assume that there exists some constant C > 0 C>0 C > 0 such that all function u u u in the function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , the real solution u ∗ u^* u ∗ and f , V f,V f , V satisfy the following two conditions.
max { sup u ∈ F ( Ω ) ∥ u ∥ L ∞ ( Ω ) , sup u ∈ F ( Ω ) ∥ ∇ u ∥ L ∞ ( Ω ) , sup u ∈ F ( Ω ) ∥ Δ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ ∇ u ∗ ∥ L ∞ ( Ω ) , ∥ Δ u ∗ ∥ L ∞ ( Ω ) , V max , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (34) \max\left\{
\sup_{u\in\mathcal{F}(\Omega)}\|u\|_{L^\infty(\Omega)},
\sup_{u\in\mathcal{F}(\Omega)}\|\nabla u\|_{L^\infty(\Omega)},
\sup_{u\in\mathcal{F}(\Omega)}\|\Delta u\|_{L^\infty(\Omega)},
\|u^*\|_{L^\infty(\Omega)},
\|\nabla u^*\|_{L^\infty(\Omega)},
\|\Delta u^*\|_{L^\infty(\Omega)},
V_{\max},
\|f\|_{L^\infty(\Omega)}
\right\}\le C. \tag*{(34)} max { u ∈ F ( Ω ) sup ∥ u ∥ L ∞ ( Ω ) , u ∈ F ( Ω ) sup ∥∇ u ∥ L ∞ ( Ω ) , u ∈ F ( Ω ) sup ∥Δ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥∇ u ∗ ∥ L ∞ ( Ω ) , ∥Δ u ∗ ∥ L ∞ ( Ω ) , V m a x , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (34) u = 0 on ∂ Ω . u=0\text{ on }\partial\Omega. u = 0 on ∂ Ω. At the the same time, for any ρ > 0 \rho>0 ρ > 0 , we assume the Rademacher complexity of a localized function space T ρ ( Ω ) : = { h : = ∣ Ω ∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] | ∥ u − u ∗ ∥ H 2 ≤ ρ } \mathcal{T}_{\rho}(\Omega):=\left\{h:=|\Omega|\cdot\left[(\Delta u-Vu+f)^2-(\Delta u^*-Vu^*+f)^2\right]\ \middle|\ \|u-u^*\|_{H^2}\le\rho\right\} T ρ ( Ω ) := { h := ∣Ω∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] ∥ u − u ∗ ∥ H 2 ≤ ρ } can be upper bounded by a sub-root function ϕ = ϕ ( ρ ) : [ 0 , ∞ ) → [ 0 , ∞ ) \phi=\phi(\rho):[0,\infty)\to[0,\infty) ϕ = ϕ ( ρ ) : [ 0 , ∞ ) → [ 0 , ∞ ) , i.e.
ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( T ρ ( Ω ) ) ≤ ϕ ( ρ ) ( ∀ ρ > 0 ) . (35) \phi(4\rho)\le2\phi(\rho)\ \text{ and }\ R_n(\mathcal{T}_{\rho}(\Omega))\le\phi(\rho)\quad(\forall\ \rho>0). \tag*{(35)} ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( T ρ ( Ω )) ≤ ϕ ( ρ ) ( ∀ ρ > 0 ) . (35)
For all constant t > 0 t > 0 t > 0 . We denote r ∗ r^* r ∗ to be the solution of the fix point equation of local Rademacher complexity r = ϕ ( r ) r = \phi(r) r = ϕ ( r ) . There exists a constant C p C_p C p such that for probability 1 − C p exp ( − t ) 1 - C_p\exp(-t) 1 − C p exp ( − t ) , we have the following upper bound for the Physics Informed Neural Network Estimator
∥ u ^ P I N N − u ∗ ∥ H 2 2 ≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \left\|\hat{u}_{PINN} - u^*\right\|_{H^2}^2 \lesssim\inf_{u_F \in\mathcal{F}(\Omega)} \left(\mathcal{E}(u_F) - \mathcal{E}(u^*)\right) + \max\left\{r^*, \frac{t}{n}\right\}. ∥ u ^ P I N N − u ∗ ∥ H 2 2 ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Then we aim to calculate the final upper bound for DNN and truncated Fourier series based PINN estimator.
Theorem 4.15. (Informal Upper Bound of PINN with Deep Neural Network Estimator) With proper assumptions, consider the sparse Deep Neural Network function space Φ ( L , W , S , B ) \Phi(L,W,S,B) Φ ( L , W , S , B ) with parameters L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( n d d + 2 s − 4 ) W = O(n^{\frac{d}{d+2s-4}}) W = O ( n d + 2 s − 4 d ) , S = O ( n d d + 2 s − 4 ) S = O(n^{\frac{d}{d+2s-4}}) S = O ( n d + 2 s − 4 d ) , B = O ( 1 ) B = O(1) B = O ( 1 ) , then the Physics Informed estimator u ^ P I N N D N N = min u ∈ Φ ( L , W , S , B ) E n P I N N ( u ) \hat{u}_{PINN}^{DNN} = \min_{u \in\Phi(L,W,S,B)} \mathcal{E}_{n}^{PINN}(u) u ^ P I N N D N N = min u ∈ Φ ( L , W , S , B ) E n P I N N ( u ) satisfies the following upper bound with high probability:
∥ u ^ P I N N D N N − u ∗ ∥ H 2 2 ≲ n − 2 s − 4 d + 2 s − 4 log n . \left\|\hat{u}_{PINN}^{DNN} - u^*\right\|_{H^2}^2 \lesssim n^{-\frac{2s-4}{d+2s-4}}\log n. u ^ P I N N D N N − u ∗ H 2 2 ≲ n − d + 2 s − 4 2 s − 4 log n .
Proof. On the one hand, by taking s = 2 s = 2 s = 2 and p = 2 p = 2 p = 2 in Theorem 4.5 proved above, we have that there exists some Deep Neural Network u D N N ∈ Φ ( L , W , S , B ) u_{DNN} \in\Phi(L,W,S,B) u D N N ∈ Φ ( L , W , S , B ) with L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( N ) W = O(N) W = O ( N ) , S = O ( N ) S = O(N) S = O ( N ) , B = O ( 1 ) B = O(1) B = O ( 1 ) , such that.
∥ u D N N − u ∗ ∥ H 2 ( Ω ) 2 ≤ N − 2 s − 4 d ∥ u ∥ H s ( Ω ) . \left\|u_{DNN} - u^*\right\|_{H^2(\Omega)}^2 \leq N^{-\frac{2s-4}{d}}\left\|u\right\|_{H^s(\Omega)}. ∥ u D N N − u ∗ ∥ H 2 ( Ω ) 2 ≤ N − d 2 s − 4 ∥ u ∥ H s ( Ω ) .
Applying strong convexity of the DRM objective function proved in § 2.1 further implies:
Δ E a p p ≲ ∥ u D N N − u ∗ ∥ H 2 ( Ω ) 2 ≤ N − 2 s − 4 d . \Delta\mathcal{E}_{app} \lesssim\left\|u_{DNN} - u^*\right\|_{H^2(\Omega)}^2 \leq N^{-\frac{2s-4}{d}}. Δ E a pp ≲ ∥ u D N N − u ∗ ∥ H 2 ( Ω ) 2 ≤ N − d 2 s − 4 .
On the other hand, from lemma 4.10 proved above, we know that the function ϕ ( ρ ) \phi(\rho) ϕ ( ρ ) that upper bounds the local Rademacher complexity of the Deep Neural Networks u D N N u_{DNN} u D N N is dominated by the term ρ 3 L S n log ( W ( B ∨ 1 ) n ) \sqrt{\frac{\rho^3LS}{n}\log(W(B \vee1)n)} n ρ 3 L S log ( W ( B ∨ 1 ) n ) . By plugging in the magnitudes of L , W , S , B L,W,S,B L , W , S , B , we can determine the critical radius r ^ \hat{r} r ^ :
ρ 3 L S n log ( W ( B ∨ 1 ) n ) ≃ ρ N n ( log N + log n ) ≃ ρ ⇒ r ^ ≃ N ( log N + log n ) n . \sqrt{\frac{\rho^3LS}{n}\log(W(B \vee1)n)} \simeq\sqrt{\frac{\rho N}{n}(\log N + \log n)} \simeq\rho\Rightarrow\hat{r} \simeq\frac{N(\log N + \log n)}{n}. n ρ 3 L S log ( W ( B ∨ 1 ) n ) ≃ n ρN ( log N + log n ) ≃ ρ ⇒ r ^ ≃ n N ( log N + log n ) .
Combining the two bounds above gives us:
E x ∼ μ [ Δ E n ] ≲ Δ E a p p + r ^ ≲ N − 2 ( s − 2 ) d + N ( log N + log n ) n . \mathbb{E}_{x\sim\mu}[\Delta\mathcal{E}_n] \lesssim\Delta\mathcal{E}_{app} + \hat{r} \lesssim N^{-\frac{2(s-2)}{d}} + \frac{N(\log N + \log n)}{n}. E x ∼ μ [ Δ E n ] ≲ Δ E a pp + r ^ ≲ N − d 2 ( s − 2 ) + n N ( log N + log n ) .
By equating the two terms above, we can solve for the optimal N N N that yields the desired bound:
N − 2 ( s − 2 ) d ≃ N n ⇒ N ≃ n d d + 2 s − 4 . N^{-\frac{2(s-2)}{d}} \simeq\frac{N}{n} \Rightarrow N \simeq n^{\frac{d}{d+2s-4}}. N − d 2 ( s − 2 ) ≃ n N ⇒ N ≃ n d + 2 s − 4 d .
Plugging in the optimal N N N gives us the magnitudes of the four parameters L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( n d d + 2 s − 4 ) W = O(n^{\frac{d}{d+2s-4}}) W = O ( n d + 2 s − 4 d ) , S = O ( n d d + 2 s − 4 ) S = O(n^{\frac{d}{d+2s-4}}) S = O ( n d + 2 s − 4 d ) , B = O ( 1 ) B = O(1) B = O ( 1 ) , as well as the final rate:
E x ∼ μ [ Δ E n ] ≲ N − 2 ( s − 2 ) d + N log N n ≲ n − 2 ( s − 2 ) d + 2 ( s − 2 ) log n . \mathbb{E}_{x\sim\mu}[\Delta\mathcal{E}_n] \lesssim N^{-\frac{2(s-2)}{d}} + \frac{N\log N}{n} \lesssim n^{-\frac{2(s-2)}{d+2(s-2)}}\log n. E x ∼ μ [ Δ E n ] ≲ N − d 2 ( s − 2 ) + n N log N ≲ n − d + 2 ( s − 2 ) 2 ( s − 2 ) log n .
Theorem 4.16. (Informal Upper Bound of PINN with Truncated Fourier Series Estimator ) With proper assumptions, consider the Physics Informed Neural Network objective with a plug-in Fourier Series estimator u ^ P I N N F o u r i e r = min u ∈ F ξ ( Ω ) E n P I N N ( u ) \hat{u}^{\mathrm{Fourier}}_{\mathrm{PINN}} = \min_{u\in\mathcal{F}_{\xi}(\Omega)} \mathcal{E}^{\mathrm{PINN}}_{n}(u) u ^ PINN Fourier = min u ∈ F ξ ( Ω ) E n PINN ( u ) with ξ = Θ ( n 1 d + 2 s − 4 ) \xi= \Theta\left(n^{\frac{1}{d+2s-4}}\right) ξ = Θ ( n d + 2 s − 4 1 ) , then with high probability we have
∥ u ^ P I N N F o u r i e r − u ∗ ∥ H 2 2 ≲ n − 2 s − 4 d + 2 s − 4 . \left\|\hat{u}^{\mathrm{Fourier}}_{\mathrm{PINN}}-u^{*}\right\|_{H^{2}}^{2} \lesssim n^{-\frac{2s-4}{d+2s-4}}. u ^ PINN Fourier − u ∗ H 2 2 ≲ n − d + 2 s − 4 2 s − 4 .
Proof. On the one hand, from Lemma 4.6 and Lemma 4.7 proved above, we know that the function ϕ ( ρ ) \phi(\rho) ϕ ( ρ ) that upper bounds the local Rademacher complexity for Truncated Fourier Series is dominated by
R n ( S ρ ( Ω ) ) ≲ R n ( { ∣ Δ u − Δ u ∗ ∣ : u ∈ F ρ , ξ ( Ω ) , ∣ Δ u − Δ u ∗ ∣ L 2 ( Ω ) ≤ ρ } ) ≲ E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ∣ Δ u ( X i ) − Δ u ∗ ( X i ) ∣ ] ∥ u − Π ξ u ∗ ∥ H 2 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ 1 n ∑ i = 1 n σ i ∣ Δ Π > ξ u ∗ ( X i ) ∣ ] ] ≲ E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ∣ Δ f ( X i ) ∣ ] ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ 1 n ∑ i = 1 n σ i ∥ ∇ Π > ξ u ∗ ( X i ) ∥ ] ] ≲ E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ∣ Δ f ( X i ) ∣ ] ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ 1 n ∑ i = 1 n σ i ∥ Δ Π > ξ u ∗ ( X i ) ∥ ] ] ≤ ρ n ξ d 2 + ∥ Π ξ f ∥ H 2 2 n ≤ ρ n ξ d 2 + ξ − 2 ( s − 1 ) n ≲ ρ n ξ d 2 + 1 n + ξ − 2 ( s − 2 ) . (36) \begin{aligned}
R_n(\mathcal{S}_{\rho}(\Omega)) &\lesssim R_n\left(\left\{|\Delta u-\Delta u^{*}|:u\in\mathcal{F}_{\rho,\xi}(\Omega),|\Delta u-\Delta u^{*}|_{L^2(\Omega)}\leq\sqrt{\rho}\right\}\right)\\
&\lesssim\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i|\Delta u(X_i)-\Delta u^{*}(X_i)|\right]\left\|u-\Pi_{\xi}u^{*}\right\|_{H^{2}(\Omega)}^{2}\leq\rho\right]\\
&\quad+\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i|\Delta\Pi_{>\xi}u^{*}(X_i)|\right]\right]\\
&\lesssim\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i|\Delta f(X_i)|\right]\|f\|_{H^{2}(\Omega)}^{2}\leq\rho\right]\\
&\quad+\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla\Pi_{>\xi}u^{*}(X_i)\|\right]\right]\\
&\lesssim\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i|\Delta f(X_i)|\right]\|f\|_{H^{2}(\Omega)}^{2}\leq\rho\right]\\
&\quad+\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\Delta\Pi_{>\xi}u^{*}(X_i)\|\right]\right]\\
&\leq\sqrt{\frac{\rho}{n}\xi^{\frac{d}{2}}}+\sqrt{\frac{\|\Pi_{\xi}f\|_{H^{2}}^{2}}{n}}\leq\sqrt{\frac{\rho}{n}\xi^{\frac{d}{2}}}+\sqrt{\frac{\xi^{-2(s-1)}}{n}}\lesssim\sqrt{\frac{\rho}{n}\xi^{\frac{d}{2}}}+\frac{1}{n}+\xi^{-2(s-2)}. \tag*{(36)}
\end{aligned} R n ( S ρ ( Ω )) ≲ R n ( { ∣Δ u − Δ u ∗ ∣ : u ∈ F ρ , ξ ( Ω ) , ∣Δ u − Δ u ∗ ∣ L 2 ( Ω ) ≤ ρ } ) ≲ E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ∣Δ u ( X i ) − Δ u ∗ ( X i ) ∣ ] ∥ u − Π ξ u ∗ ∥ H 2 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ n 1 i = 1 ∑ n σ i ∣Δ Π > ξ u ∗ ( X i ) ∣ ] ] ≲ E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ∣Δ f ( X i ) ∣ ] ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ n 1 i = 1 ∑ n σ i ∥∇ Π > ξ u ∗ ( X i ) ∥ ] ] ≲ E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ∣Δ f ( X i ) ∣ ] ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ ] + E X [ E σ [ n 1 i = 1 ∑ n σ i ∥Δ Π > ξ u ∗ ( X i ) ∥ ] ] ≤ n ρ ξ 2 d + n ∥ Π ξ f ∥ H 2 2 ≤ n ρ ξ 2 d + n ξ − 2 ( s − 1 ) ≲ n ρ ξ 2 d + n 1 + ξ − 2 ( s − 2 ) . (36)
Thus, the localization radius r ^ \hat{r} r ^ can be determined as follows:
ρ n ξ d 2 + 1 n + ξ − 2 ( s − 2 ) ≃ ρ ⇒ r ^ ≃ ξ d n + 1 n + ξ − 2 ( s − 2 ) . \sqrt{\frac{\rho}{n}\xi^{\frac{d}{2}}+\frac{1}{n}+\xi^{-2(s-2)}}\simeq\rho\Rightarrow\hat{r}\simeq\frac{\xi^{d}}{n}+\frac{1}{n}+\xi^{-2(s-2)}. n ρ ξ 2 d + n 1 + ξ − 2 ( s − 2 ) ≃ ρ ⇒ r ^ ≃ n ξ d + n 1 + ξ − 2 ( s − 2 ) .
On the other hand, by taking α = s \alpha=s α = s and β = 1 \beta=1 β = 1 in Lemma 4.1 and applying strong convexity of the DRM objective function proved in Theorem A.1 , we can upper bound the approximation error Δ E a p p \Delta\mathcal{E}_{\mathrm{app}} Δ E app as below:
Δ E a p p ≲ ξ − 2 ( s − 2 ) , \Delta\mathcal{E}_{\mathrm{app}}\lesssim\xi^{-2(s-2)}, Δ E app ≲ ξ − 2 ( s − 2 ) ,
By equating the two terms above, we can solve for ξ \xi ξ that yields the desired bound:
ξ d n ≃ ξ − 2 ( s − 2 ) ⇒ ξ ≃ n 1 d + 2 s − 4 , \frac{\xi^{d}}{n}\simeq\xi^{-2(s-2)}\Rightarrow\xi\simeq n^{\frac{1}{d+2s-4}}, n ξ d ≃ ξ − 2 ( s − 2 ) ⇒ ξ ≃ n d + 2 s − 4 1 ,
Plugging in the expression of ξ \xi ξ gives the final upper bound:
E X ∼ μ [ Δ E n ] ≲ r ^ + Δ E a p p ≲ ξ d n + ξ − 2 ( s − 2 ) ≍ n − 2 s − 4 d + 2 s − 4 . \mathbb{E}_{X\sim\mu}[\Delta\mathcal{E}_n] \lesssim\hat r+\Delta\mathcal{E}_{\mathrm{app}}\lesssim\frac{\xi^d}{n}+\xi^{-2(s-2)}\asymp n^{-\frac{2s-4}{d+2s-4}}. E X ∼ μ [ Δ E n ] ≲ r ^ + Δ E app ≲ n ξ d + ξ − 2 ( s − 2 ) ≍ n − d + 2 s − 4 2 s − 4 .
Remark.
There is a common belief that Machine learning based PDE solvers can break the curse of dimensionality [15 , 24 , 40 ] . However we obtained an n − 2 s − 2 2 s − 4 + d n^{-\frac{2s-2}{2s-4+d}} n − 2 s − 4 + d 2 s − 2 convergence rate which can become super slow in high dimension. Our analysis showed that it is essential to constrain the function space to break the curse of dimensionality. [47 ] considered the DRM in Barron spaces. [56 ] showed that functions in the Barron space enjoy a smoothness s s s at the same magnitude as d d d , which will also leads to convergence rate independent of the dimension using our upper bound. Neural network can also approximate mixed sparse grid spaces [52 , 70 ] , function on manifold [55 , 9 ] without curse of dimensionality. Combined with these approximation bounds, we can also achieve a bound that breaks the curse of dimensionality using Theorem 4.14 and 4.11. In this paper, we aim to consider the statistical power of the loss function in common function spaces and leave the curse of dimensionality as a separate topic.
Our bound is faster than the concurrent bound [14 , 34 ] for we provided a fast rate O ( 1 / n ) O(1/n) O ( 1/ n ) by utilizing the strong convexity of the objective function and improves the convergence rate from n − 2 s − 2 d + 4 s − 4 n^{-\frac{2s-2}{d+4s-4}} n − d + 4 s − 4 2 s − 2 to n − 2 s − 2 d + 2 s − 2 n^{-\frac{2s-2}{d+2s-2}} n − d + 2 s − 2 2 s − 2 for Deep Ritz and from n − 2 s − 4 d + 4 s − 8 n^{-\frac{2s-4}{d+4s-8}} n − d + 4 s − 8 2 s − 4 to n − 2 s − 4 d + 2 s − 4 n^{-\frac{2s-4}{d+2s-4}} n − d + 2 s − 4 2 s − 4 for PINN. Comparing to the lower bound provided in § 3 , we show that our bound for PINN is near optimal and we’ll let our bound for DRM become near optimal in the next section.
For upper bound of DRM, due to a technical issue, we assume the observation we access is clean, i.e f i = f ( X i ) f_i=f(X_i) f i = f ( X i ) . We conjecture that add noising on observation will not effect the rate and leave this to future work.
Modified Deep Ritz Methods
Comparing the lower bound in § 3 and the upper bound in § 4 , we find out that the Physics Informed Neural Network achieved min-max optimality while the Deep Ritz Method does not. In this section, we proposed a modified version of deep Ritz which can be statistically optimal.
As discussed in Appendix § 11 , the reason behind the suboptimality of DRM comes from the high complexity introduced via the uniform concentration bound of the gradient term in the variational form. At the same time, we further observed that the ∫ ∥ ∇ u ∥ 2 d x \int\lVert\nabla u\rVert^2dx ∫ ∥ ∇ u ∥ 2 d x does not require any query from the right hand side function f f f , which means that we can easily make another splitted sample to approximate the ∫ ∥ ∇ u ∥ 2 d x \int\lVert\nabla u\rVert^2dx ∫ ∥ ∇ u ∥ 2 d x term more precisely.
E N , n ( u ) = 1 N ∑ j = 1 N [ ∣ Ω ∣ ⋅ 1 2 ∥ ∇ u ( X j ′ ) ∥ 2 ] + 1 n ∑ j = 1 n [ ∣ Ω ∣ ⋅ ( 1 2 ∇ u ( X j ) ∣ u ( X j ) ∣ 2 − f j u ( X j ) ) ] (37) \mathcal{E}_{N,n}(u)=\frac{1}{N}\sum_{j=1}^{N}\left[|\Omega|\cdot\frac{1}{2}\lVert\nabla u(X'_j)\rVert^2\right]+\frac{1}{n}\sum_{j=1}^{n}\left[|\Omega|\cdot\left(\frac{1}{2}\nabla u(X_j)|u(X_j)|^2-f_j u(X_j)\right)\right] \tag*{(37)} E N , n ( u ) = N 1 j = 1 ∑ N [ ∣Ω∣ ⋅ 2 1 ∥ ∇ u ( X j ′ ) ∥ 2 ] + n 1 j = 1 ∑ n [ ∣Ω∣ ⋅ ( 2 1 ∇ u ( X j ) ∣ u ( X j ) ∣ 2 − f j u ( X j ) ) ] (37)
Once we sampled more data for approximating ∫ ∥ ∇ u ∥ 2 d x \int\lVert\nabla u\rVert^2dx ∫ ∥ ∇ u ∥ 2 d x , we can achieve an near optimal bound for the Truncated Fourier Estimator when N n ≳ n 2 s − 4 d + 2 s − 4 \frac{N}{n}\gtrsim n^{\frac{2s-4}{d+2s-4}} n N ≳ n d + 2 s − 4 2 s − 4 . The proof is based on a similar meta-theorem as following.
Theorem 5.1 (Meta-theorem for Upper Bounds of Modified Deep Ritz Method). Let u ∗ ∈ H s ( Ω ) u^* \in H^s(\Omega) u ∗ ∈ H s ( Ω ) denote the true solution to the PDE model with Dirichlet boundary condition:
− Δ u + V u = f on Ω , u = 0 on ∂ Ω , (38) \begin{aligned}
-\Delta u + V u &= f \text{ on } \Omega,\\
u &= 0 \text{ on } \partial\Omega, \tag*{(38)}
\end{aligned} − Δ u + V u u = f on Ω , = 0 on ∂ Ω , (38) where f ∈ L 2 ( Ω ) f \in L^2(\Omega) f ∈ L 2 ( Ω ) and V ∈ L ∞ ( Ω ) V \in L^\infty(\Omega) V ∈ L ∞ ( Ω ) with 0 < V min ≤ V ( x ) ≤ V max > 0 0 < V_{\min} \le V(x) \le V_{\max} > 0 0 < V m i n ≤ V ( x ) ≤ V m a x > 0 . For a fixed function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , consider the empirical loss induced by the Modified Deep Ritz Method (N ≥ n N \ge n N ≥ n ):
E N , n ( u ) = 1 N ∑ i = 1 N [ ∣ Ω ∣ ⋅ 1 2 ∣ ∇ u ( X i ′ ) ∣ 2 ] + 1 n ∑ j = 1 n [ ∣ Ω ∣ ⋅ ( 1 2 V ( X j ) ∣ u ( X j ) ∣ 2 − f ( X j ) u ( X j ) ) ] . (39) \mathcal{E}_{N,n}(u) = \frac{1}{N}\sum_{i=1}^{N}\left[|\Omega|\cdot\frac{1}{2}|\nabla u(X_i')|^2\right] + \frac{1}{n}\sum_{j=1}^{n}\left[|\Omega|\cdot\left(\frac{1}{2}V(X_j)|u(X_j)|^2-f(X_j)u(X_j)\right)\right]. \tag*{(39)} E N , n ( u ) = N 1 i = 1 ∑ N [ ∣Ω∣ ⋅ 2 1 ∣∇ u ( X i ′ ) ∣ 2 ] + n 1 j = 1 ∑ n [ ∣Ω∣ ⋅ ( 2 1 V ( X j ) ∣ u ( X j ) ∣ 2 − f ( X j ) u ( X j ) ) ] . (39) where { X i ′ } i = 1 N \{X_i'\}_{i=1}^{N} { X i ′ } i = 1 N and { X j } j = 1 n \{X_j\}_{j=1}^{n} { X j } j = 1 n are datapoints uniformly and independently sampled from the domain Ω \Omega Ω . Then the Modified Deep Ritz estimator associated with function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) is defined as the minimizer of E N , n ( u ) \mathcal{E}_{N,n}(u) E N , n ( u ) over the function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) :
u ^ M D R M = min u ∈ F ( Ω ) E N , n ( u ) \widehat{u}_{\mathrm{MDRM}}=\min_{u\in\mathcal{F}(\Omega)}\mathcal{E}_{N,n}(u) u MDRM = u ∈ F ( Ω ) min E N , n ( u ) Moreover, we assume that there exists some constant C > 0 C>0 C > 0 such that all function u u u in the function space F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , the real solution u ∗ u^* u ∗ and f , V f,V f , V satisfy the following two conditions.
max { sup u ∈ F ( Ω ) ∥ u ∥ L ∞ ( Ω ) , sup u ∈ F ( Ω ) ∥ ∇ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ ∇ u ∗ ∥ L ∞ ( Ω ) , V max , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (40) \max\left\{\sup_{u\in\mathcal{F}(\Omega)}\|u\|_{L^\infty(\Omega)},\sup_{u\in\mathcal{F}(\Omega)}\|\nabla u\|_{L^\infty(\Omega)},\|u^*\|_{L^\infty(\Omega)},\|\nabla u^*\|_{L^\infty(\Omega)},V_{\max},\|f\|_{L^\infty(\Omega)}\right\}\le C. \tag*{(40)} max { u ∈ F ( Ω ) sup ∥ u ∥ L ∞ ( Ω ) , u ∈ F ( Ω ) sup ∥∇ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥∇ u ∗ ∥ L ∞ ( Ω ) , V m a x , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (40) u = 0 on ∂ Ω . u=0\text{ on }\partial\Omega. u = 0 on ∂ Ω. At the the same time, for any ρ > 0 \rho>0 ρ > 0 , we assume the Rademacher complexity of two localized function spaces
S ρ ( Ω ) : = { ( h 1 , h 2 ) ∣ h 1 : = ∣ Ω ∣ ⋅ [ 1 2 ( ∣ ∇ u ∣ 2 − ∣ ∇ u ∗ ∣ 2 ) ] , h 2 : = ∣ Ω ∣ ⋅ [ 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] , ∥ u − u ∗ ∥ H 1 ≤ ρ } S_\rho(\Omega):=\left\{(h_1,h_2)\mid h_1:=|\Omega|\cdot\left[\frac{1}{2}\left(|\nabla u|^2-|\nabla u^*|^2\right)\right],\ h_2:=|\Omega|\cdot\left[\frac{1}{2}V\left(|u|^2-|u^*|^2\right)-f\left(u-u^*\right)\right],\ \|u-u^*\|_{H^1}\le\rho\right\} S ρ ( Ω ) := { ( h 1 , h 2 ) ∣ h 1 := ∣Ω∣ ⋅ [ 2 1 ( ∣∇ u ∣ 2 − ∣∇ u ∗ ∣ 2 ) ] , h 2 := ∣Ω∣ ⋅ [ 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] , ∥ u − u ∗ ∥ H 1 ≤ ρ } can be upper bounded by a sub-root function ϕ = ϕ ( ρ ) : [ 0 , ∞ ) → [ 0 , ∞ ) \phi=\phi(\rho):[0,\infty)\to[0,\infty) ϕ = ϕ ( ρ ) : [ 0 , ∞ ) → [ 0 , ∞ ) , i.e.
ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R N , n ( S ρ ( Ω ) ) ≤ ϕ ( ρ ) ( ∀ ρ > 0 ) , (41) \phi(4\rho)\le2\phi(\rho)\ \text{ and }\ R_{N,n}(S_\rho(\Omega))\le\phi(\rho)\quad(\forall\ \rho>0), \tag*{(41)} ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R N , n ( S ρ ( Ω )) ≤ ϕ ( ρ ) ( ∀ ρ > 0 ) , (41) where R N , n ( S ) : = R N ( { h 1 ∣ ( h 1 , h 2 ) ∈ S } ) + R n ( { h 2 ∣ ( h 1 , h 2 ) ∈ S } ) R_{N,n}(S):=R_N(\{h_1\mid(h_1,h_2)\in S\})+R_n(\{h_2\mid(h_1,h_2)\in S\}) R N , n ( S ) := R N ({ h 1 ∣ ( h 1 , h 2 ) ∈ S }) + R n ({ h 2 ∣ ( h 1 , h 2 ) ∈ S }) . For all constant t > 0 t>0 t > 0 . We denote r ∗ r^* r ∗ to be the solution of the fix point equation of local Rademacher complexity r = ϕ ( r ) r=\phi(r) r = ϕ ( r ) . There exist a constant C p C_p C p such that for probability 1 − C p exp ( − t ) 1-C_p\exp(-t) 1 − C p exp ( − t ) , we have the following upper bound for the Modified Deep Ritz Estimator
∥ u ^ M D R M − u ∗ ∥ H 1 2 ≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \left\|\widehat{u}_{\mathrm{MDRM}}-u^*\right\|_{H^1}^2\lesssim\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right)+\max\left\{r^*,\frac{t}{n}\right\}. ∥ u MDRM − u ∗ ∥ H 1 2 ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Theorem 5.2. (Informal Upper Bound of DRM with Truncated Fourier Series Estimator ) With proper assumptions, consider the Deep Ritz objective with a plug in Fourier Series estimator u ^ D R M F o u r i e r = min u ∈ F ξ ( Ω ) E n D R M ( u ) \widehat{u}_{DRM}^{\mathrm{Fourier}} = \min_{u\in\mathcal{F}_{\xi}(\Omega)} \mathcal{E}_{n}^{DRM}(u) u D R M Fourier = min u ∈ F ξ ( Ω ) E n D R M ( u ) with ξ = Θ ( n 1 d + 2 s − 4 ) \xi= \Theta\left(n^{\frac{1}{d+2s-4}}\right) ξ = Θ ( n d + 2 s − 4 1 ) and N n ≳ n 2 d + 2 s − 4 \frac{N}{n} \gtrsim n^{\frac{2}{d+2s-4}} n N ≳ n d + 2 s − 4 2 , then we have
∥ u ^ D R M F o u r i e r − u ∗ ∥ H 1 2 ≲ n − 2 s − 2 d + 2 s − 4 . \left\|\widehat{u}_{DRM}^{\mathrm{Fourier}}-u^{*}\right\|_{H^{1}}^{2} \lesssim n^{-\frac{2s-2}{d+2s-4}}. u D R M Fourier − u ∗ H 1 2 ≲ n − d + 2 s − 4 2 s − 2 .
Proof. On the one hand, from Lemma 4.6 and Lemma 4.7 proved above, we know that the function ϕ ( ρ ) \phi(\rho) ϕ ( ρ ) that upper bounds the local Rademacher complexity is dominated by the term ρ n ξ d 2 \sqrt{\frac{\rho}{n}\xi^{\frac{d}{2}}} n ρ ξ 2 d for Truncated Fourier Series in F ξ ( Ω ) \mathcal{F}_{\xi}(\Omega) F ξ ( Ω ) . Following the same proof as shown for DRM upper bound, the localization radius r ^ \widehat{r} r can be determined as follows:
ρ n ξ d − 2 2 + ρ N ξ d 2 + 1 n + 1 N + ξ − 2 ( s − 1 ) ≃ ρ . \sqrt{\frac{\rho}{n}}\xi^{\frac{d-2}{2}}+\sqrt{\frac{\rho}{N}}\xi^{\frac{d}{2}}+\frac{1}{n}+\frac{1}{N}+\xi^{-2(s-1)}\simeq\rho. n ρ ξ 2 d − 2 + N ρ ξ 2 d + n 1 + N 1 + ξ − 2 ( s − 1 ) ≃ ρ .
For we have assumed ξ d N < ξ d − 2 n \frac{\xi^{d}}{N}<\frac{\xi^{d-2}}{n} N ξ d < n ξ d − 2 , the solution of the fixed point equation is r ^ ≃ ξ d n \widehat{r}\simeq\frac{\xi^{d}}{n} r ≃ n ξ d . On the other hand, by taking α = s \alpha=s α = s and β = 1 \beta=1 β = 1 in Lemma 4.1 and applying strong convexity of the DRM objective function proved in Theorem A.1 , we can upper bound the approximation error Δ E a p p \Delta\mathcal{E}_{\mathrm{app}} Δ E app as below:
Δ E a p p ≲ ξ − 2 ( s − 1 ) . \Delta\mathcal{E}_{\mathrm{app}}\lesssim\xi^{-2(s-1)}. Δ E app ≲ ξ − 2 ( s − 1 ) .
By equating the two terms above, we can solve for ξ \xi ξ that yields the desired bound:
ξ d − 2 n + ξ − 2 ( s − 1 ) ≃ ξ − 2 ( s − 1 ) ⇒ ξ ≃ n 1 d + 2 s − 4 + ξ − 2 ( s − 1 ) , \frac{\xi^{d-2}}{n}+\xi^{-2(s-1)}\simeq\xi^{-2(s-1)}\Rightarrow\xi\simeq n^{\frac{1}{d+2s-4}}+\xi^{-2(s-1)}, n ξ d − 2 + ξ − 2 ( s − 1 ) ≃ ξ − 2 ( s − 1 ) ⇒ ξ ≃ n d + 2 s − 4 1 + ξ − 2 ( s − 1 ) ,
Plugging in the expression of ξ \xi ξ gives the final upper bound:
E x ∼ μ [ Δ E n ] ≲ r ^ + Δ E a p p ≲ ξ d − 2 n + ξ − 2 ( s − 2 ) ≃ n − 2 s − 2 d + 2 s − 4 . \mathbb{E}_{x\sim\mu}[\Delta\mathcal{E}_{n}]\lesssim\widehat{r}+\Delta\mathcal{E}_{\mathrm{app}}\lesssim\frac{\xi^{d-2}}{n}+\xi^{-2(s-2)}\simeq n^{-\frac{2s-2}{d+2s-4}}. E x ∼ μ [ Δ E n ] ≲ r + Δ E app ≲ n ξ d − 2 + ξ − 2 ( s − 2 ) ≃ n − d + 2 s − 4 2 s − 2 .
Remark. We still cannot achieve optimal rate for neural network even with modified DRM methods. The reason is because the number of neuron is not a good complexity measure for the gradient of the function thus the bound for ψ ( r ) = R n ( { I ( u ) ∣ ∥ u − u ∗ ∥ H 1 2 ≤ r } ) \psi(r)=R_{n}(\{\mathcal{I}(u)\mid\|u-u^{*}\|_{H^{1}}^{2}\leq r\}) ψ ( r ) = R n ({ I ( u ) ∣ ∥ u − u ∗ ∥ H 1 2 ≤ r }) is not enough for achieving optimal convergence rate. However, following [63 , 70 , 33 , 9 , 18 ] using deep networks for estimating functions, we optimize the best neural network with constrained sparsity in our paper. Here we conjecture that there exists a computable complexity measure can makes DRM statistical optimal and leave finding the right complexity of the neural network’s gradient to be future work.
Experiments
In this section, we conduct several numerical experiments to verify our theory. We follow the neural network and hyper-parameter setting in [7 ] . Due to the page limit, we only put the experiments for Deep Ritz Methods here.
The Modified Deep Ritz Methods
In this section, we conduct experiments which substantiate our theoretical results for modified Deep Ritz methods. For simplicity, we take V ( x ) = 1 V(x)=1 V ( x ) = 1 in our experiment. We conduct experiment in 2-dimension and select the solution of the PDE as u ∗ = ∑ z ∥ z ∥ − s ϕ z ( x ) ∈ H s u^*=\sum_z \lVert z\rVert^{-s}\phi_z(x)\in H^s u ∗ = ∑ z ∥ z ∥ − s ϕ z ( x ) ∈ H s . We show the log-log plot of the H 1 H^1 H 1 loss against the number of sampled data for s = 4 s=4 s = 4 in Figure 1 . We use an OLS estimator to fit the log-log plot and put the estimated slope and corresponding R 2 R^2 R 2 score in Figure 1 . As our theory predicts, the modified Deep Ritz Method converges faster than the original one. All the derivation of the two estimators is listed in Appendix § 11 .
Figure 1. The Log-Log plot and estimated convergence slope for Modified DRM and DRM using Fourier basis, showing the median error over 5 replicates.
Dimension Dependent Scaling Law.
We conduct experiments to illustrate that the population loss of well-trained and well-tuned Deep Ritz method will scale with the d d d -dimensional training data number N N N as a power-law L ∝ 1 N α \mathcal{L}\propto\frac{1}{N^\alpha} L ∝ N α 1 . We also scan over a range of d d d and α \alpha α and verify an approximately α ∝ 1 d \alpha\propto\frac{1}{d} α ∝ d 1 scaling law as our theory suggests. We use the same test function in § 6.1 as the solution of our PDE. For simplicity, we take V ( x ) = 1 V(x)=1 V ( x ) = 1 in our experiment. We train the deep Ritz method on 20, 80, 320, 1280, 10240 sampled data points for 5,6,7,8,9,10 dimensional problems and we plot our results on the log-log scale. Results are shown in Figure 2 . We discover the L ∝ n 1 d + 2 \mathcal{L}\propto n^{\frac{1}{d+2}} L ∝ n d + 2 1 scaling law in practical situations.
Figure 2. We verify the dimension dependent scaling law empirically. The multiplicative inverse of the scaling law coefficient is highly linear with the dimension d d d , showing the mean error over 2 replicates.
Adaptation To The Simpler Functions.
[65 ] showed that the neural scaling law will adapt to the structure that the target function enjoys. This adaptivity enables the neural network to break the cure of the dimensionality for simple functions in high dimension. [72 , 8 ] also observed this theoretically. For solving PDEs, we also observed this adaptivity in practice. Here we tested the following two hypothesis
observed a scaling law at the parametric rate. Specifically, we obtained a linear estimate with slope α = − 0.50679429 \alpha= -0.50679429 α = − 0.50679429 and a R 2 R^2 R 2 score = 0.96 in the log-log plot. See Figure 3 (a).
Figure 3. Neural network have the ability to adapt to simple functions and achieves convergence without curse of dimensionality, showing the median error over 5 replicates.
Simple Polynomials. Neural network can approximate simple polynomials exponentially fast [76 ] . Thus, we select the ground truth solution to be the following simple polynomial in 10 dimensional spaces u ∗ ( x ) = x 1 x 2 + ⋯ + x 9 x 10 u^*(x) = x_1x_2 + \cdots+ x_9x_{10} u ∗ ( x ) = x 1 x 2 + ⋯ + x 9 x 10 . In this example, we obtained a linear estimate with slope α = − 0.49755418 \alpha= -0.49755418 α = − 0.49755418 and a R 2 R^2 R 2 score = 0.99 in the log-log plot. See Figure 3 (b).
Conclusion and Discussion
Conclusion. In this paper, We considered the statistical min-max optimality of solving a PDE from random samples. We improved the previous bounds [78 , 47 , 14 , 34 ] by providing the first fast rate generalization bound for learning PDE solutions via the strongly convex nature of the two objective functions. We achieved the optimal rate via the PINN and a modified Deep Ritz method. We verified our theory via numerical experiments and explored the dimension dependent scaling laws of Deep PDE solvers.
Discussion and Future Work. Here we discuss several drawbacks of our theory
We restricted our target function and estimators in W 1 , ∞ W^{1,\infty} W 1 , ∞ instead of H 1 H^1 H 1 due to boundedness assumption made in the local Rademacher complexity arguments. However, typical functional used in physics is always unbounded, such as the Newtonian potential 1 ∥ x − y ∥ d − 2 \frac{1}{\lVert x-y\rVert^{d-2}} ∥ x − y ∥ d − 2 1 , which limits the application of our theory.
This paper did not discuss any optimization aspect of the deep PDE solvers and always assumed achieves global optimum. However, it is important to investigate whether the optimization error [71 , 11 ] will finally dominate.
Instead of solving a single PDE, recent works [45 , 44 , 42 , 40 , 4 , 17 , 19 ] considered the so-called "operator learning", which aims to learn a family of PDE/inverse problems using a single network. It is interesting to investigate the generalization bound and neural scaling law there.
We find out that the sparsity of the neural network is not a good complexity measure of neural network’s gradient. We conjecture that there exists an oracle complexity measure, whose approximation and generalization bounds can lead Modified DRM to achieve the optimal convergence rate.
Acknowledgments
Yiping Lu is supported by the Stanford Interdisciplinary Graduate Fellowship (SIGF). Jianfeng Lu is supported in part by National Science Foundation via grants DMS-2012286 and CCF1934964. Lexing Ying is supported by National Science Foundation under award DMS-2011699. Jose Blanchet is supported in part by the Air Force Office of Scientific Research under award number FA9550-20-1-0397 and NSF grants 1915967, 1820942, 1838576. Yiping Lu also thanks Taiji Suzuki, Atsushi Nitanda, Yifan Chen, Junbin Huang, Wenlong Ji, Greg Yang, Yufan Chen, Zong Shang, Denny Wu, Jikai Hou, Jun Hu, Fang Yao and Bin Dong for helpful comments and feedback.
Proof of the Upper Bounds
Notations
In this section, we provide all the notations we need in the proof. Let Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d be some open set. We denote C ( Ω ) C(\Omega) C ( Ω ) the space of continuous functions on Ω \Omega Ω and C k ( Ω ) C^{k}(\Omega) C k ( Ω ) the space of all functions that are k k k times continuously differentiable on Ω \Omega Ω (∀ k ∈ Z + \forall k \in\mathbb{Z}^{+} ∀ k ∈ Z + ). For any n ∈ N 0 n \in\mathbb{N}_{0} n ∈ N 0 (N 0 : = Z + ∪ { 0 } \mathbb{N}_{0} := \mathbb{Z}^{+} \cup\{0\} N 0 := Z + ∪ { 0 } is the set of all non-negative integers) and 1 ≤ p ≤ ∞ 1 \le p \le\infty 1 ≤ p ≤ ∞ , we define the Sobolev space W n , p ( Ω ) W^{n,p}(\Omega) W n , p ( Ω ) by
W n , p ( Ω ) : = { f ∈ L p ( Ω ) : D α f ∈ L p ( Ω ) , ∀ α ∈ N 0 d with ∣ α ∣ ≤ n } . W^{n,p}(\Omega) := \{f \in L^{p}(\Omega) : D^{\alpha}f \in L^{p}(\Omega),\ \forall\alpha\in\mathbb{N}_{0}^{d}\ \text{with } |\alpha| \le n\}. W n , p ( Ω ) := { f ∈ L p ( Ω ) : D α f ∈ L p ( Ω ) , ∀ α ∈ N 0 d with ∣ α ∣ ≤ n } .
In particular, when p = 2 p = 2 p = 2 , we define H n ( Ω ) : = W n , 2 ( Ω ) H^{n}(\Omega) := W^{n,2}(\Omega) H n ( Ω ) := W n , 2 ( Ω ) for any n ∈ N 0 n \in\mathbb{N}_{0} n ∈ N 0 . Moreover, for any f ∈ W n , p ( Ω ) f \in W^{n,p}(\Omega) f ∈ W n , p ( Ω ) with 1 ≤ p < ∞ 1 \le p < \infty 1 ≤ p < ∞ , we define the Sobolev norm by:
∥ f ∥ W n , p ( Ω ) : = ( ∑ 0 ≤ ∣ α ∣ ≤ n ∥ D α f ∥ L p ( Ω ) p ) 1 p . \|f\|_{W^{n,p}(\Omega)} := \left(\sum_{0 \le|\alpha| \le n}\|D^{\alpha}f\|_{L^{p}(\Omega)}^{p}\right)^{\frac{1}{p}}. ∥ f ∥ W n , p ( Ω ) := 0 ≤ ∣ α ∣ ≤ n ∑ ∥ D α f ∥ L p ( Ω ) p p 1 .
In particular, when p = ∞ p = \infty p = ∞ , we have:
∥ f ∥ W n , ∞ ( Ω ) : = max 0 ≤ ∣ α ∣ ≤ n ∥ D α f ∥ L ∞ ( Ω ) . \|f\|_{W^{n,\infty}(\Omega)} := \max_{0 \le|\alpha| \le n}\|D^{\alpha}f\|_{L^{\infty}(\Omega)}. ∥ f ∥ W n , ∞ ( Ω ) := 0 ≤ ∣ α ∣ ≤ n max ∥ D α f ∥ L ∞ ( Ω ) .
Consider the Fourier expansion f : = ∑ z ∈ N d f z ϕ z ( x ) f := \sum_{z \in\mathbb{N}^{d}} f_{z}\phi_{z}(x) f := ∑ z ∈ N d f z ϕ z ( x ) of the function f ∈ W n , p ( Ω ) f \in W^{n,p}(\Omega) f ∈ W n , p ( Ω ) . We can equivalently express the Sobolev norm as:
∥ f ∥ W n , p ( Ω ) = ( ∑ z ∥ z ∥ n p ∣ f z ∣ p ) 1 / p . \|f\|_{W^{n,p}(\Omega)} = \left(\sum_{z}\|z\|^{np}|f_{z}|^{p}\right)^{1/p}. ∥ f ∥ W n , p ( Ω ) = ( z ∑ ∥ z ∥ n p ∣ f z ∣ p ) 1/ p .
where f z = ∫ Ω f ( x ) ϕ z ( x ) ‾ d x = ∫ Ω f ( x ) e − 2 π i ⟨ z , x ⟩ d x f_{z} = \int_{\Omega} f(x)\overline{\phi_{z}(x)}\,dx = \int_{\Omega}f(x)e^{-2\pi i\langle z,x\rangle}\,dx f z = ∫ Ω f ( x ) ϕ z ( x ) d x = ∫ Ω f ( x ) e − 2 π i ⟨ z , x ⟩ d x ( x ∈ Ω ) (x \in\Omega) ( x ∈ Ω ) is the z z z -th Fourier coefficient of f f f . Moreover, we use W 0 1 , p ( Ω ) W_{0}^{1,p}(\Omega) W 0 1 , p ( Ω ) to denote the closure of C c 1 ( Ω ) C_{c}^{1}(\Omega) C c 1 ( Ω ) in W 1 , p ( Ω ) W^{1,p}(\Omega) W 1 , p ( Ω ) . In particular, when p = 2 p = 2 p = 2 , we define H 0 1 ( Ω ) : = W 0 1 , 2 ( Ω ) H_{0}^{1}(\Omega) := W_{0}^{1,2}(\Omega) H 0 1 ( Ω ) := W 0 1 , 2 ( Ω ) .
Furthermore, we use ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ to present the vector 2 norm and, given a data sample { X i } i = 1 n ⊂ Ω \{X_{i}\}_{i=1}^{n} \subset\Omega { X i } i = 1 n ⊂ Ω , ∥ ⋅ ∥ n , p : = ( E n ⋅ p ) 1 / p \|\cdot\|_{n,p} := (\mathbb{E}_{n}\cdot^{p})^{1/p} ∥ ⋅ ∥ n , p := ( E n ⋅ p ) 1/ p denote the empirical p p p norm, where E n : L 2 ( Ω ) → R \mathbb{E}_{n} : L^{2}(\Omega) \to\mathbb{R} E n : L 2 ( Ω ) → R is the corresponding empirical average operator defined as E n f : = 1 n ∑ i = 1 n f ( X i ) \mathbb{E}_{n}f := \frac{1}{n}\sum_{i=1}^{n}f(X_{i}) E n f := n 1 ∑ i = 1 n f ( X i ) , ∀ f ∈ L 2 ( Ω ) \forall f \in L^{2}(\Omega) ∀ f ∈ L 2 ( Ω ) . Given two quantities X X X and Y Y Y , we write X ≲ Y X \lesssim Y X ≲ Y when the inequality X ≤ C Y X \le CY X ≤ C Y holds, where C C C is some constant. For two functions f f f and g g g mapping from R + \mathbb{R}^{+} R + to R \mathbb{R} R , we write f = O ( g ) f = O(g) f = O ( g ) when there exist two constants C ′ C' C ′ and x 0 x_{0} x 0 independent of f f f and g g g , such that the inequality f ( x ) ≤ C ′ g ( x ) f(x) \le C'g(x) f ( x ) ≤ C ′ g ( x ) holds for any x ≥ x 0 x \ge x_{0} x ≥ x 0 . We use X ≃ Y X \simeq Y X ≃ Y to denote X ≲ Y X \lesssim Y X ≲ Y and Y ≲ X Y \lesssim X Y ≲ X .
Regularity Result For the PDE model..
Regularity Results of the DRM Objective Function.
Theorem A.1. We consider the static Schrödinger equation on the unit hypercube on R d \mathbb{R}^{d} R d with the zero Dirichlet boundary condition:
− Δ u + V u = f on Ω , u = 0 on ∂ Ω . (42) \begin{aligned}
-\Delta u + Vu &= f \quad\text{on } \Omega,\\
u &= 0 \quad\text{on } \partial\Omega. \tag*{(42)}
\end{aligned} − Δ u + V u u = f on Ω , = 0 on ∂ Ω. (42)
where f ∈ L 2 ( Ω ) f \in L^2(\Omega) f ∈ L 2 ( Ω ) and V ∈ L ∞ ( Ω ) V \in L^\infty(\Omega) V ∈ L ∞ ( Ω ) with 0 < V min ≤ V ( x ) ≤ V max > 0 0 < V_{\min} \le V(x) \le V_{\max} > 0 0 < V m i n ≤ V ( x ) ≤ V m a x > 0 . There exists a unique weak solution u S ∗ u^*_S u S ∗ to the equivalent variational problem [16 ] :
u S ∗ = arg min u ∈ H 0 1 ( Ω ) E S D R M ( u ) : = arg min u ∈ H 0 1 ( Ω ) { 1 2 ∫ Ω [ ∥ ∇ u ∥ 2 + V ∣ u ∣ 2 ] d x − ∫ Ω f u d x } . (43) u^*_S = \underset{u\in H^1_0(\Omega)}{\arg\min}\,\mathcal{E}^{\mathrm{DRM}}_S(u) := \underset{u\in H^1_0(\Omega)}{\arg\min}\left\{\frac{1}{2}\int_{\Omega}\left[\lVert\nabla u\rVert^2+V\lvert u\rvert^2\right]\,dx-\int_{\Omega}fu\,dx\right\}. \tag*{(43)} u S ∗ = u ∈ H 0 1 ( Ω ) arg min E S DRM ( u ) := u ∈ H 0 1 ( Ω ) arg min { 2 1 ∫ Ω [ ∥ ∇ u ∥ 2 + V ∣ u ∣ 2 ] d x − ∫ Ω f u d x } . (43)
Then for any u ∈ H 1 ( Ω ) u \in H^1(\Omega) u ∈ H 1 ( Ω ) , we have:
min ( 1 , V min ) 2 ∥ u − u S ∗ ∥ H 1 ( Ω ) 2 ≤ E S D R M ( u ) − E S D R M ( u S ∗ ) ≤ max ( 1 , V max ) 2 ∥ u − u S ∗ ∥ H 1 ( Ω ) 2 . (44) \frac{\min(1,V_{\min})}{2}\lVert u-u^*_S\rVert^2_{H^1(\Omega)}
\le\mathcal{E}^{\mathrm{DRM}}_S(u)-\mathcal{E}^{\mathrm{DRM}}_S(u^*_S)
\le\frac{\max(1,V_{\max})}{2}\lVert u-u^*_S\rVert^2_{H^1(\Omega)}. \tag*{(44)} 2 min ( 1 , V m i n ) ∥ u − u S ∗ ∥ H 1 ( Ω ) 2 ≤ E S DRM ( u ) − E S DRM ( u S ∗ ) ≤ 2 max ( 1 , V m a x ) ∥ u − u S ∗ ∥ H 1 ( Ω ) 2 . (44)
Proof. To show that u S ∗ u^*_S u S ∗ satisfies estimate A.3, we first claim that for any u ∈ H 1 ( Ω ) u \in H^1(\Omega) u ∈ H 1 ( Ω ) ,
E S D R M ( u ) − E S D R M ( u S ∗ ) = 1 2 ∫ Ω ∥ ∇ u − ∇ u S ∗ ∥ 2 d x + 1 2 ∫ Ω V ( u S ∗ − u ) 2 d x . (45) \mathcal{E}^{\mathrm{DRM}}_S(u)-\mathcal{E}^{\mathrm{DRM}}_S(u^*_S)
=\frac{1}{2}\int_{\Omega}\lVert\nabla u-\nabla u^*_S\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V(u^*_S-u)^2\,dx. \tag*{(45)} E S DRM ( u ) − E S DRM ( u S ∗ ) = 2 1 ∫ Ω ∥ ∇ u − ∇ u S ∗ ∥ 2 d x + 2 1 ∫ Ω V ( u S ∗ − u ) 2 d x . (45)
In fact, by plugging in the first equation of A.1, one has that
E S D R M ( u S ∗ ) = 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 1 2 ∫ Ω V ∣ u S ∗ ∣ 2 d x − ∫ Ω f u S ∗ d x = 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 1 2 ∫ Ω V ∣ u S ∗ ∣ 2 d x + ∫ Ω ( Δ u S ∗ − V u S ∗ ) u S ∗ d x = 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + ∫ Ω ( Δ u S ∗ ) u S ∗ d x − 1 2 ∫ Ω V ∣ u S ∗ ∣ 2 d x . \begin{aligned}
\mathcal{E}^{\mathrm{DRM}}_S(u^*_S)
=\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V\lvert u^*_S\rvert^2\,dx-\int_{\Omega}f u^*_S\,dx \\
&=\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V\lvert u^*_S\rvert^2\,dx+\int_{\Omega}(\Delta u^*_S-Vu^*_S)u^*_S\,dx \\
&=\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx+\int_{\Omega}(\Delta u^*_S)u^*_S\,dx-\frac{1}{2}\int_{\Omega}V\lvert u^*_S\rvert^2\,dx.
\end{aligned} E S DRM ( u S ∗ ) = 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 2 1 ∫ Ω V ∣ u S ∗ ∣ 2 d x − ∫ Ω f u S ∗ d x = 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 2 1 ∫ Ω V ∣ u S ∗ ∣ 2 d x + ∫ Ω ( Δ u S ∗ − V u S ∗ ) u S ∗ d x = 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + ∫ Ω ( Δ u S ∗ ) u S ∗ d x − 2 1 ∫ Ω V ∣ u S ∗ ∣ 2 d x .
Furthermore, applying Green’s formula to the true solution u S ∗ u^*_S u S ∗ yields:
E S D R M ( u S ∗ ) = 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + ∫ Ω ( Δ u S ∗ ) u S ∗ d x − 1 2 ∫ Ω V ∣ u S ∗ ∣ 2 d x = ∫ ∂ Ω ∂ u S ∗ ∂ n u S ∗ d x − 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x − 1 2 ∫ Ω V ∣ u S ∗ ∣ 2 d x = − 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x − 1 2 ∫ Ω V ∣ u S ∗ ∣ 2 d x , \begin{aligned}
\mathcal{E}^{\mathrm{DRM}}_S(u^*_S)
=\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx+\int_{\Omega}(\Delta u^*_S)u^*_S\,dx-\frac{1}{2}\int_{\Omega}V\lvert u^*_S\rvert^2\,dx \\
&=\int_{\partial\Omega}\frac{\partial u^*_S}{\partial n}u^*_S\,dx-\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx-\frac{1}{2}\int_{\Omega}V\lvert u^*_S\rvert^2\,dx \\
&=-\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx-\frac{1}{2}\int_{\Omega}V\lvert u^*_S\rvert^2\,dx,
\end{aligned} E S DRM ( u S ∗ ) = 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + ∫ Ω ( Δ u S ∗ ) u S ∗ d x − 2 1 ∫ Ω V ∣ u S ∗ ∣ 2 d x = ∫ ∂ Ω ∂ n ∂ u S ∗ u S ∗ d x − 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x − 2 1 ∫ Ω V ∣ u S ∗ ∣ 2 d x = − 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x − 2 1 ∫ Ω V ∣ u S ∗ ∣ 2 d x ,
where the last identity above follows from the second equality in A.1. Now for any u ∈ H 1 ( Ω ) u \in H^1(\Omega) u ∈ H 1 ( Ω ) , applying Green’s formula to u u u and the true solution u S ∗ u^*_S u S ∗ implies:
E S D R M ( u ) − E S D R M ( u S ∗ ) = 1 2 ∫ Ω ∥ ∇ u ∥ 2 d x + 1 2 ∫ Ω V ∣ u ∣ 2 d x − ∫ Ω f u d x + 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 1 2 ∫ Ω V ∣ u S ∗ ∣ 2 d x = 1 2 ∫ Ω ∥ ∇ u ∥ 2 d x + 1 2 ∫ Ω V ∣ u ∣ 2 d x + ∫ Ω ( Δ u S ∗ − V u S ∗ ) u d x + 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 1 2 ∫ Ω V ∣ u S ∗ ∣ 2 d x = 1 2 ∫ Ω ∥ ∇ u ∥ 2 d x + ∫ Ω ( Δ u S ∗ ) u d x + 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 1 2 ∫ Ω V ( u S ∗ − u ) 2 d x = 1 2 ∫ Ω ∥ ∇ u ∥ 2 d x + ∫ ∂ Ω ∂ u S ∗ ∂ n u d x − ∫ Ω ∇ u S ∗ ⋅ ∇ u d x + 1 2 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 1 2 ∫ Ω V ( u S ∗ − u ) 2 d x = 1 2 ∫ Ω ∥ ∇ u − ∇ u S ∗ ∥ 2 d x + 1 2 ∫ Ω V ( u S ∗ − u ) 2 d x . \begin{aligned}
\mathcal{E}^{\mathrm{DRM}}_S(u)-\mathcal{E}^{\mathrm{DRM}}_S(u^*_S)
=\frac{1}{2}\int_{\Omega}\lVert\nabla u\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V\lvert u\rvert^2\,dx-\int_{\Omega}fu\,dx+\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V\lvert u^*_S\rvert^2\,dx \\
&=\frac{1}{2}\int_{\Omega}\lVert\nabla u\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V\lvert u\rvert^2\,dx+\int_{\Omega}(\Delta u^*_S-Vu^*_S)u\,dx+\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V\lvert u^*_S\rvert^2\,dx \\
&=\frac{1}{2}\int_{\Omega}\lVert\nabla u\rVert^2\,dx+\int_{\Omega}(\Delta u^*_S)u\,dx+\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V(u^*_S-u)^2\,dx \\
&=\frac{1}{2}\int_{\Omega}\lVert\nabla u\rVert^2\,dx+\int_{\partial\Omega}\frac{\partial u^*_S}{\partial n}u\,dx-\int_{\Omega}\nabla u^*_S\cdot\nabla u\,dx+\frac{1}{2}\int_{\Omega}\lVert\nabla u^*_S\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V(u^*_S-u)^2\,dx \\
&=\frac{1}{2}\int_{\Omega}\lVert\nabla u-\nabla u^*_S\rVert^2\,dx+\frac{1}{2}\int_{\Omega}V(u^*_S-u)^2\,dx.
\end{aligned} E S DRM ( u ) − E S DRM ( u S ∗ ) = 2 1 ∫ Ω ∥ ∇ u ∥ 2 d x + 2 1 ∫ Ω V ∣ u ∣ 2 d x − ∫ Ω f u d x + 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 2 1 ∫ Ω V ∣ u S ∗ ∣ 2 d x = 2 1 ∫ Ω ∥ ∇ u ∥ 2 d x + 2 1 ∫ Ω V ∣ u ∣ 2 d x + ∫ Ω ( Δ u S ∗ − V u S ∗ ) u d x + 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 2 1 ∫ Ω V ∣ u S ∗ ∣ 2 d x = 2 1 ∫ Ω ∥ ∇ u ∥ 2 d x + ∫ Ω ( Δ u S ∗ ) u d x + 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 2 1 ∫ Ω V ( u S ∗ − u ) 2 d x = 2 1 ∫ Ω ∥ ∇ u ∥ 2 d x + ∫ ∂ Ω ∂ n ∂ u S ∗ u d x − ∫ Ω ∇ u S ∗ ⋅ ∇ u d x + 2 1 ∫ Ω ∥ ∇ u S ∗ ∥ 2 d x + 2 1 ∫ Ω V ( u S ∗ − u ) 2 d x = 2 1 ∫ Ω ∥ ∇ u − ∇ u S ∗ ∥ 2 d x + 2 1 ∫ Ω V ( u S ∗ − u ) 2 d x .
where the last identity above again follows from the second equality in A.1. This completes our proof of identity A.4. Using the assumptions on the potential function V V V then implies:
E S D R M ( u ) − E S D R M ( u S ∗ ) ≤ max ( 1 , V max ) 2 [ ∫ Ω ∥ ∇ u − ∇ u S ∗ ∥ 2 d x + ∫ Ω ( u S ∗ − u ) 2 d x ] = max ( 1 , V max ) 2 ∥ u − u S ∗ ∥ H 1 ( Ω ) 2 . \begin{aligned}
\mathcal{E}^{\mathrm{DRM}}_S(u)-\mathcal{E}^{\mathrm{DRM}}_S(u_S^*) \le\frac{\max(1,V_{\max})}{2}\left[\int_{\Omega}\|\nabla u-\nabla u_S^*\|^2\,dx+\int_{\Omega}(u_S^*-u)^2\,dx\right] \\
&=\frac{\max(1,V_{\max})}{2}\|u-u_S^*\|_{H^1(\Omega)}^2.
\end{aligned} E S DRM ( u ) − E S DRM ( u S ∗ ) ≤ 2 max ( 1 , V m a x ) [ ∫ Ω ∥∇ u − ∇ u S ∗ ∥ 2 d x + ∫ Ω ( u S ∗ − u ) 2 d x ] = 2 max ( 1 , V m a x ) ∥ u − u S ∗ ∥ H 1 ( Ω ) 2 .
E S D R M ( u ) − E S D R M ( u S ∗ ) ≥ max ( 1 , V min ) 2 [ ∫ Ω ∥ ∇ u − ∇ u S ∗ ∥ 2 d x + ∫ Ω ( u S ∗ − u ) 2 d x ] = max ( 1 , V min ) 2 ∥ u − u S ∗ ∥ H 1 ( Ω ) 2 . \begin{aligned}
\mathcal{E}^{\mathrm{DRM}}_S(u)-\mathcal{E}^{\mathrm{DRM}}_S(u_S^*) \ge\frac{\max(1,V_{\min})}{2}\left[\int_{\Omega}\|\nabla u-\nabla u_S^*\|^2\,dx+\int_{\Omega}(u_S^*-u)^2\,dx\right] \\
&=\frac{\max(1,V_{\min})}{2}\|u-u_S^*\|_{H^1(\Omega)}^2.
\end{aligned} E S DRM ( u ) − E S DRM ( u S ∗ ) ≥ 2 max ( 1 , V m i n ) [ ∫ Ω ∥∇ u − ∇ u S ∗ ∥ 2 d x + ∫ Ω ( u S ∗ − u ) 2 d x ] = 2 max ( 1 , V m i n ) ∥ u − u S ∗ ∥ H 1 ( Ω ) 2 .
This completes our proof of A.1.
Regularity Results of the PINN Objective Function.
Theorem A.2. We consider the static Schrödinger equation on the unit hypercube on R d \mathbb{R}^d R d with the Neumann boundary condition:
− Δ u + V u = f on Ω , -\Delta u+Vu=f\ \text{on }\Omega, − Δ u + V u = f on Ω ,
u = 0 on ∂ Ω . (46) u=0\ \text{on }\partial\Omega. \tag*{(46)} u = 0 on ∂ Ω. (46)
where f ∈ L 2 ( Ω ) f\in L^2(\Omega) f ∈ L 2 ( Ω ) and V ∈ L ∞ ( Ω ) V\in L^\infty(\Omega) V ∈ L ∞ ( Ω ) with V − 1 2 Δ V > C min V-\frac{1}{2}\Delta V>C_{\min} V − 2 1 Δ V > C m i n , 0 < C min < V ( x ) ≤ V max 0<C_{\min}<V(x)\le V_{\max} 0 < C m i n < V ( x ) ≤ V m a x and − Δ V ( x ) ≤ V max -\Delta V(x)\le V_{\max} − Δ V ( x ) ≤ V m a x . Then there exists a unique solution u S ∗ ∈ H 0 1 ( Ω ) u_S^*\in H_0^1(\Omega) u S ∗ ∈ H 0 1 ( Ω ) to the following minimization problem [5 ] :
u S ∗ = arg min u ∈ H 0 1 ( Ω ) E S P I N N ( u ) : = arg min u ∈ H 0 1 ( Ω ) { ∫ Ω ∣ Δ u − V u + f ∣ 2 d x } . (47) u_S^*=\underset{u\in H_0^1(\Omega)}{\arg\min}\,\mathcal{E}^{\mathrm{PINN}}_S(u):=\underset{u\in H_0^1(\Omega)}{\arg\min}\left\{\int_{\Omega}|\Delta u-Vu+f|^2\,dx\right\}. \tag*{(47)} u S ∗ = u ∈ H 0 1 ( Ω ) arg min E S PINN ( u ) := u ∈ H 0 1 ( Ω ) arg min { ∫ Ω ∣Δ u − V u + f ∣ 2 d x } . (47)
Then for any u ∈ H 0 1 ( Ω ) u\in H_0^1(\Omega) u ∈ H 0 1 ( Ω ) , we have:
min { 1 , C min } ∥ u − u S ∗ ∥ H 2 ( Ω ) 2 ≤ E S P I N N ( u ) − E S P I N N ( u S ∗ ) ≤ 2 ( 1 + V max + V max 2 ) ∥ u − u S ∗ ∥ H 2 ( Ω ) 2 . (48) \min\{1,C_{\min}\}\|u-u_S^*\|_{H^2(\Omega)}^2\le\mathcal{E}^{\mathrm{PINN}}_S(u)-\mathcal{E}^{\mathrm{PINN}}_S(u_S^*)\le2(1+V_{\max}+V_{\max}^2)\|u-u_S^*\|_{H^2(\Omega)}^2. \tag*{(48)} min { 1 , C m i n } ∥ u − u S ∗ ∥ H 2 ( Ω ) 2 ≤ E S PINN ( u ) − E S PINN ( u S ∗ ) ≤ 2 ( 1 + V m a x + V m a x 2 ) ∥ u − u S ∗ ∥ H 2 ( Ω ) 2 . (48)
Proof. For any u ∈ H 0 1 ( Ω ) u\in H_0^1(\Omega) u ∈ H 0 1 ( Ω ) , we let u ~ = u − u ∗ \tilde{u}=u-u^* u ~ = u − u ∗ , then we have u ~ ∈ H 0 1 ( Ω ) \tilde{u}\in H_0^1(\Omega) u ~ ∈ H 0 1 ( Ω ) .
E S P I N N ( u ) − E S P I N N ( u S ∗ ) = ∫ Ω ∣ Δ u − V u − Δ u ∗ + V u ∗ ∣ 2 d x = ∫ Ω ∣ Δ u ~ − V u ~ ∣ 2 d x = ∫ Ω ( Δ u ~ ) 2 d x + ∫ Ω V 2 u ~ 2 d x − 2 ∫ Ω V u ~ Δ u ~ d x . (49) \begin{aligned}
\mathcal{E}^{\mathrm{PINN}}_S(u)-\mathcal{E}^{\mathrm{PINN}}_S(u_S^*)=\int_{\Omega}|\Delta u-Vu-\Delta u^*+Vu^*|^2dx=\int_{\Omega}|\Delta\tilde{u}-V\tilde{u}|^2dx \\
&=\int_{\Omega}(\Delta\tilde{u})^2dx+\int_{\Omega}V^2\tilde{u}^2dx-2\int_{\Omega}V\tilde{u}\Delta\tilde{u}dx. \tag*{(49)}
\end{aligned} E S PINN ( u ) − E S PINN ( u S ∗ ) = ∫ Ω ∣Δ u − V u − Δ u ∗ + V u ∗ ∣ 2 d x = ∫ Ω ∣Δ u ~ − V u ~ ∣ 2 d x = ∫ Ω ( Δ u ~ ) 2 d x + ∫ Ω V 2 u ~ 2 d x − 2 ∫ Ω V u ~ Δ u ~ d x . (49)
Using Green’s formula, we have:
∫ Ω V u ~ Δ u ~ d x + ∫ Ω ∇ ( V u ~ ) ⋅ ∇ u ~ d x = ∫ ∂ Ω ∂ u ~ ∂ n V u ~ d s = 0 , \int_{\Omega}V\tilde{u}\Delta\tilde{u}dx+\int_{\Omega}\nabla(V\tilde{u})\cdot\nabla\tilde{u}dx=\int_{\partial\Omega}\frac{\partial\tilde{u}}{\partial n}V\tilde{u}ds=0, ∫ Ω V u ~ Δ u ~ d x + ∫ Ω ∇ ( V u ~ ) ⋅ ∇ u ~ d x = ∫ ∂ Ω ∂ n ∂ u ~ V u ~ d s = 0 ,
where the last equality above follows from the fact that u ~ ∈ H 0 1 ( Ω ) \tilde{u}\in H_0^1(\Omega) u ~ ∈ H 0 1 ( Ω ) . This further implies:
E S P I N N ( u ) − E S P I N N ( u S ∗ ) = ∫ Ω ( Δ u ~ ) 2 d x + ∫ Ω V 2 u ~ 2 d x + 2 ∫ Ω ∇ ( V u ~ ) ⋅ ∇ u ~ d x = ∫ Ω ( Δ u ~ ) 2 d x + ∫ Ω V 2 u ~ 2 d x + 2 ∫ Ω V ∥ ∇ u ~ ∥ 2 d x + 2 ∫ Ω u ~ ∇ V ⋅ ∇ u ~ d x . \begin{aligned}
\mathcal{E}^{\mathrm{PINN}}_S(u)-\mathcal{E}^{\mathrm{PINN}}_S(u_S^*)=\int_{\Omega}(\Delta\tilde{u})^2dx+\int_{\Omega}V^2\tilde{u}^2dx+2\int_{\Omega}\nabla(V\tilde{u})\cdot\nabla\tilde{u}dx \\
&=\int_{\Omega}(\Delta\tilde{u})^2dx+\int_{\Omega}V^2\tilde{u}^2dx+2\int_{\Omega}V\|\nabla\tilde{u}\|^2dx+2\int_{\Omega}\tilde{u}\nabla V\cdot\nabla\tilde{u}dx.
\end{aligned} E S PINN ( u ) − E S PINN ( u S ∗ ) = ∫ Ω ( Δ u ~ ) 2 d x + ∫ Ω V 2 u ~ 2 d x + 2 ∫ Ω ∇ ( V u ~ ) ⋅ ∇ u ~ d x = ∫ Ω ( Δ u ~ ) 2 d x + ∫ Ω V 2 u ~ 2 d x + 2 ∫ Ω V ∥∇ u ~ ∥ 2 d x + 2 ∫ Ω u ~ ∇ V ⋅ ∇ u ~ d x .
Using Green’s formula again, we have:
2 ∫ Ω u ~ ∇ V ⋅ ∇ u ~ d x = ∫ Ω ∇ ( u ~ 2 ) ⋅ ∇ V d x = ∫ ∂ Ω ∂ V ∂ n u ~ 2 d s − ∫ Ω u ~ 2 Δ V d x = − ∫ Ω u ~ 2 Δ V d x . 2\int_{\Omega}\widetilde{u}\nabla V\cdot\nabla\widetilde{u}\,dx=\int_{\Omega}\nabla(\widetilde{u}^{2})\cdot\nabla V\,dx=\int_{\partial\Omega}\frac{\partial V}{\partial n}\widetilde{u}^{2}\,ds-\int_{\Omega}\widetilde{u}^{2}\Delta V\,dx=-\int_{\Omega}\widetilde{u}^{2}\Delta V\,dx. 2 ∫ Ω u ∇ V ⋅ ∇ u d x = ∫ Ω ∇ ( u 2 ) ⋅ ∇ V d x = ∫ ∂ Ω ∂ n ∂ V u 2 d s − ∫ Ω u 2 Δ V d x = − ∫ Ω u 2 Δ V d x .
Then we can further deduce that:
E S P I N N ( u ) − E S P I N N ( u S ∗ ) = ∫ Ω ( Δ u ~ ) 2 d x + ∫ Ω ( V 2 − Δ V ) u ~ 2 d x + 2 ∫ Ω V ∥ ∇ u ~ ∥ 2 d x . \mathcal{E}_{S}^{\mathrm{PINN}}(u)-\mathcal{E}_{S}^{\mathrm{PINN}(u_{S}^{*})}=\int_{\Omega}(\Delta\widetilde{u})^{2}\,dx+\int_{\Omega}(V^{2}-\Delta V)\widetilde{u}^{2}\,dx+2\int_{\Omega}V\lVert\nabla\widetilde{u}\rVert^{2}\,dx. E S PINN ( u ) − E S PINN ( u S ∗ ) = ∫ Ω ( Δ u ) 2 d x + ∫ Ω ( V 2 − Δ V ) u 2 d x + 2 ∫ Ω V ∥ ∇ u ∥ 2 d x .
For we have assumed V ∈ L ∞ ( Ω ) V\in L^{\infty}(\Omega) V ∈ L ∞ ( Ω ) with 0 < C min < V 2 − Δ V 0<C_{\min}<V^{2}-\Delta V 0 < C m i n < V 2 − Δ V , 0 < C min < V ( x ) ≤ V max 0<C_{\min}<V(x)\le V_{\max} 0 < C m i n < V ( x ) ≤ V m a x and − Δ V ( x ) ≤ V max -\Delta V(x)\le V_{\max} − Δ V ( x ) ≤ V m a x , thus we have
min { 1 , C min } ∥ u − u S ∗ ∥ H 2 ( Ω ) 2 ≤ E S P I N N ( u ) − E S P I N N ( u S ∗ ) ≤ 2 ( 1 + V max + V max 2 ) ∥ u − u S ∗ ∥ H 2 ( Ω ) 2 (50) \min\{1,C_{\min}\}\lVert u-u_{S}^{*}\rVert_{H^{2}(\Omega)}^{2}\le\mathcal{E}_{S}^{\mathrm{PINN}}(u)-\mathcal{E}_{S}^{\mathrm{PINN}}(u_{S}^{*})\le2(1+V_{\max}+V_{\max}^{2})\lVert u-u_{S}^{*}\rVert_{H^{2}(\Omega)}^{2} \tag*{(50)} min { 1 , C m i n } ∥ u − u S ∗ ∥ H 2 ( Ω ) 2 ≤ E S PINN ( u ) − E S PINN ( u S ∗ ) ≤ 2 ( 1 + V m a x + V m a x 2 ) ∥ u − u S ∗ ∥ H 2 ( Ω ) 2 (50)
Auxiliary definitions and lemmata On Generalization Error
To bound the generalization error, we use the localized Rademacher complexity [3 ] . Recall that the Rademacher complexity of a function class G \mathcal{G} G is defined by
R n ( G ) = E Z E σ [ sup g ∈ G ∣ 1 n ∑ j = 1 n σ j g ( Z j ) ∣ | Z 1 , ⋯ , Z n ] . R_{n}(\mathcal{G})=\mathbb{E}_{Z}\mathbb{E}_{\sigma}\left[\sup_{g\in\mathcal{G}}\left|\frac{1}{n}\sum_{j=1}^{n}\sigma_{j}g(Z_{j})\right|\middle|Z_{1},\cdots,Z_{n}\right]. R n ( G ) = E Z E σ [ g ∈ G sup n 1 j = 1 ∑ n σ j g ( Z j ) Z 1 , ⋯ , Z n ] .
where Z i Z_i Z i are i.i.d samples according to the data distributions and σ j \sigma_j σ j are i.i.d Rademacher random variables which take the value 1 with probability 1 2 \frac{1}{2} 2 1 and value − 1 -1 − 1 with probability 1 2 \frac{1}{2} 2 1 . The following important symmetrization lemma makes the connection between the uniform law of large numbers and the Rademacher complexity.
Lemma A.3 (Symmetrization Lemma). Let F \mathcal{F} F be a set of functions. Then
E sup u ∈ F ∣ 1 n ∑ j = 1 n u ( X j ) − E X ∼ P Ω u ( X ) ∣ ≤ 2 R n ( F ) . \mathbb{E}\sup_{u\in\mathcal{F}}\left|\frac{1}{n}\sum_{j=1}^{n}u(X_j)-\mathbb{E}_{X\sim P_{\Omega}}u(X)\right|\le2R_n(\mathcal{F}). E u ∈ F sup n 1 j = 1 ∑ n u ( X j ) − E X ∼ P Ω u ( X ) ≤ 2 R n ( F ) .
Lemma A.4 (Ledoux–Talagrand contraction [41 Theorem 4.12] ). Assume that ϕ : R → R \phi:\mathbb{R}\to\mathbb{R} ϕ : R → R is L L L -Lipschitz with ϕ ( 0 ) = 0 \phi(0)=0 ϕ ( 0 ) = 0 . Let { σ i } i = 1 n \{\sigma_i\}_{i=1}^{n} { σ i } i = 1 n be independent Rademacher random variables. Then for any T ⊂ R n T\subset\mathbb{R}^{n} T ⊂ R n
E σ [ sup ( t 1 , ⋯ , t n ) ∈ T ∑ i = 1 n σ i ϕ ( t i ) ] ≤ 2 L ⋅ E σ [ sup ( t 1 , ⋯ , t n ) ∈ T ∑ i = 1 n σ i t i ] . \mathbb{E}_{\sigma}\left[\sup_{(t_1,\cdots,t_n)\in T}\sum_{i=1}^{n}\sigma_i\phi(t_i)\right]\le2L\cdot\mathbb{E}_{\sigma}\left[\sup_{(t_1,\cdots,t_n)\in T}\sum_{i=1}^{n}\sigma_i t_i\right]. E σ [ ( t 1 , ⋯ , t n ) ∈ T sup i = 1 ∑ n σ i ϕ ( t i ) ] ≤ 2 L ⋅ E σ [ ( t 1 , ⋯ , t n ) ∈ T sup i = 1 ∑ n σ i t i ] .
Let ( E , ρ ) (E,\rho) ( E , ρ ) be a metric space with metric ρ \rho ρ . A δ \delta δ -cover of a set A ⊂ E A\subset E A ⊂ E with respect to ρ \rho ρ is a collection of points { x 1 , ⋯ , x n } ⊂ A \{x_1,\cdots,x_n\}\subset A { x 1 , ⋯ , x n } ⊂ A such that for every x ∈ A x\in A x ∈ A , there exists i ∈ { 1 , ⋯ , n } i\in\{1,\cdots,n\} i ∈ { 1 , ⋯ , n } such that ρ ( x , x i ) ≤ δ \rho(x,x_i)\le\delta ρ ( x , x i ) ≤ δ . The δ \delta δ -covering number N ( δ , A , ρ ) \mathcal{N}(\delta,A,\rho) N ( δ , A , ρ ) is the cardinality of the smallest δ \delta δ -cover of the set A A A with respect to the metric ρ \rho ρ . Equivalently, the δ \delta δ -covering number N ( δ , A , ρ ) \mathcal{N}(\delta,A,\rho) N ( δ , A , ρ ) is the minimal number of balls B ρ ( x , δ ) B_{\rho}(x,\delta) B ρ ( x , δ ) of radius δ \delta δ needed to cover the set A A A .
Theorem A.5 (Dudley’s Integral theorem). Let F \mathcal{F} F be a function class such that sup f ∈ F ∥ f ∥ n , 2 ≤ M \sup_{f\in\mathcal{F}}\lVert f\rVert_{n,2}\le M sup f ∈ F ∥ f ∥ n , 2 ≤ M . Then the Rademacher complexity R n ( F ) R_n(\mathcal{F}) R n ( F ) satisfies that
R n ( F ) ≤ inf 0 ≤ δ ≤ M { 4 δ + 12 n ∫ δ M log N ( ε , F , ∥ ⋅ ∥ n , 2 ) d ε } . R_n(\mathcal{F})\le\inf_{0\le\delta\le M}\left\{4\delta+\frac{12}{\sqrt{n}}\int_{\delta}^{M}\sqrt{\log\mathcal{N}(\varepsilon,\mathcal{F},\lVert\cdot\rVert_{n,2})}\,d\varepsilon\right\}. R n ( F ) ≤ 0 ≤ δ ≤ M inf { 4 δ + n 12 ∫ δ M log N ( ε , F , ∥ ⋅ ∥ n , 2 ) d ε } .
Lemma A.6 (Talagrand Concentration Inequality). Consider a function class F \mathcal{F} F defined on a probability measure μ \mu μ such that for all f ∈ F f \in\mathcal{F} f ∈ F , we have ∥ f ∥ ∞ ≤ β \lVert f\rVert_{\infty} \le\beta ∥ f ∥ ∞ ≤ β , E μ [ f ] = 0 \mathbb{E}_{\mu}[f] = 0 E μ [ f ] = 0 , E μ [ f 2 ] ≤ σ 2 \mathbb{E}_{\mu}[f^2] \le\sigma^2 E μ [ f 2 ] ≤ σ 2 . Then for any t > 0 t > 0 t > 0 , we can have the following concentration results.
P z 1 , … , z n ∼ μ [ sup f ∈ F 1 n ∑ i = 1 n f ( z i ) ≥ 2 sup f ∈ F E z 1 ′ , … , z n ′ ∼ μ 1 n ∑ i = 1 n f ( z i ′ ) + 2 t σ 2 n + 2 t β n ] ≤ e − t . \mathbb{P}_{z_1,\ldots,z_n\sim\mu}\left[\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}f(z_i)\geq2\sup_{f\in\mathcal{F}}\mathbb{E}_{z'_1,\ldots,z'_n\sim\mu}\frac{1}{n}\sum_{i=1}^{n}f(z'_i)+\sqrt{\frac{2t\sigma^2}{n}}+\frac{2t\beta}{n}\right]\leq e^{-t}. P z 1 , … , z n ∼ μ [ f ∈ F sup n 1 i = 1 ∑ n f ( z i ) ≥ 2 f ∈ F sup E z 1 ′ , … , z n ′ ∼ μ n 1 i = 1 ∑ n f ( z i ′ ) + n 2 t σ 2 + n 2 tβ ] ≤ e − t .
Lemma A.7 (Peeling lemma [3 ] ). Consider some measurable function class F \mathcal{F} F . Assume that there exists a sub-root function ϕ ( r ) \phi(r) ϕ ( r ) satisfying
R n ( { f ∈ F ∣ E [ f ] ≤ r } ) ≤ ϕ ( r ) ( ∀ r > 0 ) . (51) R_n(\{f\in\mathcal{F}\mid\mathbb{E}[f]\leq r\})\leq\phi(r)\quad(\forall\ r>0). \tag*{(51)} R n ({ f ∈ F ∣ E [ f ] ≤ r }) ≤ ϕ ( r ) ( ∀ r > 0 ) . (51) Then we have
E σ i , z n [ sup f ∈ F 1 n ∑ i = 1 n σ i f ( z i ) E [ f ] + r ] ≤ 4 ϕ ( r ) r . \mathbb{E}_{\sigma_i,z_n}\left[\sup_{f\in\mathcal{F}}\frac{\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(z_i)}{\mathbb{E}[f]+r}\right]\leq\frac{4\phi(r)}{r}. E σ i , z n [ f ∈ F sup E [ f ] + r n 1 ∑ i = 1 n σ i f ( z i ) ] ≤ r 4 ϕ ( r ) . Proof. Denote F ( r ) = { f ∈ F ∣ E [ f ] ≤ r } \mathcal{F}(r)=\{f\in\mathcal{F}\mid\mathbb{E}[f]\leq r\} F ( r ) = { f ∈ F ∣ E [ f ] ≤ r } to be the localized set with radius r r r . Then for a fixed set of datapoints { z i } i = 1 n \{z_i\}_{i=1}^{n} { z i } i = 1 n and a fixed set of Rademacher random variables { σ i } i = 1 n \{\sigma_i\}_{i=1}^{n} { σ i } i = 1 n , we have:
E σ i , z n [ sup f ∈ F 1 n ∑ i = 1 n σ i f ( z i ) E [ f ] + r ] ≤ E σ i , z n [ sup f ∈ F ( r ) 1 n ∑ i = 1 n σ i f ( z i ) r ] + ∑ j = 0 ∞ E σ i , z n [ sup f ∈ F ( r 4 j + 1 ) ∖ F ( r 4 j ) 1 n ∑ i = 1 n σ i f ( z i ) r 4 j + r ] \mathbb{E}_{\sigma_i,z_n}\left[\sup_{f\in\mathcal{F}}\frac{\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(z_i)}{\mathbb{E}[f]+r}\right]\leq\mathbb{E}_{\sigma_i,z_n}\left[\sup_{f\in\mathcal{F}(r)}\frac{\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(z_i)}{r}\right]+\sum_{j=0}^{\infty}\mathbb{E}_{\sigma_i,z_n}\left[\sup_{f\in\mathcal{F}(r4^{j+1})\setminus\mathcal{F}(r4^j)}\frac{\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(z_i)}{r4^j+r}\right] E σ i , z n [ f ∈ F sup E [ f ] + r n 1 ∑ i = 1 n σ i f ( z i ) ] ≤ E σ i , z n [ f ∈ F ( r ) sup r n 1 ∑ i = 1 n σ i f ( z i ) ] + j = 0 ∑ ∞ E σ i , z n [ f ∈ F ( r 4 j + 1 ) ∖ F ( r 4 j ) sup r 4 j + r n 1 ∑ i = 1 n σ i f ( z i ) ] ≤ R n ( F ( r ) ) r + ∑ j = 0 ∞ R n ( F ( r 4 j + 1 ) ) r 4 j + r ≤ ϕ ( r ) r + ∑ j = 0 ∞ ϕ ( r 4 j + 1 ) r 4 j + r \leq\frac{R_n(\mathcal{F}(r))}{r}+\sum_{j=0}^{\infty}\frac{R_n(\mathcal{F}(r4^{j+1}))}{r4^j+r}\leq\frac{\phi(r)}{r}+\sum_{j=0}^{\infty}\frac{\phi(r4^{j+1})}{r4^j+r} ≤ r R n ( F ( r )) + j = 0 ∑ ∞ r 4 j + r R n ( F ( r 4 j + 1 )) ≤ r ϕ ( r ) + j = 0 ∑ ∞ r 4 j + r ϕ ( r 4 j + 1 ) ≤ ϕ ( r ) r + ∑ j = 0 ∞ 2 j + 1 ϕ ( r ) r 4 j + r ≤ 4 ϕ ( r ) r . \leq\frac{\phi(r)}{r}+\sum_{j=0}^{\infty}\frac{2^{j+1}\phi(r)}{r4^j+r}\leq\frac{4\phi(r)}{r}. ≤ r ϕ ( r ) + j = 0 ∑ ∞ r 4 j + r 2 j + 1 ϕ ( r ) ≤ r 4 ϕ ( r ) .
We also modify the peeling lemma above, as we aim to apply it to derive the upper bound for the Modified Deep Ritz Method (MDRM).
Lemma A.8 (Peeling Lemma For MDRM). Given some measurable function class F \mathcal{F} F and two continuous mappings g , h : F → R g,h:\mathcal{F}\to\mathbb{R} g , h : F → R , we define a set F \mathcal{F} F of vector functions by:
F : = { ( g ∘ f , h ∘ f ) ∣ f ∈ F } . \mathcal{F}:=\{(g\circ f,h\circ f)\mid f\in\mathcal{F}\}. F := {( g ∘ f , h ∘ f ) ∣ f ∈ F } . For any r > 0 r>0 r > 0 , the localized set F r \mathcal{F}_r F r is defined by:
F r = { ( g f , h f ) ∈ F ∣ E x [ g f ( x ) ] + E y [ h f ( y ) ] ≤ r } . \mathcal{F}_r=\{(g_f,h_f)\in\mathcal{F}\mid\mathbb{E}_x[g_f(x)]+\mathbb{E}_y[h_f(y)]\leq r\}. F r = {( g f , h f ) ∈ F ∣ E x [ g f ( x )] + E y [ h f ( y )] ≤ r } . Moreover, the modified Rademacher Complexity of F r \mathcal{F}_r F r is defined by:
R n , m ( F r ) : = R n ( { g f ∣ ( g f , h f ) ∈ F r } ) + R m ( { h f ∣ ( g f , h f ) ∈ F r } ) . R_{n,m}(\mathcal{F}_r):=R_n(\{g_f\mid(g_f,h_f)\in\mathcal{F}_r\})+R_m(\{h_f\mid(g_f,h_f)\in\mathcal{F}_r\}). R n , m ( F r ) := R n ({ g f ∣ ( g f , h f ) ∈ F r }) + R m ({ h f ∣ ( g f , h f ) ∈ F r }) . Assume that there exists some function ϕ : [ 0 , ∞ ) → [ 0 , ∞ ) \phi:[0,\infty)\to[0,\infty) ϕ : [ 0 , ∞ ) → [ 0 , ∞ ) and some r ⋆ > 0 r^\star>0 r ⋆ > 0 , such that for any r > r ⋆ r>r^\star r > r ⋆ , we have:
ϕ ( 4 r ) ≤ 2 ϕ ( r ) and R n , m ( F r ) ≤ ϕ ( r ) . \phi(4r)\leq2\phi(r)\quad\text{and}\quad R_{n,m}(\mathcal{F}_r)\leq\phi(r). ϕ ( 4 r ) ≤ 2 ϕ ( r ) and R n , m ( F r ) ≤ ϕ ( r ) .
Then for any r > r ⋆ r > r^{\star} r > r ⋆ , we have:
E σ , τ [ E x , y [ sup f ∈ F 1 n ∑ i = 1 n σ i g f ( x i ) + 1 m ∑ j = 1 m τ j h f ( y j ) E x [ g f ( x ) ] + E y [ h f ( y ) ] + r ] ] ≤ 4 ϕ ( r ) r . \mathbb{E}_{\sigma,\tau}\left[\mathbb{E}_{x,y}\left[\sup_{f\in\mathcal{F}}\frac{\frac{1}{n}\sum_{i=1}^{n}\sigma_i g_f(x_i)+\frac{1}{m}\sum_{j=1}^{m}\tau_j h_f(y_j)}{\mathbb{E}_x[g_f(x)]+\mathbb{E}_y[h_f(y)]+r}\right]\right]\leq\frac{4\phi(r)}{r}. E σ , τ [ E x , y [ f ∈ F sup E x [ g f ( x )] + E y [ h f ( y )] + r n 1 ∑ i = 1 n σ i g f ( x i ) + m 1 ∑ j = 1 m τ j h f ( y j ) ] ] ≤ r 4 ϕ ( r ) .
Proof. The proof is the same as the original peeling lemma, thus we omit the detailed proof here.
Local Rademacher Complexity of Truncated Fourier Basis
Definition A.9. (Fourier Series ) Given a domain Ω ⊂ [ 0 , 1 ] d \Omega\subset[0,1]^d Ω ⊂ [ 0 , 1 ] d . For any z ∈ N d z\in\mathbb{N}^d z ∈ N d , we consider the corresponding Fourier basis function ϕ z ( x ) : = e 2 π i ⟨ z , x ⟩ \phi_z(x):=e^{2\pi i\langle z,x\rangle} ϕ z ( x ) := e 2 π i ⟨ z , x ⟩ (x ∈ Ω x\in\Omega x ∈ Ω ). With respect to the Fourier basis, any function f ∈ L 2 ( Ω ) f\in L^2(\Omega) f ∈ L 2 ( Ω ) can be decomposed as the following sum:
f ( x ) : = ∑ z ∈ N d f z ϕ z ( x ) . (52) f(x):=\sum_{z\in\mathbb{N}^d}f_z\phi_z(x). \tag*{(52)} f ( x ) := z ∈ N d ∑ f z ϕ z ( x ) . (52)
where for any z ∈ N d z\in\mathbb{N}^d z ∈ N d , the Fourier coefficient f z = ∫ Ω f ( x ) ϕ z ( x ) ‾ d x f_z=\int_{\Omega}f(x)\overline{\phi_z(x)}dx f z = ∫ Ω f ( x ) ϕ z ( x ) d x .
Definition A.10. (Truncated Fourier Series ) For a fixed positive integer ξ ∈ Z + \xi\in\mathbb{Z}^+ ξ ∈ Z + , we define the space F ξ ( Ω ) F_{\xi}(\Omega) F ξ ( Ω ) of truncated Fourier series as follows:
F ξ ( Ω ) : = { f = ∑ z ∈ N d f z ϕ z | f z = 0 , ∀ ∥ z ∥ ∞ > ξ } . (53) F_{\xi}(\Omega):=\left\{f=\sum_{z\in\mathbb{N}^d}f_z\phi_z\ \middle|\ f_z=0,\ \forall\ \lVert z\rVert_{\infty}>\xi\right\}. \tag*{(53)} F ξ ( Ω ) := { f = z ∈ N d ∑ f z ϕ z f z = 0 , ∀ ∥ z ∥ ∞ > ξ } . (53)
Equivalently, we can decompose any f ∈ F ξ ( Ω ) f\in F_{\xi}(\Omega) f ∈ F ξ ( Ω ) as f : = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z f:=\sum_{\lVert z\rVert_{\infty}\leq\xi}f_z\phi_z f := ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z .
Lemma A.11. (Local Rademacher Complexity of Localized Truncated Fourier Series ) For a fixed ξ ∈ Z + \xi\in\mathbb{Z}^+ ξ ∈ Z + , we consider a localized class of functions F ρ , ξ ( Ω ) = { f ∈ F ξ ( Ω ) | ∥ f ∥ H 1 ( Ω ) 2 ≤ ρ } F_{\rho,\xi}(\Omega)=\left\{f\in F_{\xi}(\Omega)\ \middle|\ \lVert f\rVert_{H^1(\Omega)}^2\leq\rho\right\} F ρ , ξ ( Ω ) = { f ∈ F ξ ( Ω ) ∥ f ∥ H 1 ( Ω ) 2 ≤ ρ } , where ρ > 0 \rho>0 ρ > 0 is fixed. Then we have the following upper bound on the local Rademacher complexity:
R n ( F ρ , ξ ( Ω ) ) = E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i f ( X i ) | X 1 , … , X n ] ] ≲ ρ n ξ d − 2 . (54) R_n(F_{\rho,\xi}(\Omega))=\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in F_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(X_i)\ \middle|\ X_1,\ldots,X_n\right]\right]\lesssim\sqrt{\frac{\rho}{n}\xi^{d-2}}. \tag*{(54)} R n ( F ρ , ξ ( Ω )) = E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i f ( X i ) X 1 , … , X n ] ] ≲ n ρ ξ d − 2 . (54)
Proof. Take an arbitrary function f ∈ F ρ , ξ ( Ω ) f\in F_{\rho,\xi}(\Omega) f ∈ F ρ , ξ ( Ω ) . Let f = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z f=\sum_{\lVert z\rVert_{\infty}\leq\xi}f_z\phi_z f = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z be the Fourier basis expansion of f f f . ρ ≥ ∥ f ∥ H 1 ( Ω ) 2 \rho\geq\lVert f\rVert_{H^1(\Omega)}^2 ρ ≥ ∥ f ∥ H 1 ( Ω ) 2 implies constraint ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ≲ ρ \sum_{\lVert z\rVert_{\infty}\leq\xi}|f_z|^2\lVert z\rVert^2\lesssim\rho ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ≲ ρ on the Fourier coefficients[1 ] .
On the other hand, substituting the Fourier expansion into the average sum 1 n ∑ i = 1 n σ i f ( X i ) \frac{1}{n}\sum_{i=1}^{n}\sigma_i f(X_i) n 1 ∑ i = 1 n σ i f ( X i ) and using Cauchy-Schwarz inequality imply:
1 n ∑ i = 1 n σ i f ( X i ) = 1 n ∑ i = 1 n σ i ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z ( X i ) = 1 n ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n σ i f z ϕ z ( X i ) ≤ 1 n ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ) 1 2 ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ϕ z ( X i ) ∣ 2 ) 1 2 \begin{aligned}
\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(X_i)=\frac{1}{n}\sum_{i=1}^{n}\sigma_i\sum_{\lVert z\rVert_{\infty}\leq\xi}f_z\phi_z(X_i)=\frac{1}{n}\sum_{\lVert z\rVert_{\infty}\leq\xi}\sum_{i=1}^{n}\sigma_i f_z\phi_z(X_i) \\
&\leq\frac{1}{n}\left(\sum_{\lVert z\rVert_{\infty}\leq\xi}|f_z|^2\lVert z\rVert^2\right)^{\frac{1}{2}}\left(\sum_{\lVert z\rVert_{\infty}\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\lVert z\rVert}\phi_z(X_i)\right|^2\right)^{\frac{1}{2}}
\end{aligned} n 1 i = 1 ∑ n σ i f ( X i ) = n 1 i = 1 ∑ n σ i ∥ z ∥ ∞ ≤ ξ ∑ f z ϕ z ( X i ) = n 1 ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n σ i f z ϕ z ( X i ) ≤ n 1 ∥ z ∥ ∞ ≤ ξ ∑ ∣ f z ∣ 2 ∥ z ∥ 2 2 1 ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ϕ z ( X i ) 2 2 1
≲ ρ n ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ϕ z ( X i ) ∣ 2 ) 1 2 . \lesssim\frac{\sqrt{\rho}}{n}\left(\sum_{\lVert z\rVert_{\infty}\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\lVert z\rVert}\phi_z(X_i)\right|^2\right)^{\frac{1}{2}}. ≲ n ρ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ϕ z ( X i ) 2 2 1 .
where we have used the constraint ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ≲ ρ \sum_{\lVert z\rVert_{\infty}\leq\xi}|f_z|^2\lVert z\rVert^2\lesssim\rho ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ≲ ρ in the last step above. Moreover, by taking expectation with respect to the i.i.d. Rademacher random variables σ i \sigma_i σ i (1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n ) and the uniformly sampled data points { X i } i = 1 n \{X_i\}_{i=1}^n { X i } i = 1 n on both sides and applying Jensen’s inequality, we can deduce that:
E X E σ [ 1 n ∑ i = 1 n σ i f ( X i ) ] ≲ ρ n E X , σ [ ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ϕ z ( X i ) ∣ 2 ) 1 2 ] ≤ ρ n ( E X , σ [ ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ϕ z ( X i ) ∣ 2 ) ] ) 1 2 . \begin{aligned}
\mathbb{E}_{X}\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(X_i)\right]\lesssim\frac{\sqrt{\rho}}{n}\mathbb{E}_{X,\sigma}\left[\left(\sum_{\lVert z\rVert_{\infty}\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\lVert z\rVert}\phi_z(X_i)\right|^2\right)^{\frac{1}{2}}\right] \\
&\leq\frac{\sqrt{\rho}}{n}\left(\mathbb{E}_{X,\sigma}\left[\left(\sum_{\lVert z\rVert_{\infty}\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\lVert z\rVert}\phi_z(X_i)\right|^2\right)\right]\right)^{\frac{1}{2}}.
\end{aligned} E X E σ [ n 1 i = 1 ∑ n σ i f ( X i ) ] ≲ n ρ E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ϕ z ( X i ) 2 2 1 ≤ n ρ E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ϕ z ( X i ) 2 2 1 .
Using independence between the random variables σ i \sigma_i σ i (1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n ), we can further simplify the expectation inside the square root above as below:
E X , σ [ ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ϕ z ( X i ) ∣ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ E X , σ [ ∣ ∑ i = 1 n σ i ∥ z ∥ ϕ z ( X i ) ∣ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n E X , σ [ σ i 2 ∥ z ∥ 2 ∣ ϕ z ( X i ) ∣ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n ∣ Ω ∣ ∥ z ∥ 2 ≲ n ∑ ∥ z ∥ ∞ ≤ ξ 1 ∥ z ∥ 2 ≲ n ξ d ξ 2 = n ξ d − 2 . \begin{aligned}
\mathbb{E}_{X,\sigma}\left[\sum_{\lVert z\rVert_{\infty}\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\lVert z\rVert}\phi_z(X_i)\right|^2\right]=\sum_{\lVert z\rVert_{\infty}\leq\xi}\mathbb{E}_{X,\sigma}\left[\left|\sum_{i=1}^{n}\frac{\sigma_i}{\lVert z\rVert}\phi_z(X_i)\right|^2\right] \\
&=\sum_{\lVert z\rVert_{\infty}\leq\xi}\sum_{i=1}^{n}\mathbb{E}_{X,\sigma}\left[\frac{\sigma_i^2}{\lVert z\rVert^2}\left|\phi_z(X_i)\right|^2\right] \\
&=\sum_{\lVert z\rVert_{\infty}\leq\xi}\sum_{i=1}^{n}\frac{|\Omega|}{\lVert z\rVert^2}\lesssim n\sum_{\lVert z\rVert_{\infty}\leq\xi}\frac{1}{\lVert z\rVert^2}\lesssim n\frac{\xi^{d}}{\xi^2}=n\xi^{d-2}.
\end{aligned} E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ϕ z ( X i ) 2 = ∥ z ∥ ∞ ≤ ξ ∑ E X , σ i = 1 ∑ n ∥ z ∥ σ i ϕ z ( X i ) 2 = ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n E X , σ [ ∥ z ∥ 2 σ i 2 ∣ ϕ z ( X i ) ∣ 2 ] = ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ 2 ∣Ω∣ ≲ n ∥ z ∥ ∞ ≤ ξ ∑ ∥ z ∥ 2 1 ≲ n ξ 2 ξ d = n ξ d − 2 .
Combining the two bounds above yields the desired upper bound:
E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i f ( X i ) | X 1 , … , X n ] ] ≲ ρ n n ξ d − 2 = ρ n ξ d − 2 . \mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i f(X_i)\,\middle|\,X_1,\ldots,X_n\right]\right]\lesssim\frac{\sqrt{\rho}}{n}\sqrt{n\xi^{d-2}}=\sqrt{\frac{\rho}{n}\xi^{d-2}}. E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i f ( X i ) X 1 , … , X n ] ] ≲ n ρ n ξ d − 2 = n ρ ξ d − 2 .
Lemma A.12. (Local Rademacher Complexity of Localized Truncated Fourier Series’ Gradient) For a fixed ξ ∈ Z + \xi\in\mathbb{Z}^{+} ξ ∈ Z + , we consider a localized class of functions G ρ , ξ ( Ω ) = { ∥ ∇ f ∥ ∣ f ∈ F ρ , ξ ( Ω ) } \mathcal{G}_{\rho,\xi}(\Omega)=\{\lVert\nabla f\rVert\mid f\in F_{\rho,\xi}(\Omega)\} G ρ , ξ ( Ω ) = {∥ ∇ f ∥ ∣ f ∈ F ρ , ξ ( Ω )} , where ρ > 0 \rho>0 ρ > 0 is fixed. Then for any sample { X i } i = 1 n ⊂ Ω \{X_i\}_{i=1}^{n}\subset\Omega { X i } i = 1 n ⊂ Ω , we have the following upper bound on the local Rademacher complexity:
R n ( G ρ , ξ ( Ω ) ) = E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ∥ ∇ f ( X i ) ∥ | X 1 , … , X n ] ] ≲ ρ n ξ d . (55) R_n(\mathcal{G}_{\rho,\xi}(\Omega))=\mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in F_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\lVert\nabla f(X_i)\rVert\,\middle|\,X_1,\ldots,X_n\right]\right]\lesssim\sqrt{\frac{\rho}{n}\xi^{d}}. \tag*{(55)} R n ( G ρ , ξ ( Ω )) = E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ∥ ∇ f ( X i )∥ X 1 , … , X n ] ] ≲ n ρ ξ d . (55)
Proof. Take an arbitrary function f ∈ F ρ , ξ ( Ω ) f\in F_{\rho,\xi}(\Omega) f ∈ F ρ , ξ ( Ω ) . Let f = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z f=\sum_{\lVert z\rVert_{\infty}\leq\xi}f_z\phi_z f = ∑ ∥ z ∥ ∞ ≤ ξ f z ϕ z be the Fourier basis expansion of f f f . Similarly, the norm restriction condition ∥ f ∥ H 1 ( Ω ) ≤ ρ \lVert f\rVert_{H^1(\Omega)}\leq\rho ∥ f ∥ H 1 ( Ω ) ≤ ρ can be reduced to the following condition about Fourier coefficients:
∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ≲ ρ . \sum_{\lVert z\rVert_{\infty}\leq\xi}|f_z|^2\lVert z\rVert^2\lesssim\rho. ∥ z ∥ ∞ ≤ ξ ∑ ∣ f z ∣ 2 ∥ z ∥ 2 ≲ ρ .
Moreover, substituting the Fourier expansion into the average sum 1 n ∑ i = 1 n σ i ∥ ∇ f ( X i ) ∥ \frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla f(X_i)\| n 1 ∑ i = 1 n σ i ∥∇ f ( X i ) ∥ and using Cauchy–Schwarz inequality imply:
1 n ∑ i = 1 n σ i ∥ ∇ f ( X i ) ∥ = 1 n ∑ i = 1 n σ i ∥ ∑ ∥ z ∥ ∞ ≤ ξ f z ∇ ϕ z ( X i ) ∥ ≤ 1 n ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n σ i ∣ f z ∇ ϕ z ( X i ) ∣ ≤ 1 n ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ) 1 2 ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ∥ ∇ ϕ z ( X i ) ∥ ∣ 2 ) 1 2 \begin{aligned}
\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla f(X_i)\|=\frac{1}{n}\sum_{i=1}^{n}\sigma_i\left\|\sum_{\|z\|_\infty\leq\xi}f_z\nabla\phi_z(X_i)\right\|\leq\frac{1}{n}\sum_{\|z\|_\infty\leq\xi}\sum_{i=1}^{n}\sigma_i\left|f_z\nabla\phi_z(X_i)\right| \\
&\leq\frac{1}{n}\left(\sum_{\|z\|_\infty\leq\xi}|f_z|^2\|z\|^2\right)^{\frac{1}{2}}\left(\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\|z\|}\|\nabla\phi_z(X_i)\|\right|^2\right)^{\frac{1}{2}}
\end{aligned} n 1 i = 1 ∑ n σ i ∥∇ f ( X i ) ∥ = n 1 i = 1 ∑ n σ i ∥ z ∥ ∞ ≤ ξ ∑ f z ∇ ϕ z ( X i ) ≤ n 1 ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n σ i ∣ f z ∇ ϕ z ( X i ) ∣ ≤ n 1 ∥ z ∥ ∞ ≤ ξ ∑ ∣ f z ∣ 2 ∥ z ∥ 2 2 1 ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ∥∇ ϕ z ( X i ) ∥ 2 2 1
≲ ρ n ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ∥ ∇ ϕ z ( X i ) ∥ ∣ 2 ) 1 2 . \lesssim\frac{\sqrt{\rho}}{n}\left(\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\|z\|}\|\nabla\phi_z(X_i)\|\right|^2\right)^{\frac{1}{2}}. ≲ n ρ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ∥∇ ϕ z ( X i ) ∥ 2 2 1 .
where we have used the constraint ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ≲ ρ \sum_{\|z\|_\infty\leq\xi}|f_z|^2\|z\|^2\lesssim\rho ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 2 ≲ ρ in the last step above. Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σ i \sigma_i σ i (1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n ) and the uniformly sampled data points { X i } i = 1 n \{X_i\}_{i=1}^{n} { X i } i = 1 n on both sides and applying Jensen’s inequality, we can deduce that:
E X E σ [ 1 n ∑ i = 1 n σ i ∥ ∇ f ( X i ) ∥ ] ≲ ρ n E X , σ [ ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ∥ ∇ ϕ z ( X i ) ∥ ∣ 2 ) 1 2 ] ≤ ρ n ( E X , σ [ ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ∥ ∇ ϕ z ( X i ) ∥ ∣ 2 ] ) 1 2 . \begin{aligned}
\mathbb{E}_X\mathbb{E}_\sigma\left[\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla f(X_i)\|\right]\lesssim\frac{\sqrt{\rho}}{n}\mathbb{E}_{X,\sigma}\left[\left(\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\|z\|}\|\nabla\phi_z(X_i)\|\right|^2\right)^{\frac{1}{2}}\right] \\
&\leq\frac{\sqrt{\rho}}{n}\left(\mathbb{E}_{X,\sigma}\left[\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\|z\|}\|\nabla\phi_z(X_i)\|\right|^2\right]\right)^{\frac{1}{2}}.
\end{aligned} E X E σ [ n 1 i = 1 ∑ n σ i ∥∇ f ( X i ) ∥ ] ≲ n ρ E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ∥∇ ϕ z ( X i ) ∥ 2 2 1 ≤ n ρ E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ∥∇ ϕ z ( X i ) ∥ 2 2 1 .
Using independence between the random variables σ i \sigma_i σ i (1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n ), we can further simplify the expectation inside the square root above as below:
E X , σ [ ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ ∥ ∇ ϕ z ( X i ) ∥ ∣ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ E X , σ [ ∣ ∑ i = 1 n σ i ∥ z ∥ ∥ ∇ ϕ z ( X i ) ∥ ∣ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n E X , σ [ σ i 2 ∥ z ∥ 2 ∥ ∇ ϕ z ( X i ) ∥ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n ∣ Ω ∣ 4 π 2 ∥ z ∥ 2 ∥ z ∥ 2 ≲ n ∑ ∥ z ∥ ∞ ≤ ξ 1 ≲ n ξ c d . \begin{aligned}
\mathbb{E}_{X,\sigma}\left[\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^{n}\frac{\sigma_i}{\|z\|}\|\nabla\phi_z(X_i)\|\right|^2\right]=\sum_{\|z\|_\infty\leq\xi}\mathbb{E}_{X,\sigma}\left[\left|\sum_{i=1}^{n}\frac{\sigma_i}{\|z\|}\|\nabla\phi_z(X_i)\|\right|^2\right] \\
&=\sum_{\|z\|_\infty\leq\xi}\sum_{i=1}^{n}\mathbb{E}_{X,\sigma}\left[\frac{\sigma_i^2}{\|z\|^2}\|\nabla\phi_z(X_i)\|^2\right] \\
&=\sum_{\|z\|_\infty\leq\xi}\sum_{i=1}^{n}|\Omega|\frac{4\pi^2\|z\|^2}{\|z\|^2}\lesssim n\sum_{\|z\|_\infty\leq\xi}1\lesssim n\xi^{cd}.
\end{aligned} E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ σ i ∥∇ ϕ z ( X i ) ∥ 2 = ∥ z ∥ ∞ ≤ ξ ∑ E X , σ i = 1 ∑ n ∥ z ∥ σ i ∥∇ ϕ z ( X i ) ∥ 2 = ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n E X , σ [ ∥ z ∥ 2 σ i 2 ∥∇ ϕ z ( X i ) ∥ 2 ] = ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∣Ω∣ ∥ z ∥ 2 4 π 2 ∥ z ∥ 2 ≲ n ∥ z ∥ ∞ ≤ ξ ∑ 1 ≲ n ξ c d .
Combining the two bounds above yields the desired upper bound:
E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i ∥ ∇ f ( X i ) ∥ ∣ X 1 , ⋯ , X n ] ] ≲ ρ n n ξ c d = ρ n ξ d 2 . \mathbb{E}_X\left[\mathbb{E}_\sigma\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\|\nabla f(X_i)\|\mid X_1,\cdots,X_n\right]\right]\lesssim\frac{\sqrt{\rho}}{n}\sqrt{n\xi^{cd}}=\sqrt{\frac{\rho}{n}}\xi^{\frac{d}{2}}. E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i ∥∇ f ( X i ) ∥ ∣ X 1 , ⋯ , X n ] ] ≲ n ρ n ξ c d = n ρ ξ 2 d .
Lemma A.13. (Local Rademacher Complexity of Localized Truncated Fourier Series’ Laplacian) For a fixed ξ ∈ Z + \xi\in\mathbb{Z}^{+} ξ ∈ Z + , we consider a localized class of functions J ρ , ξ ( Ω ) : = { f ∈ F ξ ( Ω ) ∣ ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ } \mathcal{J}_{\rho,\xi}(\Omega):=\left\{f\in F_\xi(\Omega)\mid\|f\|_{H^2(\Omega)}^2\leq\rho\right\} J ρ , ξ ( Ω ) := { f ∈ F ξ ( Ω ) ∣ ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ } , where ρ > 0 \rho>0 ρ > 0 is fixed. Correspondingly, we define a localized class of Laplacians K ρ , ξ ( Ω ) : = { Δ f ∣ f ∈ J ρ , ξ ( Ω ) } \mathcal{K}_{\rho,\xi}(\Omega) := \{\Delta f \mid f \in\mathcal{J}_{\rho,\xi}(\Omega)\} K ρ , ξ ( Ω ) := { Δ f ∣ f ∈ J ρ , ξ ( Ω )} . Then for any sample { X i } i = 1 n ⊂ Ω \{X_i\}_{i=1}^n \subset\Omega { X i } i = 1 n ⊂ Ω , we have the following upper bound on the local Rademacher complexity:
R n ( K ρ , ξ ( Ω ) ) = E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i Δ f ( X i ) | X 1 , … , X n ] ] ≲ ρ n ξ d 2 . (56) R_n(\mathcal{K}_{\rho,\xi}(\Omega)) = \mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^n \sigma_i\Delta f(X_i)\ \middle|\ X_1,\ldots,X_n\right]\right] \lesssim\sqrt{\frac{\rho}{n}\xi^{\frac{d}{2}}}. \tag*{(56)} R n ( K ρ , ξ ( Ω )) = E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i Δ f ( X i ) X 1 , … , X n ] ] ≲ n ρ ξ 2 d . (56)
Proof. Take an arbitrary function f ∈ J ρ , ξ ( Ω ) f \in\mathcal{J}_{\rho,\xi}(\Omega) f ∈ J ρ , ξ ( Ω ) . Let f = ∑ ∥ z ∥ ∞ ≤ ξ f z φ z f=\sum_{\|z\|_\infty\leq\xi}f_z\varphi_z f = ∑ ∥ z ∥ ∞ ≤ ξ f z φ z be the Fourier basis expansion of f f f . Similarly, the norm restriction condition ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ \|f\|_{H^2(\Omega)}^2\leq\rho ∥ f ∥ H 2 ( Ω ) 2 ≤ ρ can be reduced to the following condition about fourier coefficients:
∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 4 ≲ ρ . \sum_{\|z\|_\infty\leq\xi}|f_z|^2\|z\|^4\lesssim\rho. ∥ z ∥ ∞ ≤ ξ ∑ ∣ f z ∣ 2 ∥ z ∥ 4 ≲ ρ .
Moreover, substituting the Fourier expansion into the average sum 1 n ∑ i = 1 n σ i Δ f ( X i ) \frac{1}{n}\sum_{i=1}^n\sigma_i\Delta f(X_i) n 1 ∑ i = 1 n σ i Δ f ( X i ) and using Cauchy–Schwarz inequality imply:
1 n ∑ i = 1 n σ i Δ f ( X i ) = 1 n ∑ i = 1 n σ i ∑ ∥ z ∥ ∞ ≤ ξ f z Δ φ z ( X i ) = 1 n ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n σ i f z Δ φ z ( X i ) ≤ 1 n ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 4 ) 1 2 ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ 2 Δ φ z ( X i ) ∣ 2 ) 1 2 ≲ ρ n ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ 2 Δ φ z ( X i ) ∣ 2 ) 1 2 . \begin{aligned}
\frac{1}{n}\sum_{i=1}^n\sigma_i\Delta f(X_i)
&=\frac{1}{n}\sum_{i=1}^n\sigma_i\sum_{\|z\|_\infty\leq\xi}f_z\Delta\varphi_z(X_i)
=\frac{1}{n}\sum_{\|z\|_\infty\leq\xi}\sum_{i=1}^n\sigma_if_z\Delta\varphi_z(X_i)\\
&\leq\frac{1}{n}\left(\sum_{\|z\|_\infty\leq\xi}|f_z|^2\|z\|^4\right)^{\frac{1}{2}}
\left(\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^n\frac{\sigma_i}{\|z\|^2}\Delta\varphi_z(X_i)\right|^2\right)^{\frac{1}{2}}\\
&\lesssim\frac{\sqrt{\rho}}{n}\left(\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^n\frac{\sigma_i}{\|z\|^2}\Delta\varphi_z(X_i)\right|^2\right)^{\frac{1}{2}}.
\end{aligned} n 1 i = 1 ∑ n σ i Δ f ( X i ) = n 1 i = 1 ∑ n σ i ∥ z ∥ ∞ ≤ ξ ∑ f z Δ φ z ( X i ) = n 1 ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n σ i f z Δ φ z ( X i ) ≤ n 1 ∥ z ∥ ∞ ≤ ξ ∑ ∣ f z ∣ 2 ∥ z ∥ 4 2 1 ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ 2 σ i Δ φ z ( X i ) 2 2 1 ≲ n ρ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ 2 σ i Δ φ z ( X i ) 2 2 1 .
where we have used the constraint ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 4 ≲ ρ \sum_{\|z\|_\infty\leq\xi}|f_z|^2\|z\|^4\lesssim\rho ∑ ∥ z ∥ ∞ ≤ ξ ∣ f z ∣ 2 ∥ z ∥ 4 ≲ ρ in the last step above. Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σ i \sigma_i σ i (1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n ) and the uniformly sampled data points { X i } i = 1 n \{X_i\}_{i=1}^n { X i } i = 1 n on both sides and applying Jensen’s inequality, we can deduce that:
E X E σ [ 1 n ∑ i = 1 n σ i Δ f ( X i ) ] ≲ ρ n E X , σ [ ( ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ 2 Δ φ z ( X i ) ∣ 2 ) 1 2 ] ≤ ρ n ( E X , σ [ ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ 2 Δ φ z ( X i ) ∣ 2 ] ) 1 2 . \begin{aligned}
\mathbb{E}_{X}\mathbb{E}_{\sigma}\left[\frac{1}{n}\sum_{i=1}^n\sigma_i\Delta f(X_i)\right]\lesssim\frac{\sqrt{\rho}}{n}\mathbb{E}_{X,\sigma}\left[\left(\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^n\frac{\sigma_i}{\|z\|^2}\Delta\varphi_z(X_i)\right|^2\right)^{\frac{1}{2}}\right] \\
&\leq\frac{\sqrt{\rho}}{n}\left(\mathbb{E}_{X,\sigma}\left[\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^n\frac{\sigma_i}{\|z\|^2}\Delta\varphi_z(X_i)\right|^2\right]\right)^{\frac{1}{2}}.
\end{aligned} E X E σ [ n 1 i = 1 ∑ n σ i Δ f ( X i ) ] ≲ n ρ E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ 2 σ i Δ φ z ( X i ) 2 2 1 ≤ n ρ E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ 2 σ i Δ φ z ( X i ) 2 2 1 .
Using independence between the random variables σ i \sigma_i σ i (1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n ), we can further simplify the expectation inside the square root above as below:
E X , σ [ ∑ ∥ z ∥ ∞ ≤ ξ ∣ ∑ i = 1 n σ i ∥ z ∥ 2 Δ φ z ( X i ) ∣ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ E X , σ [ ∣ ∑ i = 1 n σ i ∥ z ∥ 2 Δ φ z ( X i ) ∣ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n E X , σ [ σ i 2 ∥ z ∥ 4 ∣ Δ φ z ( X i ) ∣ 2 ] = ∑ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 n ∣ Ω ∣ 16 π 4 ∥ z ∥ 4 ∥ z ∥ 4 ≲ n ∑ ∥ z ∥ ∞ ≤ ξ 1 ≲ n ξ d . \begin{aligned}
\mathbb{E}_{X,\sigma}\left[\sum_{\|z\|_\infty\leq\xi}\left|\sum_{i=1}^n\frac{\sigma_i}{\|z\|^2}\Delta\varphi_z(X_i)\right|^2\right]
&=\sum_{\|z\|_\infty\leq\xi}\mathbb{E}_{X,\sigma}\left[\left|\sum_{i=1}^n\frac{\sigma_i}{\|z\|^2}\Delta\varphi_z(X_i)\right|^2\right]\\
&=\sum_{\|z\|_\infty\leq\xi}\sum_{i=1}^n\mathbb{E}_{X,\sigma}\left[\frac{\sigma_i^2}{\|z\|^4}|\Delta\varphi_z(X_i)|^2\right]\\
&=\sum_{\|z\|_\infty\leq\xi}\sum_{i=1}^n|\Omega|\frac{16\pi^4\|z\|^4}{\|z\|^4}\lesssim n\sum_{\|z\|_\infty\leq\xi}1\lesssim n\xi^d.
\end{aligned} E X , σ ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∥ z ∥ 2 σ i Δ φ z ( X i ) 2 = ∥ z ∥ ∞ ≤ ξ ∑ E X , σ i = 1 ∑ n ∥ z ∥ 2 σ i Δ φ z ( X i ) 2 = ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n E X , σ [ ∥ z ∥ 4 σ i 2 ∣Δ φ z ( X i ) ∣ 2 ] = ∥ z ∥ ∞ ≤ ξ ∑ i = 1 ∑ n ∣Ω∣ ∥ z ∥ 4 16 π 4 ∥ z ∥ 4 ≲ n ∥ z ∥ ∞ ≤ ξ ∑ 1 ≲ n ξ d .
Combining the two bounds above yields the desired upper bound:
E X [ E σ [ sup f ∈ F ρ , ξ ( Ω ) 1 n ∑ i = 1 n σ i Δ f ( X i ) | X 1 , ⋯ , X n ] ] ≲ ρ n n ξ d = ρ n ξ d . \mathbb{E}_{X}\left[\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}_{\rho,\xi}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\Delta f(X_i)\,\middle|\,X_1,\cdots,X_n\right]\right]\lesssim\frac{\sqrt{\rho}}{n}\sqrt{n\xi^{d}}=\sqrt{\frac{\rho}{n}\xi^{d}}. E X [ E σ [ f ∈ F ρ , ξ ( Ω ) sup n 1 i = 1 ∑ n σ i Δ f ( X i ) X 1 , ⋯ , X n ] ] ≲ n ρ n ξ d = n ρ ξ d .
Local Rademacher Complexity of the Deep Neural Network Model
In this section we aim to bound the local Rademacher Complexity of a Deep Neural Network. We first bound the covering number of the function space composed by the gradient of all possible neural networks and then apply a Duley Integral to achieve the final bound.
Definition A.14. Let η l \eta_l η l denote the l l l -ReLU activation function. Here we use η 3 : = max { 0 , x } \eta_3:=\max\{0,x\} η 3 := max { 0 , x } [15 ] as the activation function to ensure smoothness. We can define the space consisting of all neural network models with depth L L L , width W W W , sparsity constraint S S S and norm constraint B B B as follows:
\Phi(L,W,S,B):=\{(\mathcal{W}^{(L)}\eta_3(\cdot)+b^{(L)})\cdots(\mathcal{W}^{(1)}x+b^{(1)})\,\middle|\,\mathcal{W}^{(L)}\in\mathbb{R}^{1\times W},\ b^{(L)}\in\mathbb{R}, \tag*{(57)}
W ( 1 ) ∈ R W × d , b ( 1 ) ∈ R W , W ( l ) ∈ R W × W , b ( l ) ∈ R W ( 1 < l < L ) , (58) \mathcal{W}^{(1)}\in\mathbb{R}^{W\times d},\ b^{(1)}\in\mathbb{R}^{W},\ \mathcal{W}^{(l)}\in\mathbb{R}^{W\times W},\ b^{(l)}\in\mathbb{R}^{W}\ (1<l<L), \tag*{(58)} W ( 1 ) ∈ R W × d , b ( 1 ) ∈ R W , W ( l ) ∈ R W × W , b ( l ) ∈ R W ( 1 < l < L ) , (58) ∑ l = 1 L ( ∥ W ( l ) ∥ 0 + ∥ b ( l ) ∥ 0 ) ≤ S , max l ∥ W ( l ) ∥ ∞ , ∞ ∨ ∥ b ( l ) ∥ ∞ ≤ B } . (59) \sum_{l=1}^{L}(\|\mathcal{W}^{(l)}\|_{0}+\|b^{(l)}\|_{0})\leq S,\max_l\|\mathcal{W}^{(l)}\|_{\infty,\infty}\vee\|b^{(l)}\|_{\infty}\leq B\}. \tag*{(59)} l = 1 ∑ L ( ∥ W ( l ) ∥ 0 + ∥ b ( l ) ∥ 0 ) ≤ S , l max ∥ W ( l ) ∥ ∞ , ∞ ∨ ∥ b ( l ) ∥ ∞ ≤ B } . (59) where ∥ ⋅ ∥ 0 \|\cdot\|_0 ∥ ⋅ ∥ 0 measures the number of nonzero entries in a matrix and ∥ ⋅ ∥ ∞ , ∞ \|\cdot\|_{\infty,\infty} ∥ ⋅ ∥ ∞ , ∞ measures the maximum of the absolute values of the entries in a matrix. For any d ∈ Z + d\in\mathbb{Z}^{+} d ∈ Z + , we refer to an arbitrary element in Φ ( L , W , S , B ) \Phi(L,W,S,B) Φ ( L , W , S , B ) as a ReLU3 Deep Neural Network. Then for any index 1 ≤ k ≤ L 1\leq k\leq L 1 ≤ k ≤ L , we use F k F_k F k to denote the k k k -ReLU3 Deep Neural Network composed by the first k k k layers, i.e:
F k ( x ) : = ( W F ( k ) η 3 ( ⋅ ) + b F ( k ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) . F_k(x):=\left(\mathcal{W}^{(k)}_F\eta_3(\cdot)+b^{(k)}_F\right)\cdots\left(\mathcal{W}^{(1)}_F x+b^{(1)}_F\right). F k ( x ) := ( W F ( k ) η 3 ( ⋅ ) + b F ( k ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) . Also, we use Φ k ( L , W , S , B ) \Phi_k(L,W,S,B) Φ k ( L , W , S , B ) to denote the space consisting of all F k F_k F k . In particular, when k = L k=L k = L , we have:
F ( x ) : = F L ( x ) = ( W F ( L ) η 3 ( ⋅ ) + b F ( L ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) , and Φ L ( L , W , S , B ) = Φ ( L , W , S , B ) . F(x):=F_L(x)=\left(\mathcal{W}^{(L)}_F\eta_3(\cdot)+b^{(L)}_F\right)\cdots\left(\mathcal{W}^{(1)}_F x+b^{(1)}_F\right),\quad\text{and}\quad\Phi_L(L,W,S,B)=\Phi(L,W,S,B). F ( x ) := F L ( x ) = ( W F ( L ) η 3 ( ⋅ ) + b F ( L ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) , and Φ L ( L , W , S , B ) = Φ ( L , W , S , B ) . Furthermore, given that the domain Ω ⊂ [ 0 , 1 ] d \Omega\subset[0,1]^d Ω ⊂ [ 0 , 1 ] d is bounded, we have sup x ∈ Ω ∥ x ∥ ∞ = 1 \sup_{x\in\Omega}\|x\|_{\infty}=1 sup x ∈ Ω ∥ x ∥ ∞ = 1 .
Lemma A.15. (Upper bound on ∞ \infty ∞ -norm of functions in DNN space ) For any 1 ≤ k ≤ L 1\leq k\leq L 1 ≤ k ≤ L , the following inequality holds:
sup x ∈ Ω , F k ∈ Φ k ( L , W , S , B ) ∥ F k ( x ) ∥ ∞ ≤ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − k + 1 . \sup_{x\in\Omega,\ F_k\in\Phi_k(L,W,S,B)}\left\|F_k(x)\right\|_{\infty}\leq W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-1}{2}-k+1}. x ∈ Ω , F k ∈ Φ k ( L , W , S , B ) sup ∥ F k ( x ) ∥ ∞ ≤ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − k + 1 . Proof. We use induction to prove this claim. Base cases: When k = 1 k=1 k = 1 , we have that for any x ∈ Ω x\in\Omega x ∈ Ω and any F 1 ∈ Φ 1 ( L , W , S , B ) F_1\in\Phi_1(L,W,S,B) F 1 ∈ Φ 1 ( L , W , S , B ) , the following holds:
∥ F 1 ( x ) ∥ ∞ = ∥ W F ( 1 ) x + b F ( 1 ) ∥ ∞ ≤ ∥ W F ( 1 ) ∥ ∞ ∥ x ∥ ∞ + ∥ b F ( 1 ) ∥ ∞ \left\|F_1(x)\right\|_{\infty}=\left\|\mathcal{W}^{(1)}_F x+b^{(1)}_F\right\|_{\infty}\leq\left\|\mathcal{W}^{(1)}_F\right\|_{\infty}\|x\|_{\infty}+\left\|b^{(1)}_F\right\|_{\infty} ∥ F 1 ( x ) ∥ ∞ = W F ( 1 ) x + b F ( 1 ) ∞ ≤ W F ( 1 ) ∞ ∥ x ∥ ∞ + b F ( 1 ) ∞ ≤ d ∥ W F ( 1 ) ∥ ∞ , ∞ + B ≤ d B + B ≤ 2 ( B ∨ d ) 2 . (60) \leq d\left\|\mathcal{W}^{(1)}_F\right\|_{\infty,\infty}+B\leq dB+B\leq2(B\vee d)^2. \tag*{(60)} ≤ d W F ( 1 ) ∞ , ∞ + B ≤ d B + B ≤ 2 ( B ∨ d ) 2 . (60)
When k = 2 k=2 k = 2 , we have that for any x ∈ Ω x \in\Omega x ∈ Ω and any F 2 ∈ Φ 2 ( L , W , S , B ) F_2 \in\Phi_2(L,W,S,B) F 2 ∈ Φ 2 ( L , W , S , B ) , the following holds:
∥ F 2 ( x ) ∥ ∞ = ∥ W F ( 2 ) η 3 ( F 1 ( x ) ) + b F ( 2 ) ∥ ∞ ≤ ∥ W F ( 2 ) ∥ ∞ ∥ η 3 ( F 1 ( x ) ) ∥ ∞ + ∥ b F ( 2 ) ∥ ∞ ≤ W ∥ W F ( 2 ) ∥ ∞ , ∞ ∥ F 1 ( x ) ∥ ∞ 3 + B . \lVert F_2(x)\rVert_\infty= \lVert\mathcal{W}^{(2)}_F\eta_3(F_1(x))+b^{(2)}_F\rVert_\infty\leq\lVert\mathcal{W}^{(2)}_F\rVert_\infty\lVert\eta_3(F_1(x))\rVert_\infty+\lVert b^{(2)}_F\rVert_\infty\leq W\lVert\mathcal{W}^{(2)}_F\rVert_{\infty,\infty}\lVert F_1(x)\rVert_\infty^3+B. ∥ F 2 ( x ) ∥ ∞ = ∥ W F ( 2 ) η 3 ( F 1 ( x )) + b F ( 2 ) ∥ ∞ ≤ ∥ W F ( 2 ) ∥ ∞ ∥ η 3 ( F 1 ( x )) ∥ ∞ + ∥ b F ( 2 ) ∥ ∞ ≤ W ∥ W F ( 2 ) ∥ ∞ , ∞ ∥ F 1 ( x ) ∥ ∞ 3 + B .
By applying the bound proved in the case when k = 1 k=1 k = 1 , we have:
∥ F 2 ( x ) ∥ ∞ ≤ W B ( d B + B ) 3 + B = W B 4 ( d + 1 ) 3 + B = W B 4 ( d 3 + 3 d 2 + 3 d + 1 ) + B ≤ 8 W ( B ∨ d ) 7 . \begin{aligned}
\lVert F_2(x)\rVert_\infty&\leq WB(dB+B)^3+B=WB^4(d+1)^3+B\\
&=WB^4(d^3+3d^2+3d+1)+B\leq8W(B\vee d)^7.
\end{aligned} ∥ F 2 ( x ) ∥ ∞ ≤ W B ( d B + B ) 3 + B = W B 4 ( d + 1 ) 3 + B = W B 4 ( d 3 + 3 d 2 + 3 d + 1 ) + B ≤ 8 W ( B ∨ d ) 7 .
where the last inequality follows from the assumption that W ≥ 2 W\geq2 W ≥ 2 .
Inductive Step: Now we assume that the claim has been proved for k − 1 k-1 k − 1 , where 3 ≤ k ≤ L 3\leq k\leq L 3 ≤ k ≤ L .
Similarly, for any x ∈ Ω x\in\Omega x ∈ Ω and any F k ∈ Φ k ( L , W , S , B ) F_k\in\Phi_k(L,W,S,B) F k ∈ Φ k ( L , W , S , B ) , we have:
∥ F k ( x ) ∥ ∞ = ∥ W F ( k ) η 3 ( F k − 1 ( x ) ) + b F ( k ) ∥ ∞ ≤ ∥ W F ( k ) ∥ ∞ ∥ η 3 ( F k − 1 ( x ) ) ∥ ∞ + ∥ b F ( k ) ∥ ∞ ≤ W ∥ W F ( k ) ∥ ∞ , ∞ ∥ F k − 1 ( x ) ∥ ∞ 3 + B ≤ W B ∥ F k − 1 ( x ) ∥ ∞ 3 + B . \begin{aligned}
\lVert F_k(x)\rVert_\infty&= \lVert\mathcal{W}^{(k)}_F\eta_3(F_{k-1}(x))+b^{(k)}_F\rVert_\infty\\
&\leq\lVert\mathcal{W}^{(k)}_F\rVert_\infty\lVert\eta_3(F_{k-1}(x))\rVert_\infty+\lVert b^{(k)}_F\rVert_\infty\\
&\leq W\lVert\mathcal{W}^{(k)}_F\rVert_{\infty,\infty}\lVert F_{k-1}(x)\rVert_\infty^3+B\leq WB\lVert F_{k-1}(x)\rVert_\infty^3+B.
\end{aligned} ∥ F k ( x ) ∥ ∞ = ∥ W F ( k ) η 3 ( F k − 1 ( x )) + b F ( k ) ∥ ∞ ≤ ∥ W F ( k ) ∥ ∞ ∥ η 3 ( F k − 1 ( x )) ∥ ∞ + ∥ b F ( k ) ∥ ∞ ≤ W ∥ W F ( k ) ∥ ∞ , ∞ ∥ F k − 1 ( x ) ∥ ∞ 3 + B ≤ W B ∥ F k − 1 ( x ) ∥ ∞ 3 + B .
Using inductive hypothesis, we can further deduce that:
∥ F k ( x ) ∥ ∞ ≤ W B × W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − 3 k + 6 + B ≤ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 3 2 − 3 k + 6 + B ∨ d ≤ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 [ 2 3 k − 3 2 − 3 k + 6 + 1 ] ≤ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 3 2 − k + 2 ( k ≥ 3 ) = W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − k + 1 . \begin{aligned}
\lVert F_k(x)\rVert_\infty
&\leq WB\times W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-3k+6}+B\\
&\leq W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-3}{2}-3k+6}+B\vee d\\
&\leq W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}\left[2^{\frac{3^k-3}{2}-3k+6}+1\right]\\
&\leq W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-3}{2}-k+2}\quad(k\geq3)\\
&=W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-1}{2}-k+1}.
\end{aligned} ∥ F k ( x ) ∥ ∞ ≤ W B × W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − 3 k + 6 + B ≤ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 3 − 3 k + 6 + B ∨ d ≤ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 [ 2 2 3 k − 3 − 3 k + 6 + 1 ] ≤ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 3 − k + 2 ( k ≥ 3 ) = W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − k + 1 .
Taking supremum with respect to x ∈ Ω x\in\Omega x ∈ Ω and F k ∈ Φ k ( L , W , S , B ) F_k\in\Phi_k(L,W,S,B) F k ∈ Φ k ( L , W , S , B ) on the LHS implies that the given upper bound also holds for k k k . By induction, the claim is proved.
We also need to show that the ReLU3 activation function is a Lipschitzness functions over a bounded domain.
Lemma A.16. For any k ∈ Z + k\in\mathbb{Z}^{+} k ∈ Z + , consider the k k k -ReLU activation function η k \eta_k η k defined on some bounded domain D ⊂ R d D\subset\mathbb{R}^{d} D ⊂ R d (i.e, sup x ∈ D ∥ x ∥ ∞ ≤ C \sup_{x\in D}\lVert x\rVert_\infty\leq C sup x ∈ D ∥ x ∥ ∞ ≤ C for some C > 0 C>0 C > 0 ). Then we have that for any x , y ∈ D x,y\in D x , y ∈ D , the following inequalities hold:
∥ η 1 ( x ) − η 1 ( y ) ∥ ∞ ≤ ∥ x − y ∥ ∞ , ∥ η 2 ( x ) − η 2 ( y ) ∥ ∞ ≤ 2 C ∥ x − y ∥ ∞ , ∥ η 3 ( x ) − η 3 ( y ) ∥ ∞ ≤ 3 C 2 ∥ x − y ∥ ∞ . \begin{aligned}
\lVert\eta_1(x)-\eta_1(y)\rVert_\infty&\leq\lVert x-y\rVert_\infty,\\
\lVert\eta_2(x)-\eta_2(y)\rVert_\infty&\leq2C\lVert x-y\rVert_\infty,\\
\lVert\eta_3(x)-\eta_3(y)\rVert_\infty&\leq3C^2\lVert x-y\rVert_\infty.
\end{aligned} ∥ η 1 ( x ) − η 1 ( y ) ∥ ∞ ∥ η 2 ( x ) − η 2 ( y ) ∥ ∞ ∥ η 3 ( x ) − η 3 ( y ) ∥ ∞ ≤ ∥ x − y ∥ ∞ , ≤ 2 C ∥ x − y ∥ ∞ , ≤ 3 C 2 ∥ x − y ∥ ∞ .
Proof. This is because ∣ ∇ η 1 ( x ) ∣ = ∣ max { 1 , 0 } ∣ = 1 \lvert\nabla\eta_1(x)\rvert=\lvert\max\{1,0\}\rvert=1 ∣ ∇ η 1 ( x )∣ = ∣ max { 1 , 0 }∣ = 1 , ∣ ∇ η 2 ( x ) ∣ = ∣ 2 max { x , 0 } ∣ ≤ 2 C \lvert\nabla\eta_2(x)\rvert=\lvert2\max\{x,0\}\rvert\leq2C ∣ ∇ η 2 ( x )∣ = ∣ 2 max { x , 0 }∣ ≤ 2 C and ∣ ∇ η 3 ( x ) ∣ = ∣ 3 max { x , 0 } 2 ∣ ≤ 3 C 2 \lvert\nabla\eta_3(x)\rvert=\lvert3\max\{x,0\}^2\rvert\leq3C^2 ∣ ∇ η 3 ( x )∣ = ∣ 3 max { x , 0 } 2 ∣ ≤ 3 C 2 .
Lemma A.17. (Relation between the covering number of DNN space and parameter space ) For any 1 ≤ k ≤ L 1 \le k \le L 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L , W , S , B ) F_k,G_k \in\Phi_k(L,W,S,B) F k , G k ∈ Φ k ( L , W , S , B ) are given by:
F k ( x ) : = ( W F ( k ) η 3 ( ⋅ ) + b F ( k ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) , F_k(x) := \left(\mathcal{W}^{(k)}_F \eta_3(\cdot)+b_F^{(k)}\right)\cdots\left(\mathcal{W}^{(1)}_F x+b_F^{(1)}\right), F k ( x ) := ( W F ( k ) η 3 ( ⋅ ) + b F ( k ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) , G k ( x ) : = ( W G ( k ) η 3 ( ⋅ ) + b G ( k ) ) ⋯ ( W G ( 1 ) x + b G ( 1 ) ) . G_k(x) := \left(\mathcal{W}^{(k)}_G \eta_3(\cdot)+b_G^{(k)}\right)\cdots\left(\mathcal{W}^{(1)}_G x+b_G^{(1)}\right). G k ( x ) := ( W G ( k ) η 3 ( ⋅ ) + b G ( k ) ) ⋯ ( W G ( 1 ) x + b G ( 1 ) ) . Furthermore, assume that the ∥ ⋅ ∥ ∞ \|\cdot\|_\infty ∥ ⋅ ∥ ∞ norm of the distance between the parameter spaces of F k F_k F k and G k G_k G k is uniformly upper bounded by δ \delta δ , i.e
∥ W F ( l ) − W G ( l ) ∥ ∞ , ∞ ≤ δ , ∥ b F ( l ) − b G ( l ) ∥ ∞ ≤ δ , ( ∀ 1 ≤ l ≤ k ) . (61) \|\mathcal{W}^{(l)}_F-\mathcal{W}^{(l)}_G\|_{\infty,\infty}\le\delta,\quad\|b_F^{(l)}-b_G^{(l)}\|_\infty\le\delta,\quad(\forall\ 1\le l\le k). \tag*{(61)} ∥ W F ( l ) − W G ( l ) ∥ ∞ , ∞ ≤ δ , ∥ b F ( l ) − b G ( l ) ∥ ∞ ≤ δ , ( ∀ 1 ≤ l ≤ k ) . (61) Then we have:
sup x ∈ Ω ∥ F k ( x ) − G k ( x ) ∥ ∞ ≤ δ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − k + 1 3 k − 1 . (62) \sup_{x\in\Omega}\|F_k(x)-G_k(x)\|_\infty\le\delta W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-1}{2}-k+1}3^{k-1}. \tag*{(62)} x ∈ Ω sup ∥ F k ( x ) − G k ( x ) ∥ ∞ ≤ δ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − k + 1 3 k − 1 . (62) Proof. Let’s prove the claim by using induction on k k k .
Base Case: When k = 1 k=1 k = 1 , we have that for any x ∈ Ω x\in\Omega x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L , W , S , B ) F_1,G_1\in\Phi_1(L,W,S,B) F 1 , G 1 ∈ Φ 1 ( L , W , S , B ) satisfying constraint A.20, the following holds:
∥ F 1 ( x ) − G 1 ( x ) ∥ ∞ = ∥ W F ( 1 ) x + b F ( 1 ) − W G ( 1 ) x − b G ( 1 ) ∥ ∞ ≤ ∥ W F ( 1 ) − W G ( 1 ) ∥ ∞ , ∞ ∥ x ∥ ∞ + ∥ b F ( 1 ) − b G ( 1 ) ∥ ∞ ≤ δ d + δ = δ ( d + 1 ) ≤ 2 δ ( B ∨ d ) ≤ 2 δ ( B ∨ d ) 2 . (63) \begin{aligned}
\|F_1(x)-G_1(x)\|_\infty
&=\|\mathcal{W}^{(1)}_F x+b_F^{(1)}-\mathcal{W}^{(1)}_Gx-b_G^{(1)}\|_\infty\\
&\le\|\mathcal{W}^{(1)}_F-\mathcal{W}^{(1)}_G\|_{\infty,\infty}\|x\|_\infty+\|b_F^{(1)}-b_G^{(1)}\|_\infty\\
&\le\delta d+\delta=\delta(d+1)\le2\delta(B\vee d)\le2\delta(B\vee d)^2. \tag*{(63)}
\end{aligned} ∥ F 1 ( x ) − G 1 ( x ) ∥ ∞ = ∥ W F ( 1 ) x + b F ( 1 ) − W G ( 1 ) x − b G ( 1 ) ∥ ∞ ≤ ∥ W F ( 1 ) − W G ( 1 ) ∥ ∞ , ∞ ∥ x ∥ ∞ + ∥ b F ( 1 ) − b G ( 1 ) ∥ ∞ ≤ δ d + δ = δ ( d + 1 ) ≤ 2 δ ( B ∨ d ) ≤ 2 δ ( B ∨ d ) 2 . (63) When k = 2 k=2 k = 2 , we have that for any x ∈ Ω x\in\Omega x ∈ Ω and any F 2 , G 2 ∈ Φ 2 ( L , W , S , B ) F_2,G_2\in\Phi_2(L,W,S,B) F 2 , G 2 ∈ Φ 2 ( L , W , S , B ) satisfying constraint A.20, the following inequality holds:
∥ F 2 ( x ) − G 2 ( x ) ∥ ∞ = ∥ W F ( 2 ) η 3 ( F 1 ( x ) ) + b F ( 2 ) − W G ( 2 ) η 3 ( G 1 ( x ) ) − b G ( 2 ) ∥ ∞ ≤ ∥ W F ( 2 ) η 3 ( F 1 ( x ) ) − W G ( 2 ) η 3 ( G 1 ( x ) ) ∥ ∞ + ∥ b F ( 2 ) − b G ( 2 ) ∥ ∞ ≤ ∥ W F ( 2 ) η 3 ( F 1 ( x ) ) − W G ( 2 ) η 3 ( F 1 ( x ) ) ∥ ∞ + ∥ W G ( 2 ) η 3 ( F 1 ( x ) ) − W G ( 2 ) η 3 ( G 1 ( x ) ) ∥ ∞ + δ . \begin{aligned}
\|F_2(x)-G_2(x)\|_\infty
&=\|\mathcal{W}^{(2)}_F\eta_3(F_1(x))+b_F^{(2)}-\mathcal{W}^{(2)}_G\eta_3(G_1(x))-b_G^{(2)}\|_\infty\\
&\le\|\mathcal{W}^{(2)}_F\eta_3(F_1(x))-\mathcal{W}^{(2)}_G\eta_3(G_1(x))\|_\infty+\|b_F^{(2)}-b_G^{(2)}\|_\infty\\
&\le\|\mathcal{W}^{(2)}_F\eta_3(F_1(x))-\mathcal{W}^{(2)}_G\eta_3(F_1(x))\|_\infty\\
&\quad+\|\mathcal{W}^{(2)}_G\eta_3(F_1(x))-\mathcal{W}^{(2)}_G\eta_3(G_1(x))\|_\infty+\delta.
\end{aligned} ∥ F 2 ( x ) − G 2 ( x ) ∥ ∞ = ∥ W F ( 2 ) η 3 ( F 1 ( x )) + b F ( 2 ) − W G ( 2 ) η 3 ( G 1 ( x )) − b G ( 2 ) ∥ ∞ ≤ ∥ W F ( 2 ) η 3 ( F 1 ( x )) − W G ( 2 ) η 3 ( G 1 ( x )) ∥ ∞ + ∥ b F ( 2 ) − b G ( 2 ) ∥ ∞ ≤ ∥ W F ( 2 ) η 3 ( F 1 ( x )) − W G ( 2 ) η 3 ( F 1 ( x )) ∥ ∞ + ∥ W G ( 2 ) η 3 ( F 1 ( x )) − W G ( 2 ) η 3 ( G 1 ( x )) ∥ ∞ + δ . By applying the upper bound proved in equation A.19, we can upper bound the first part ∥ W F ( 2 ) η 3 ( F 1 ( x ) ) − W G ( 2 ) η 3 ( F 1 ( x ) ) ∥ ∞ \|\mathcal{W}^{(2)}_F\eta_3(F_1(x))-\mathcal{W}^{(2)}_G\eta_3(F_1(x))\|_\infty ∥ W F ( 2 ) η 3 ( F 1 ( x )) − W G ( 2 ) η 3 ( F 1 ( x )) ∥ ∞ by:
∥ W F ( 2 ) η 3 ( F 1 ( x ) ) − W G ( 2 ) η 3 ( F 1 ( x ) ) ∥ ∞ ≤ ∥ W F ( 2 ) − W G ( 2 ) ∥ ∞ , ∞ ∥ η 3 ( F 1 ( x ) ) ∥ ∞ ≤ W δ ∥ F 1 ( x ) ∥ ∞ 3 ≤ δ W [ 2 ( B ∨ d ) ] 2 3 . \begin{aligned}
\|\mathcal{W}^{(2)}_F\eta_3(F_1(x))-\mathcal{W}^{(2)}_G\eta_3(F_1(x))\|_\infty
&\le\|\mathcal{W}^{(2)}_F-\mathcal{W}^{(2)}_G\|_{\infty,\infty}\|\eta_3(F_1(x))\|_\infty\\
&\le W\delta\|F_1(x)\|_\infty^3\le\delta W[2(B\vee d)]^2{}^3.
\end{aligned} ∥ W F ( 2 ) η 3 ( F 1 ( x )) − W G ( 2 ) η 3 ( F 1 ( x )) ∥ ∞ ≤ ∥ W F ( 2 ) − W G ( 2 ) ∥ ∞ , ∞ ∥ η 3 ( F 1 ( x )) ∥ ∞ ≤ W δ ∥ F 1 ( x ) ∥ ∞ 3 ≤ δ W [ 2 ( B ∨ d ) ] 2 3 . By applying the Lipschitz condition proved in Lemma A.16 and the bound proved in equation A.22, we can further upper bound the second part ∥ W G ( 2 ) η 3 ( F 1 ( x ) ) − W G ( 2 ) η 3 ( G 1 ( x ) ) ∥ ∞ \|\mathcal{W}^{(2)}_G\eta_3(F_1(x))-\mathcal{W}^{(2)}_G\eta_3(G_1(x))\|_\infty ∥ W G ( 2 ) η 3 ( F 1 ( x )) − W G ( 2 ) η 3 ( G 1 ( x )) ∥ ∞ by:
∥ W G ( 2 ) η 3 ( F 1 ( x ) ) − W G ( 2 ) η 3 ( G 1 ( x ) ) ∥ ∞ ≤ ∥ W G ( 2 ) ∥ ∞ ∥ η 3 ( F 1 ( x ) ) − η 3 ( G 1 ( x ) ) ∥ ∞ ≤ W B × 3 sup F 1 ∈ Φ 1 ( L , W , S , B ) ∥ F 1 ( x ) ∥ ∞ 2 × ∥ F 1 ( x ) − G 1 ( x ) ∥ ∞ ≤ W B × 3 [ 2 ( B ∨ d ) 2 ] 2 × 2 δ ( B ∨ d ) ≤ 24 δ W ( B ∨ d ) 6 . \begin{aligned}
\|\mathcal{W}^{(2)}_G\eta_3(F_1(x))-\mathcal{W}^{(2)}_G\eta_3(G_1(x))\|_\infty
&\le\|\mathcal{W}^{(2)}_G\|_\infty\|\eta_3(F_1(x))-\eta_3(G_1(x))\|_\infty\\
&\le WB\times3\sup_{F_1\in\Phi_1(L,W,S,B)}\|F_1(x)\|_\infty^2\times\|F_1(x)-G_1(x)\|_\infty\\
&\le WB\times3[2(B\vee d)^2]^2\times2\delta(B\vee d)\\
&\le24\delta W(B\vee d)^6.
\end{aligned} ∥ W G ( 2 ) η 3 ( F 1 ( x )) − W G ( 2 ) η 3 ( G 1 ( x )) ∥ ∞ ≤ ∥ W G ( 2 ) ∥ ∞ ∥ η 3 ( F 1 ( x )) − η 3 ( G 1 ( x )) ∥ ∞ ≤ W B × 3 F 1 ∈ Φ 1 ( L , W , S , B ) sup ∥ F 1 ( x ) ∥ ∞ 2 × ∥ F 1 ( x ) − G 1 ( x ) ∥ ∞ ≤ W B × 3 [ 2 ( B ∨ d ) 2 ] 2 × 2 δ ( B ∨ d ) ≤ 24 δ W ( B ∨ d ) 6 .
Summing the two upper bounds above yields:
∥ F 2 ( x ) − G 2 ( x ) ∥ ∞ ≤ 8 δ W ( B ∨ d ) 6 + 24 δ W ( B ∨ d ) 6 + δ ≤ 24 δ W ( B ∨ d ) 7 . \left\lVert F_2(x)-G_2(x)\right\rVert_\infty\le8\delta W(B\vee d)^6+24\delta W(B\vee d)^6+\delta\le24\delta W(B\vee d)^7. ∥ F 2 ( x ) − G 2 ( x ) ∥ ∞ ≤ 8 δ W ( B ∨ d ) 6 + 24 δ W ( B ∨ d ) 6 + δ ≤ 24 δ W ( B ∨ d ) 7 .
where we again use the assumption d ≥ 2 d\ge2 d ≥ 2 in the last step. Inductive Step: Now we assume that the claim has been proved for k − 1 k-1 k − 1 , where k ≥ 3 k\ge3 k ≥ 3 . For any x ∈ Ω x\in\Omega x ∈ Ω and F k ∈ Φ k ( L , W , S , B ) F_k\in\Phi_k(L,W,S,B) F k ∈ Φ k ( L , W , S , B ) , we have that:
∥ F k ( x ) − G k ( x ) ∥ ∞ = ∥ W F ( k ) η 3 ( F k − 1 ( x ) ) + b F ( k ) − W G ( k ) η 3 ( G k − 1 ( x ) ) − b G ( k ) ∥ ∞ ≤ ∥ W F ( k ) η 3 ( F k − 1 ( x ) ) − W G ( k ) η 3 ( G k − 1 ( x ) ) ∥ ∞ + ∥ b F ( k ) − b G ( k ) ∥ ∞ ≤ ∥ W F ( k ) η 3 ( F k − 1 ( x ) ) − W G ( k ) η 3 ( G k − 1 ( x ) ) ∥ ∞ + δ . \begin{aligned}
\left\lVert F_k(x)-G_k(x)\right\rVert_\infty
&=\left\lVert W_F^{(k)}\eta_3(F_{k-1}(x))+b_F^{(k)}-W_G^{(k)}\eta_3(G_{k-1}(x))-b_G^{(k)}\right\rVert_\infty\\
&\le\left\lVert W_F^{(k)}\eta_3(F_{k-1}(x))-W_G^{(k)}\eta_3(G_{k-1}(x))\right\rVert_\infty+\left\lVert b_F^{(k)}-b_G^{(k)}\right\rVert_\infty\\
&\le\left\lVert W_F^{(k)}\eta_3(F_{k-1}(x))-W_G^{(k)}\eta_3(G_{k-1}(x))\right\rVert_\infty+\delta.
\end{aligned} ∥ F k ( x ) − G k ( x ) ∥ ∞ = W F ( k ) η 3 ( F k − 1 ( x )) + b F ( k ) − W G ( k ) η 3 ( G k − 1 ( x )) − b G ( k ) ∞ ≤ W F ( k ) η 3 ( F k − 1 ( x )) − W G ( k ) η 3 ( G k − 1 ( x )) ∞ + b F ( k ) − b G ( k ) ∞ ≤ W F ( k ) η 3 ( F k − 1 ( x )) − W G ( k ) η 3 ( G k − 1 ( x )) ∞ + δ .
Applying triangle inequality helps us upper bound the first term above as follows:
∥ W F ( k ) η 3 ( F k − 1 ( x ) ) − W G ( k ) η 3 ( G k − 1 ( x ) ) ∥ ∞ ≤ ∥ W F ( k ) η 3 ( F k − 1 ( x ) ) − W G ( k ) η 3 ( F k − 1 ( x ) ) ∥ ∞ + ∥ W G ( k ) η 3 ( F k − 1 ( x ) ) − W G ( k ) η 3 ( G k − 1 ( x ) ) ∥ ∞ ≤ ∥ W F ( k ) − W G ( k ) ∥ ∞ ∥ η 3 ( F k − 1 ( x ) ) ∥ ∞ + ∥ W G ( k ) ∥ ∞ ∥ η 3 ( F k − 1 ( x ) ) − η 3 ( G k − 1 ( x ) ) ∥ ∞ ≤ δ W ∥ F k − 1 ( x ) ∥ ∞ 3 + B W ∥ η 3 ( F k − 1 ( x ) ) − η 3 ( G k − 1 ( x ) ) ∥ ∞ . \begin{aligned}
\left\lVert W_F^{(k)}\eta_3(F_{k-1}(x))-W_G^{(k)}\eta_3(G_{k-1}(x))\right\rVert_\infty
&\le\left\lVert W_F^{(k)}\eta_3(F_{k-1}(x))-W_G^{(k)}\eta_3(F_{k-1}(x))\right\rVert_\infty\\
&\quad+\left\lVert W_G^{(k)}\eta_3(F_{k-1}(x))-W_G^{(k)}\eta_3(G_{k-1}(x))\right\rVert_\infty\\
&\le\left\lVert W_F^{(k)}-W_G^{(k)}\right\rVert_\infty\left\lVert\eta_3(F_{k-1}(x))\right\rVert_\infty\\
&\quad+\left\lVert W_G^{(k)}\right\rVert_\infty\left\lVert\eta_3(F_{k-1}(x))-\eta_3(G_{k-1}(x))\right\rVert_\infty\\
&\le\delta W\left\lVert F_{k-1}(x)\right\rVert_\infty^3+BW\left\lVert\eta_3(F_{k-1}(x))-\eta_3(G_{k-1}(x))\right\rVert_\infty.
\end{aligned} W F ( k ) η 3 ( F k − 1 ( x )) − W G ( k ) η 3 ( G k − 1 ( x )) ∞ ≤ W F ( k ) η 3 ( F k − 1 ( x )) − W G ( k ) η 3 ( F k − 1 ( x )) ∞ + W G ( k ) η 3 ( F k − 1 ( x )) − W G ( k ) η 3 ( G k − 1 ( x )) ∞ ≤ W F ( k ) − W G ( k ) ∞ ∥ η 3 ( F k − 1 ( x )) ∥ ∞ + W G ( k ) ∞ ∥ η 3 ( F k − 1 ( x )) − η 3 ( G k − 1 ( x )) ∥ ∞ ≤ δ W ∥ F k − 1 ( x ) ∥ ∞ 3 + B W ∥ η 3 ( F k − 1 ( x )) − η 3 ( G k − 1 ( x )) ∥ ∞ .
From Lemma A.15, we can upper bound the first term δ W ∥ F k − 1 ( x ) ∥ ∞ 3 \delta W\left\lVert F_{k-1}(x)\right\rVert_\infty^3 δ W ∥ F k − 1 ( x ) ∥ ∞ 3 by:
δ W ∥ F k − 1 ( x ) ∥ ∞ 3 ≤ δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − 3 k + 6 . \delta W\left\lVert F_{k-1}(x)\right\rVert_\infty^3\le\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-3k+6}. δ W ∥ F k − 1 ( x ) ∥ ∞ 3 ≤ δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − 3 k + 6 .
Moreover, applying Lemma A.16 and the inductive hypothesis let us upper bound the second term B W ∥ η 3 ( F k − 1 ( x ) ) − η 3 ( G k − 1 ( x ) ) ∥ ∞ BW\left\lVert\eta_3(F_{k-1}(x))-\eta_3(G_{k-1}(x))\right\rVert_\infty B W ∥ η 3 ( F k − 1 ( x )) − η 3 ( G k − 1 ( x )) ∥ ∞ as follows:
B W ∥ η 3 ( F k − 1 ( x ) ) − η 3 ( G k − 1 ( x ) ) ∥ ∞ ≤ B W × 3 sup x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) ∥ F k − 1 ( x ) ∥ ∞ 2 × ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ≤ 3 B W × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ≤ 3 B W × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × δ W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 ≤ 3 k − 1 δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − 3 k + 5 . \begin{aligned}
BW\left\lVert\eta_3(F_{k-1}(x))-\eta_3(G_{k-1}(x))\right\rVert_\infty
&\le BW\times3\sup_{\substack{x\in\Omega,\ F_{k-1}\in\Phi_{k-1}(L,W,S,B)}}\left\lVert F_{k-1}(x)\right\rVert_\infty^2\times\left\lVert F_{k-1}(x)-G_{k-1}(x)\right\rVert_\infty\\
&\le3BW\times W^{3^{k-2}-1}(B\vee d)^{5\cdot3^{k-2}-1}2^{3^{k-1}-1-2k+4}\left\lVert F_{k-1}(x)-G_{k-1}(x)\right\rVert_\infty\\
&\le3BW\times W^{3^{k-2}-1}(B\vee d)^{5\cdot3^{k-2}-1}2^{3^{k-1}-1-2k+4}\\
&\quad\times\delta W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}\\
&\le3^{k-1}\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-3k+5}.
\end{aligned} B W ∥ η 3 ( F k − 1 ( x )) − η 3 ( G k − 1 ( x )) ∥ ∞ ≤ B W × 3 x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) sup ∥ F k − 1 ( x ) ∥ ∞ 2 × ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ≤ 3 B W × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ≤ 3 B W × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × δ W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 ≤ 3 k − 1 δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − 3 k + 5 .
Combining the two upper bounds derived above yields:
∥ F k ( x ) − G k ( x ) ∥ ∞ ≤ δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − 3 k + 6 + 3 k − 1 δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − 3 k + 5 + δ ≤ δ 3 k W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − k + 1 , \begin{aligned}
\left\lVert F_k(x)-G_k(x)\right\rVert_\infty
&\le\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-3k+6}\\
&\quad+3^{k-1}\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-3k+5}+\delta\\
&\le\delta3^k W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-k+1},
\end{aligned} ∥ F k ( x ) − G k ( x ) ∥ ∞ ≤ δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − 3 k + 6 + 3 k − 1 δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − 3 k + 5 + δ ≤ δ 3 k W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − k + 1 ,
where the last inequality above follows from k ≥ 3 k\ge3 k ≥ 3 . Taking supremum with respect to x ∈ Ω x\in\Omega x ∈ Ω on the LHS implies the given upper bound also holds for k k k . By induction, the claim is proved.■
Theorem A.18. (Bounding the DNN space covering number ) Fix some sufficiently large N ∈ Z + N \in\mathbb{Z}^{+} N ∈ Z + . Consider a Deep Neural Network space Φ ( L , W , S , B ) \Phi(L,W,S,B) Φ ( L , W , S , B ) with L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( N ) W=O(N) W = O ( N ) , S = O ( N ) S=O(N) S = O ( N ) and B = O ( N ) B=O(N) B = O ( N ) . Then the log value of the covering number of this DNN space with respect to the inf-norm ∥ F ( x ) ∥ ∞ : = sup x ∈ Ω ∣ F ( x ) ∣ \lVert F(x)\rVert_{\infty}:=\sup_{x\in\Omega}|F(x)| ∥ F ( x ) ∥ ∞ := sup x ∈ Ω ∣ F ( x ) ∣ , which is denoted by N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) \mathcal{N}(\delta,\Phi(L,W,S,B),\lVert\cdot\rVert_{\infty}) N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) , can be upper bounded by:
log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) = O ( S [ log ( δ − 1 ) + 3 L log ( W B ) ] ) . (64) \log\mathcal{N}(\delta,\Phi(L,W,S,B),\lVert\cdot\rVert_{\infty})=O\left(S\left[\log(\delta^{-1})+3^{L}\log(WB)\right]\right). \tag*{(64)} log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) = O ( S [ log ( δ − 1 ) + 3 L log ( W B ) ] ) . (64)
Proof. We firstly fix a sparsity pattern (i.e, the locations of the non-zero entries are fixed). By picking k = L k=L k = L in Lemma A.17, we get the following upper bound on the covering number with respect to ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_{\infty} ∥ ⋅ ∥ ∞ :
( δ 3 L − 1 W 3 L − 1 − 1 2 ( B ∨ d ) 5 × 3 L − 1 − 1 2 2 3 L − 1 2 − L + 2 ) − S . \left(\frac{\delta}{3^{L-1}W^{\frac{3^{L-1}-1}{2}}(B\vee d)^{\frac{5\times3^{L-1}-1}{2}}2^{\frac{3^{L-1}}{2}-L+2}}\right)^{-S}. ( 3 L − 1 W 2 3 L − 1 − 1 ( B ∨ d ) 2 5 × 3 L − 1 − 1 2 2 3 L − 1 − L + 2 δ ) − S .
Furthermore, note that the number of feasible configurations is upper bounded by ( ( W + 1 ) L S ) ≤ ( W + 1 ) L S \binom{(W+1)^L}{S}\leq(W+1)^{LS} ( S ( W + 1 ) L ) ≤ ( W + 1 ) L S . [63 , 18 ] Plug in the previous inequality and yields:
log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≤ log [ ( W + 1 ) L S ( δ 3 L − 1 W 3 L − 1 − 1 2 ( B ∨ d ) 5 × 3 L − 1 − 1 2 2 3 L − 1 2 − L + 1 ) − S ] ≤ S log [ δ − 1 ( W + 1 ) L 3 L − 1 W 3 L − 1 − 1 2 ( B ∨ d ) 5 × 3 L − 1 − 1 2 2 3 L 2 − L + 1 ] \begin{aligned}
\log\mathcal{N}(\delta,\Phi(L,W,S,B),\lVert\cdot\rVert_{\infty})\leq\log\left[(W+1)^{LS}\left(\frac{\delta}{3^{L-1}W^{\frac{3^{L-1}-1}{2}}(B\vee d)^{\frac{5\times3^{L-1}-1}{2}}2^{\frac{3^{L-1}}{2}-L+1}}\right)^{-S}\right] \\
&\leq S\log\left[\delta^{-1}(W+1)^L3^{L-1}W^{\frac{3^{L-1}-1}{2}}(B\vee d)^{\frac{5\times3^{L-1}-1}{2}}2^{\frac{3^{L}}{2}-L+1}\right]
\end{aligned} log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≤ log ( W + 1 ) L S ( 3 L − 1 W 2 3 L − 1 − 1 ( B ∨ d ) 2 5 × 3 L − 1 − 1 2 2 3 L − 1 − L + 1 δ ) − S ≤ S log [ δ − 1 ( W + 1 ) L 3 L − 1 W 2 3 L − 1 − 1 ( B ∨ d ) 2 5 × 3 L − 1 − 1 2 2 3 L − L + 1 ]
≲ S [ log ( δ − 1 ) + L log ( 3 W ) + 3 L log ( W ( B ∨ d ) ) + 3 L log 2 ] . \lesssim S\left[\log(\delta^{-1})+L\log(3W)+3^L\log(W(B\vee d))+3^L\log2\right]. ≲ S [ log ( δ − 1 ) + L log ( 3 W ) + 3 L log ( W ( B ∨ d )) + 3 L log 2 ] .
Note that here the dimension d d d is some constant. Thus, by plugging in thee given magnitudes L = O ( 1 ) L=O(1) L = O ( 1 ) , W = O ( N ) W=O(N) W = O ( N ) , S = O ( N ) S=O(N) S = O ( N ) and B = O ( N ) B=O(N) B = O ( N ) , we can further deduce that:
log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≲ S [ log ( δ − 1 ) + 3 L log ( W B ) ] . \log\mathcal{N}(\delta,\Phi(L,W,S,B),\lVert\cdot\rVert_{\infty})\lesssim S\left[\log(\delta^{-1})+3^L\log(WB)\right]. log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≲ S [ log ( δ − 1 ) + 3 L log ( W B ) ] .
This finishes our proof.
Now let’s consider upper bounding the covering number of the l 2 l_2 l 2 norm of the sparse Deep Neural Networks’ gradients. Note that for any 1 ≤ k ≤ L − 1 1\leq k\leq L-1 1 ≤ k ≤ L − 1 , any k k k –ReLU3 Deep Neural Network F k ∈ Φ k ( L , W , S , B ) F_k\in\Phi_k(L,W,S,B) F k ∈ Φ k ( L , W , S , B ) is a map from R d \mathbb{R}^d R d to R W \mathbb{R}^W R W . For any 1 ≤ l ≤ W 1\leq l\leq W 1 ≤ l ≤ W , we use F k , l ( x ) F_{k,l}(x) F k , l ( x ) to denote the l l l -th component of the map F k F_k F k . This helps us write the map F k ( x ) F_k(x) F k ( x ) and its Jacobian matrix J [ F k ] ( x ) J[F_k](x) J [ F k ] ( x ) explicitly as:
F k ( x ) = [ F k , 1 ( x ) , F k , 2 ( x ) , ⋯ , F k , W ( x ) ] T ∈ R W . F_k(x)=[F_{k,1}(x),F_{k,2}(x),\cdots,F_{k,W}(x)]^{T}\in\mathbb{R}^{W}. F k ( x ) = [ F k , 1 ( x ) , F k , 2 ( x ) , ⋯ , F k , W ( x ) ] T ∈ R W .
J [ F k ] ( x ) = [ ∂ ∂ x 1 F k , 1 ( x ) ∂ ∂ x 2 F k , 1 ( x ) ⋯ ∂ ∂ x d F k , 1 ( x ) ∂ ∂ x 1 F k , 2 ( x ) ∂ ∂ x 2 F k , 2 ( x ) ⋯ ∂ ∂ x d F k , 2 ( x ) ⋯ ⋯ ⋱ ⋯ ∂ ∂ x 1 F k , W ( x ) ∂ ∂ x 2 F k , W ( x ) ⋯ ∂ ∂ x d F k , W ( x ) ] ∈ R W × d . J[F_k](x)=
\begin{bmatrix}
\frac{\partial}{\partial x_1}F_{k,1}(x)&\frac{\partial}{\partial x_2}F_{k,1}(x)&\cdots&\frac{\partial}{\partial x_d}F_{k,1}(x)\\
\frac{\partial}{\partial x_1}F_{k,2}(x)&\frac{\partial}{\partial x_2}F_{k,2}(x)&\cdots&\frac{\partial}{\partial x_d}F_{k,2}(x)\\
\cdots&\cdots&\ddots&\cdots\\
\frac{\partial}{\partial x_1}F_{k,W}(x)&\frac{\partial}{\partial x_2}F_{k,W}(x)&\cdots&\frac{\partial}{\partial x_d}F_{k,W}(x)
\end{bmatrix}\in\mathbb{R}^{W\times d}. J [ F k ] ( x ) = ∂ x 1 ∂ F k , 1 ( x ) ∂ x 1 ∂ F k , 2 ( x ) ⋯ ∂ x 1 ∂ F k , W ( x ) ∂ x 2 ∂ F k , 1 ( x ) ∂ x 2 ∂ F k , 2 ( x ) ⋯ ∂ x 2 ∂ F k , W ( x ) ⋯ ⋯ ⋱ ⋯ ∂ x d ∂ F k , 1 ( x ) ∂ x d ∂ F k , 2 ( x ) ⋯ ∂ x d ∂ F k , W ( x ) ∈ R W × d .
In particular, when k = L k=L k = L , we have that any F L ∈ Φ L ( L , W , S , B ) = Φ ( L , W , S , B ) F_L \in\Phi_L(L,W,S,B)=\Phi(L,W,S,B) F L ∈ Φ L ( L , W , S , B ) = Φ ( L , W , S , B ) is a map from R d \mathbb{R}^d R d to R \mathbb{R} R . Thus, its Jacobian can be explicitly written as the following row vector:
J [ F L ] ( x ) = [ ∂ ∂ x 1 F L ( x ) , ∂ ∂ x 2 F L ( x ) , ⋯ , ∂ ∂ x d F L ( x ) ] ∈ R 1 × d . J[F_L](x)=\left[\frac{\partial}{\partial x_1}F_L(x),\frac{\partial}{\partial x_2}F_L(x),\cdots,\frac{\partial}{\partial x_d}F_L(x)\right]\in\mathbb{R}^{1\times d}. J [ F L ] ( x ) = [ ∂ x 1 ∂ F L ( x ) , ∂ x 2 ∂ F L ( x ) , ⋯ , ∂ x d ∂ F L ( x ) ] ∈ R 1 × d .
Lemma A.19. (Upper bound on ∞ \infty ∞ -norm of Jacobian/Gradient of elements in the DNN space ) For any 1 ≤ k ≤ L 1\leq k\leq L 1 ≤ k ≤ L , the following inequality holds:
sup x ∈ Ω , F k ∈ Φ k ( L , W , S , B ) ∥ J [ F k ] ( x ) ∥ ∞ ≤ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − k + 1 3 k − 1 . \sup_{x\in\Omega,F_k\in\Phi_k(L,W,S,B)}\|J[F_k](x)\|_\infty\leq W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-1}{2}-k+1}3^{k-1}. x ∈ Ω , F k ∈ Φ k ( L , W , S , B ) sup ∥ J [ F k ] ( x ) ∥ ∞ ≤ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − k + 1 3 k − 1 .
Proof. We use induction on k k k to prove the claim.
Base case: k = 1 k=1 k = 1 . By the definition of Jacobian matrix, we have that for any x ∈ Ω x\in\Omega x ∈ Ω and any F 1 ∈ Φ 1 ( L , W , S , B ) F_1\in\Phi_1(L,W,S,B) F 1 ∈ Φ 1 ( L , W , S , B ) , the following holds:
∥ J [ F 1 ] ( x ) ∥ ∞ = ∥ W F ( 1 ) ∥ ∞ ≤ d B ≤ 2 ( B ∨ d ) 2 . \|J[F_1](x)\|_\infty=\|\mathcal{W}_F^{(1)}\|_\infty\leq dB\leq2(B\vee d)^2. ∥ J [ F 1 ] ( x ) ∥ ∞ = ∥ W F ( 1 ) ∥ ∞ ≤ d B ≤ 2 ( B ∨ d ) 2 .
Inductive Step: Assume that the claim has been proved for k − 1 k-1 k − 1 , where 2 ≤ k ≤ L 2\leq k\leq L 2 ≤ k ≤ L . For any x ∈ Ω x\in\Omega x ∈ Ω and any F k ∈ Φ k ( L , W , S , B ) F_k\in\Phi_k(L,W,S,B) F k ∈ Φ k ( L , W , S , B ) , by applying the Chain Rule, we can write the Jacobian matrix J [ F k ] ( x ) J[F_k](x) J [ F k ] ( x ) as J [ F k ] ( x ) = W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) J[F_k](x)=\mathcal{W}_F^{(k)}J[\eta_3\circ F_{k-1}](x) J [ F k ] ( x ) = W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) , where the ReLU3 activation function η 3 \eta_3 η 3 is applied to each component F k − 1 , l F_{k-1,l} F k − 1 , l (1 ≤ l ≤ W 1\leq l\leq W 1 ≤ l ≤ W ) of the map F k F_k F k . Then we have the following upper bound:
∥ J [ F k ] ( x ) ∥ ∞ ≤ ∥ W F ( k ) ∥ ∞ ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ W B ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ . (65) \|J[F_k](x)\|_\infty\leq\|\mathcal{W}_F^{(k)}\|_\infty\|J[\eta_3\circ F_{k-1}](x)\|_\infty\leq WB\|J[\eta_3\circ F_{k-1}](x)\|_\infty. \tag*{(65)} ∥ J [ F k ] ( x ) ∥ ∞ ≤ ∥ W F ( k ) ∥ ∞ ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ W B ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ . (65)
Note that the composition η 3 ∘ F k − 1 \eta_3\circ F_{k-1} η 3 ∘ F k − 1 is a map from R d \mathbb{R}^d R d to R W \mathbb{R}^W R W . Hence, the Jacobian matrix J [ η 3 ∘ F k − 1 ] ( x ) J[\eta_3\circ F_{k-1}](x) J [ η 3 ∘ F k − 1 ] ( x ) is of shape R W × d \mathbb{R}^{W\times d} R W × d . Applying the Chain Rule again implies:
∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ = sup 1 ≤ l ≤ W ( ∑ j = 1 d ∣ 3 η 2 ( F k − 1 , l ( x ) ) ∂ F k − 1 , l ( x ) ∂ x j ∣ ) . \|J[\eta_3\circ F_{k-1}](x)\|_\infty=\sup_{1\leq l\leq W}\left(\sum_{j=1}^{d}|3\eta_2(F_{k-1,l}(x))\frac{\partial F_{k-1,l}(x)}{\partial x_j}|\right). ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ = 1 ≤ l ≤ W sup ( j = 1 ∑ d ∣3 η 2 ( F k − 1 , l ( x )) ∂ x j ∂ F k − 1 , l ( x ) ∣ ) .
Furthermore, for any 1 ≤ l ≤ W 1\leq l\leq W 1 ≤ l ≤ W , the summation on the RHS above can be upper bounded by:
∑ j = 1 d ∣ 3 η 2 ( F k − 1 , l ( x ) ) ∂ F k − 1 , l ( x ) ∂ x j ∣ ≤ 3 ∥ F k − 1 ( x ) ∥ ∞ 2 ( ∑ j = 1 d ∣ ∂ ∂ x j F k − 1 , l ( x ) ∣ ) ≤ 3 ∥ F k − 1 ( x ) ∥ ∞ 2 ∥ J [ F k − 1 ] ( x ) ∥ ∞ . \sum_{j=1}^{d}|3\eta_2(F_{k-1,l}(x))\frac{\partial F_{k-1,l}(x)}{\partial x_j}|\leq3\|F_{k-1}(x)\|_\infty^2\left(\sum_{j=1}^{d}\left|\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right|\right)\leq3\|F_{k-1}(x)\|_\infty^2\|J[F_{k-1}](x)\|_\infty. j = 1 ∑ d ∣3 η 2 ( F k − 1 , l ( x )) ∂ x j ∂ F k − 1 , l ( x ) ∣ ≤ 3∥ F k − 1 ( x ) ∥ ∞ 2 ( j = 1 ∑ d ∂ x j ∂ F k − 1 , l ( x ) ) ≤ 3∥ F k − 1 ( x ) ∥ ∞ 2 ∥ J [ F k − 1 ] ( x ) ∥ ∞ .
Now let’s take supremum with respect to l l l and apply the inductive hypothesis and Lemma A.15. This yields:
∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ 3 W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − 2 k + 4 × W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 3 k − 2 = W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − 1 2 − 3 k + 5 3 k − 1 . (66) \begin{aligned}
\|J[\eta_3\circ F_{k-1}](x)\|_\infty
&\leq3W^{3^{k-2}-1}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{3^{k-1}-1-2k+4}\\
&\quad\times W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}3^{k-2}\\
&=W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^{k-1}-1}{2}-3k+5}3^{k-1}. \tag*{(66)}
\end{aligned} ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ 3 W 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 3 k − 2 = W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − 1 − 3 k + 5 3 k − 1 . (66)
By substituting equation A.25 into equation A.24, we can derive the final bound:
∥ J [ F k ] ( x ) ∥ ∞ ≤ W B ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − 3 k + 5 3 k − 1 ≤ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − k + 1 3 k − 1 . \begin{aligned}
\|J[F_k](x)\|_\infty
&\leq WB\|J[\eta_3\circ F_{k-1}](x)\|_\infty\\
&\leq W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^{k-1}}{2}-3k+5}3^{k-1}\\
&\leq W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-1}{2}-k+1}3^{k-1}.
\end{aligned} ∥ J [ F k ] ( x ) ∥ ∞ ≤ W B ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − 3 k + 5 3 k − 1 ≤ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − k + 1 3 k − 1 .
where the last inequality above follows from k ≥ 2 k \ge2 k ≥ 2 . Taking supremum with respect to x ∈ Ω x \in\Omega x ∈ Ω and F k ∈ Φ k ( L , W , S , B ) F_k \in\Phi_k(L,W,S,B) F k ∈ Φ k ( L , W , S , B ) on the LHS implies that the given upper bound also holds for k k k . By induction, the claim is proved.
For the convenience of the following proof, we first prove this lemma for vector 2 2 2 norm and ∞ \infty ∞ norm.
Lemma A.20. Given any two row vectors u , v ∈ R 1 × d \mathbf{u},\mathbf{v} \in\mathbb{R}^{1 \times d} u , v ∈ R 1 × d , we have:
∥ u ∥ − ∥ v ∥ ≤ ∥ u − v ∥ ∞ . \left\lVert\mathbf{u}\right\rVert-\left\lVert\mathbf{v}\right\rVert\leq\left\lVert\mathbf{u}-\mathbf{v}\right\rVert_{\infty}. ∥ u ∥ − ∥ v ∥ ≤ ∥ u − v ∥ ∞ .
Proof. Assume that the two vectors u , v ∈ R d \mathbf{u},\mathbf{v} \in\mathbb{R}^{d} u , v ∈ R d can be explicitly written as u = [ u 1 , u 2 , ⋯ , u d ] \mathbf{u}=[u_1,u_2,\cdots,u_d] u = [ u 1 , u 2 , ⋯ , u d ] and v = [ v 1 , v 2 , ⋯ , v d ] \mathbf{v}=[v_1,v_2,\cdots,v_d] v = [ v 1 , v 2 , ⋯ , v d ] , respectively. By applying Cauchy-Schwarz inequality, we have:
∥ u ∥ − ∥ v ∥ 2 = ∣ ∑ i = 1 d u i 2 − ∑ i = 1 d v i 2 ∣ 2 = ∑ i = 1 d u i 2 + ∑ i = 1 d v i 2 − 2 ∑ i = 1 d u i 2 ∑ i = 1 d v i 2 ≤ ∑ i = 1 d u i 2 + ∑ i = 1 d v i 2 − 2 ∑ i = 1 d u i v i = ∑ i = 1 d ∣ u i − v i ∣ 2 ≤ ( ∑ i = 1 d ∣ u i − v i ∣ ) 2 = ∥ u − v ∥ ∞ 2 . \begin{aligned}
\left\lVert\mathbf{u}\right\rVert-\left\lVert\mathbf{v}\right\rVert^{2}
=\left|\sqrt{\sum_{i=1}^{d}u_i^2}-\sqrt{\sum_{i=1}^{d}v_i^2}\right|^2 \\
&=\sum_{i=1}^{d}u_i^2+\sum_{i=1}^{d}v_i^2-2\sqrt{\sum_{i=1}^{d}u_i^2}\sqrt{\sum_{i=1}^{d}v_i^2} \\
&\leq\sum_{i=1}^{d}u_i^2+\sum_{i=1}^{d}v_i^2-2\sum_{i=1}^{d}u_iv_i
=\sum_{i=1}^{d}|u_i-v_i|^2 \\
&\leq\left(\sum_{i=1}^{d}|u_i-v_i|\right)^2
=\left\lVert\mathbf{u}-\mathbf{v}\right\rVert_{\infty}^2.
\end{aligned} ∥ u ∥ − ∥ v ∥ 2 = i = 1 ∑ d u i 2 − i = 1 ∑ d v i 2 2 = i = 1 ∑ d u i 2 + i = 1 ∑ d v i 2 − 2 i = 1 ∑ d u i 2 i = 1 ∑ d v i 2 ≤ i = 1 ∑ d u i 2 + i = 1 ∑ d v i 2 − 2 i = 1 ∑ d u i v i = i = 1 ∑ d ∣ u i − v i ∣ 2 ≤ ( i = 1 ∑ d ∣ u i − v i ∣ ) 2 = ∥ u − v ∥ ∞ 2 .
Taking the square root on both sides yields the desired inequality.
Then we upper bound the Lipschitz constant of the gradient of the neural network. Given a DNN space Φ ( L , W , S , B ) \Phi(L,W,S,B) Φ ( L , W , S , B ) , we define a corresponding DNN Gradient space ∇ Φ ( L , W , S , B ) \nabla\Phi(L,W,S,B) ∇Φ ( L , W , S , B ) as:
∇ Φ ( L , W , S , B ) : = { ∥ ∇ F ∥ ∣ F ∈ Φ ( L , W , S , B ) } . (67) \nabla\Phi(L,W,S,B):=\{\lVert\nabla F\rVert\mid F\in\Phi(L,W,S,B)\}. \tag*{(67)} ∇Φ ( L , W , S , B ) := {∥ ∇ F ∥ ∣ F ∈ Φ ( L , W , S , B )} . (67)
Lemma A.21. (Relation between the covering number of the DNN Gradient space and parameter space ) For any 1 ≤ k ≤ L 1\leq k\leq L 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L , W , S , B ) F_k,G_k\in\Phi_k(L,W,S,B) F k , G k ∈ Φ k ( L , W , S , B ) are given by:
F k ( x ) : = ( W F ( k ) η 3 ( ⋅ ) + b F ( k ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) , F_k(x):=(W_F^{(k)}\eta_3(\cdot)+b_F^{(k)})\cdots(W_F^{(1)}x+b_F^{(1)}), F k ( x ) := ( W F ( k ) η 3 ( ⋅ ) + b F ( k ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) , G k ( x ) : = ( W G ( k ) η 3 ( ⋅ ) + b G ( k ) ) ⋯ ( W G ( 1 ) x + b G ( 1 ) ) . G_k(x):=(W_G^{(k)}\eta_3(\cdot)+b_G^{(k)})\cdots(W_G^{(1)}x+b_G^{(1)}). G k ( x ) := ( W G ( k ) η 3 ( ⋅ ) + b G ( k ) ) ⋯ ( W G ( 1 ) x + b G ( 1 ) ) . Furthermore, assume that the ∥ ∥ ∞ \lVert\ \rVert_{\infty} ∥ ∥ ∞ norm of the distance between the parameter spaces is uniformly upper bounded by δ \delta δ , i.e
∥ W F ( l ) − W G ( l ) ∥ ∞ , ∞ ≤ δ , ∥ b F ( l ) − b G ( l ) ∥ ∞ ≤ δ , ( ∀ 1 ≤ l ≤ k ) . (68) \lVert W_F^{(l)}-W_G^{(l)}\rVert_{\infty,\infty}\leq\delta,\quad\lVert b_F^{(l)}-b_G^{(l)}\rVert_{\infty}\leq\delta,\quad(\forall\ 1\leq l\leq k). \tag*{(68)} ∥ W F ( l ) − W G ( l ) ∥ ∞ , ∞ ≤ δ , ∥ b F ( l ) − b G ( l ) ∥ ∞ ≤ δ , ( ∀ 1 ≤ l ≤ k ) . (68) Then we have:
sup x ∈ Ω ∥ J [ F k ] ( x ) − J [ G k ] ( x ) ∥ ∞ ≤ δ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 2 2 3 k − 1 2 − k + 1 3 2 k − 2 . (69) \sup_{x\in\Omega}\lVert J[F_k](x)-J[G_k](x)\rVert_{\infty}
\leq\delta W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^k-1}{2}}2^{\frac{3^k-1}{2}-k+1}3^{2k-2}. \tag*{(69)} x ∈ Ω sup ∥ J [ F k ] ( x ) − J [ G k ] ( x ) ∥ ∞ ≤ δ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 2 2 3 k − 1 − k + 1 3 2 k − 2 . (69)
In particular, when k = L k=L k = L , we have:
sup x ∈ Ω ∥ ∥ ∇ F L ( x ) ∥ − ∥ ∇ G L ( x ) ∥ ∥ ≤ δ W 3 L − 1 2 ( B ∨ d ) 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 2 − L + 1 3 2 L − 2 . (70) \sup_{x\in\Omega}\left\lVert\left\lVert\nabla F_L(x)\right\rVert-\left\lVert\nabla G_L(x)\right\rVert\right\rVert\le\delta W^{\frac{3^L-1}{2}}(B\vee d)^{\frac{5\cdot3^{L-1}-1}{2}}2^{\frac{3^L-1}{2}-L+1}3^{2L-2}. \tag*{(70)} x ∈ Ω sup ∥ ∥ ∇ F L ( x ) ∥ − ∥ ∇ G L ( x ) ∥ ∥ ≤ δ W 2 3 L − 1 ( B ∨ d ) 2 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 − L + 1 3 2 L − 2 . (70)
Proof. We use induction on k k k to prove the claim.
Base case: When k = 1 k=1 k = 1 , we have that for any x ∈ Ω x\in\Omega x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L , W , S , B ) F_1,G_1\in\Phi_1(L,W,S,B) F 1 , G 1 ∈ Φ 1 ( L , W , S , B ) , the following holds:
∥ J [ F 1 ] ( x ) − J [ G 1 ] ( x ) ∥ ∞ = ∥ W F ( 1 ) − W G ( 1 ) ∥ ∞ ≤ δ d ≤ 2 δ ( B ∨ d ) 2 . \left\lVert J[F_1](x)-J[G_1](x)\right\rVert_\infty=\left\lVert W_F^{(1)}-W_G^{(1)}\right\rVert_\infty\le\delta d\le2\delta(B\vee d)^2. ∥ J [ F 1 ] ( x ) − J [ G 1 ] ( x ) ∥ ∞ = W F ( 1 ) − W G ( 1 ) ∞ ≤ δ d ≤ 2 δ ( B ∨ d ) 2 .
Inductive Step: assume that the claim has been proved for k − 1 k-1 k − 1 , where 2 ≤ k ≤ L 2\le k\le L 2 ≤ k ≤ L . Then for any x ∈ Ω x\in\Omega x ∈ Ω and F k , G k ∈ Φ k ( L , W , S , B ) F_k,G_k\in\Phi_k(L,W,S,B) F k , G k ∈ Φ k ( L , W , S , B ) satisfying constraint A.27, applying the Chain Rule and triangle inequality help us upper bound the inf-norm ∥ J [ F k ] ( x ) − J [ G k ] ( x ) ∥ ∞ \left\lVert J[F_k](x)-J[G_k](x)\right\rVert_\infty ∥ J [ F k ] ( x ) − J [ G k ] ( x ) ∥ ∞ by:
∥ J [ F k ] ( x ) − J [ G k ] ( x ) ∥ ∞ = ∥ W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) − W G ( k ) J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ ≤ ∥ W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) − W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ + ∥ W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) − W G ( k ) J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ ≤ ∥ W F ( k ) − W G ( k ) ∥ ∞ ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ + ∥ W G ( k ) ∥ ∞ ∥ J [ η 3 ∘ F k − 1 ] ( x ) − J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ ≤ δ W ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ + B W ∥ J [ η 3 ∘ F k − 1 ] ( x ) − J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ . (71) \begin{aligned}
\left\lVert J[F_k](x)-J[G_k](x)\right\rVert_\infty
&=\left\lVert W_F^{(k)}J[\eta_3\circ F_{k-1}](x)-W_G^{(k)}J[\eta_3\circ G_{k-1}](x)\right\rVert_\infty\\
&\le\left\lVert W_F^{(k)}J[\eta_3\circ F_{k-1}](x)-W_F^{(k)}J[\eta_3\circ F_{k-1}](x)\right\rVert_\infty\\
&\quad+\left\lVert W_F^{(k)}J[\eta_3\circ F_{k-1}](x)-W_G^{(k)}J[\eta_3\circ G_{k-1}](x)\right\rVert_\infty\\
&\le\left\lVert W_F^{(k)}-W_G^{(k)}\right\rVert_\infty\left\lVert J[\eta_3\circ F_{k-1}](x)\right\rVert_\infty\\
&\quad+\left\lVert W_G^{(k)}\right\rVert_\infty\left\lVert J[\eta_3\circ F_{k-1}](x)-J[\eta_3\circ G_{k-1}](x)\right\rVert_\infty\\
&\le\delta W\left\lVert J[\eta_3\circ F_{k-1}](x)\right\rVert_\infty+BW\left\lVert J[\eta_3\circ F_{k-1}](x)-J[\eta_3\circ G_{k-1}](x)\right\rVert_\infty. \tag*{(71)}
\end{aligned} ∥ J [ F k ] ( x ) − J [ G k ] ( x ) ∥ ∞ = W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) − W G ( k ) J [ η 3 ∘ G k − 1 ] ( x ) ∞ ≤ W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) − W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) ∞ + W F ( k ) J [ η 3 ∘ F k − 1 ] ( x ) − W G ( k ) J [ η 3 ∘ G k − 1 ] ( x ) ∞ ≤ W F ( k ) − W G ( k ) ∞ ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ + W G ( k ) ∞ ∥ J [ η 3 ∘ F k − 1 ] ( x ) − J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ ≤ δ W ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ + B W ∥ J [ η 3 ∘ F k − 1 ] ( x ) − J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ . (71)
Using equation A.25 helps us upper bound the first term by:
δ W ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 − 1 2 3 k − 1 2 − 3 k + 5 3 k − 1 . (72) \delta W\left\lVert J[\eta_3\circ F_{k-1}](x)\right\rVert_\infty\le\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}-1}2^{\frac{3^k-1}{2}-3k+5}3^{k-1}. \tag*{(72)} δ W ∥ J [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 − 1 2 2 3 k − 1 − 3 k + 5 3 k − 1 . (72)
Note that the two compositions η 3 ∘ F k − 1 \eta_3\circ F_{k-1} η 3 ∘ F k − 1 and η 3 ∘ G k − 1 \eta_3\circ G_{k-1} η 3 ∘ G k − 1 both map from R d \mathbb{R}^d R d to R W \mathbb{R}^W R W . Hence, the two Jacobian matrices J [ η 3 ∘ F k − 1 ] ( x ) J[\eta_3\circ F_{k-1}](x) J [ η 3 ∘ F k − 1 ] ( x ) and J [ η 3 ∘ G k − 1 ] ( x ) J[\eta_3\circ G_{k-1}](x) J [ η 3 ∘ G k − 1 ] ( x ) are of shape R W × d \mathbb{R}^{W\times d} R W × d . Applying the Chain Rule again implies:
∥ J [ η 3 ∘ F k − 1 ] ( x ) − J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ = sup 1 ≤ l ≤ W ( ∑ j = 1 d ∣ 3 η 2 ( F k − 1 , l ( x ) ) ∂ F k − 1 , l ( x ) ∂ x j − 3 η 2 ( G k − 1 , l ( x ) ) ∂ G k − 1 , l ( x ) ∂ x j ∣ ) . \left\lVert J[\eta_3\circ F_{k-1}](x)-J[\eta_3\circ G_{k-1}](x)\right\rVert_\infty=\sup_{1\le l\le W}\left(\sum_{j=1}^{d}\left|3\eta_2(F_{k-1,l}(x))\frac{\partial F_{k-1,l}(x)}{\partial x_j}-3\eta_2(G_{k-1,l}(x))\frac{\partial G_{k-1,l}(x)}{\partial x_j}\right|\right). ∥ J [ η 3 ∘ F k − 1 ] ( x ) − J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ = 1 ≤ l ≤ W sup ( j = 1 ∑ d 3 η 2 ( F k − 1 , l ( x )) ∂ x j ∂ F k − 1 , l ( x ) − 3 η 2 ( G k − 1 , l ( x )) ∂ x j ∂ G k − 1 , l ( x ) ) .
For any 1 ≤ l ≤ W 1\le l\le W 1 ≤ l ≤ W , the summation on the RHS above can be upper bounded by:
∑ j = 1 d ∣ 3 η 2 ( F k − 1 , l ( x ) ) ∂ F k − 1 , l ( x ) ∂ x j − 3 η 2 ( G k − 1 , l ( x ) ) ∂ G k − 1 , l ( x ) ∂ x j ∣ ≤ ∑ j = 1 d ∣ 3 η 2 ( F k − 1 , l ( x ) ) ∂ F k − 1 , l ( x ) ∂ x j − 3 η 2 ( G k − 1 , l ( x ) ) ∂ F k − 1 , l ( x ) ∂ x j ∣ + ∑ j = 1 d ∣ 3 η 2 ( G k − 1 , l ( x ) ) ∂ F k − 1 , l ( x ) ∂ x j − 3 η 2 ( G k − 1 , l ( x ) ) ∂ G k − 1 , l ( x ) ∂ x j ∣ ≤ ∑ j = 1 d ∣ 3 η 2 ( F k − 1 , l ( x ) ) − 3 η 2 ( G k − 1 , l ( x ) ) ∣ ∣ ∂ F k − 1 , l ( x ) ∂ x j ∣ + ∑ j = 1 d ∣ 3 η 2 ( G k − 1 , l ( x ) ) ∣ ∣ ∂ F k − 1 , l ( x ) ∂ x j − ∂ G k − 1 , l ( x ) ∂ x j ∣ . \begin{aligned}
\sum_{j=1}^{d}\left|3\eta_2(F_{k-1,l}(x))\frac{\partial F_{k-1,l}(x)}{\partial x_j}-3\eta_2(G_{k-1,l}(x))\frac{\partial G_{k-1,l}(x)}{\partial x_j}\right|
&\le\sum_{j=1}^{d}\left|3\eta_2(F_{k-1,l}(x))\frac{\partial F_{k-1,l}(x)}{\partial x_j}-3\eta_2(G_{k-1,l}(x))\frac{\partial F_{k-1,l}(x)}{\partial x_j}\right|\\
&\quad+\sum_{j=1}^{d}\left|3\eta_2(G_{k-1,l}(x))\frac{\partial F_{k-1,l}(x)}{\partial x_j}-3\eta_2(G_{k-1,l}(x))\frac{\partial G_{k-1,l}(x)}{\partial x_j}\right|\\
&\le\sum_{j=1}^{d}\left|3\eta_2(F_{k-1,l}(x))-3\eta_2(G_{k-1,l}(x))\right|\left|\frac{\partial F_{k-1,l}(x)}{\partial x_j}\right|\\
&\quad+\sum_{j=1}^{d}\left|3\eta_2(G_{k-1,l}(x))\right|\left|\frac{\partial F_{k-1,l}(x)}{\partial x_j}-\frac{\partial G_{k-1,l}(x)}{\partial x_j}\right|.
\end{aligned} j = 1 ∑ d 3 η 2 ( F k − 1 , l ( x )) ∂ x j ∂ F k − 1 , l ( x ) − 3 η 2 ( G k − 1 , l ( x )) ∂ x j ∂ G k − 1 , l ( x ) ≤ j = 1 ∑ d 3 η 2 ( F k − 1 , l ( x )) ∂ x j ∂ F k − 1 , l ( x ) − 3 η 2 ( G k − 1 , l ( x )) ∂ x j ∂ F k − 1 , l ( x ) + j = 1 ∑ d 3 η 2 ( G k − 1 , l ( x )) ∂ x j ∂ F k − 1 , l ( x ) − 3 η 2 ( G k − 1 , l ( x )) ∂ x j ∂ G k − 1 , l ( x ) ≤ j = 1 ∑ d ∣ 3 η 2 ( F k − 1 , l ( x )) − 3 η 2 ( G k − 1 , l ( x )) ∣ ∂ x j ∂ F k − 1 , l ( x ) + j = 1 ∑ d ∣ 3 η 2 ( G k − 1 , l ( x )) ∣ ∂ x j ∂ F k − 1 , l ( x ) − ∂ x j ∂ G k − 1 , l ( x ) .
We denote the two summations above by T 1 T_1 T 1 and T 2 T_2 T 2 , respectively:
T 1 : = ∑ j = 1 d ∣ 3 η 2 ( F k − 1 , l ( x ) ) − 3 η 2 ( G k − 1 , l ( x ) ) ∣ ∣ ∂ F k − 1 , l ( x ) ∂ x j ∣ , T_1 := \sum_{j=1}^{d} \left|3\eta_2(F_{k-1,l}(x))-3\eta_2(G_{k-1,l}(x))\right|\left|\frac{\partial F_{k-1,l}(x)}{\partial x_j}\right|, T 1 := j = 1 ∑ d ∣ 3 η 2 ( F k − 1 , l ( x )) − 3 η 2 ( G k − 1 , l ( x )) ∣ ∂ x j ∂ F k − 1 , l ( x ) ,
T 2 : = ∑ j = 1 d ∣ 3 η 2 ( G k − 1 , l ( x ) ) ∣ ∣ ∂ F k − 1 , l ( x ) ∂ x j − ∂ G k − 1 , l ( x ) ∂ x j ∣ . T_2 := \sum_{j=1}^{d} \left|3\eta_2(G_{k-1,l}(x))\right|\left|\frac{\partial F_{k-1,l}(x)}{\partial x_j}-\frac{\partial G_{k-1,l}(x)}{\partial x_j}\right|. T 2 := j = 1 ∑ d ∣ 3 η 2 ( G k − 1 , l ( x )) ∣ ∂ x j ∂ F k − 1 , l ( x ) − ∂ x j ∂ G k − 1 , l ( x ) .
For the first sum T 1 T_1 T 1 , applying Lemma A.15 , Lemma A.16, Lemma A.17 and Lemma A.19 yields the following upper bound:
T 1 ≤ 6 ( sup x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) ∥ F k − 1 ( x ) ∥ ∞ ) ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ∑ j = 1 d ∣ ∂ F k − 1 , l ( x ) ∂ x j ∣ T_1 \leq6\left(\sup_{\substack{x\in\Omega,\ F_{k-1}\in\Phi_{k-1}(L,W,S,B)}}\lVert F_{k-1}(x)\rVert_\infty\right)\lVert F_{k-1}(x)-G_{k-1}(x)\rVert_\infty\sum_{j=1}^{d}\left|\frac{\partial F_{k-1,l}(x)}{\partial x_j}\right| T 1 ≤ 6 ( x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) sup ∥ F k − 1 ( x ) ∥ ∞ ) ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ j = 1 ∑ d ∂ x j ∂ F k − 1 , l ( x )
≤ 6 ( sup x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) ∥ F k − 1 ( x ) ∥ ∞ ) ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ∥ J [ F k − 1 ] ( x ) ∥ ∞ \leq6\left(\sup_{\substack{x\in\Omega,\ F_{k-1}\in\Phi_{k-1}(L,W,S,B)}}\lVert F_{k-1}(x)\rVert_\infty\right)\lVert F_{k-1}(x)-G_{k-1}(x)\rVert_\infty\lVert J[F_{k-1}](x)\rVert_\infty ≤ 6 ( x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) sup ∥ F k − 1 ( x ) ∥ ∞ ) ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ∥ J [ F k − 1 ] ( x ) ∥ ∞
≤ 3 × W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 × δ W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 3 2 k − 2 × W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 3 2 k − 2 = δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − 3 k + 6 3 2 k − 3 . \begin{aligned}
\leq3\times W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}\times\delta W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}3^{2k-2} \\
&\times W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}3^{2k-2}
=\delta W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-3k+6}3^{2k-3}.
\end{aligned} ≤ 3 × W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 × δ W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 3 2 k − 2 × W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 3 2 k − 2 = δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − 3 k + 6 3 2 k − 3 .
For the second sum T 2 T_2 T 2 , applying Lemma A.15 and inductive hypothesis yields:
T 2 ≤ 3 ( sup x ∈ Ω , G k − 1 ∈ Φ k − 1 ( L , W , S , B ) ∥ G k − 1 ( x ) ∥ ∞ ) 2 ∥ J [ F k − 1 ] ( x ) − J [ G k − 1 ] ( x ) ∥ ∞ T_2\leq3\left(\sup_{\substack{x\in\Omega,\ G_{k-1}\in\Phi_{k-1}(L,W,S,B)}}\lVert G_{k-1}(x)\rVert_\infty\right)^2\lVert J[F_{k-1}](x)-J[G_{k-1}](x)\rVert_\infty T 2 ≤ 3 ( x ∈ Ω , G k − 1 ∈ Φ k − 1 ( L , W , S , B ) sup ∥ G k − 1 ( x ) ∥ ∞ ) 2 ∥ J [ F k − 1 ] ( x ) − J [ G k − 1 ] ( x ) ∥ ∞
≤ 3 × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × δ W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 3 2 k − 4 = δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − 3 k + 6 3 2 k − 3 . \begin{aligned}
\leq3\times W^{3^{k-2}-1}(B\vee d)^{5\cdot3^{k-2}-1}2^{3^{k-1}-1-2k+4}\times\delta W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}3^{2k-4} \\
&=\delta W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-3k+6}3^{2k-3}.
\end{aligned} ≤ 3 × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × δ W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 3 2 k − 4 = δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − 3 k + 6 3 2 k − 3 .
Combining the two upper bounds on T 1 T_1 T 1 and T 2 T_2 T 2 yields:
∑ j = 1 d ∣ 3 η 2 ( F k − 1 , l ( x ) ) ∂ F k − 1 , l ( x ) ∂ x j − 3 η 2 ( G k − 1 , l ( x ) ) ∂ G k − 1 , l ( x ) ∂ x j ∣ ≤ T 1 + T 2 ≤ 2 × δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 2 − 3 k + 6 3 2 k − 3 . \begin{aligned}
\sum_{j=1}^{d}\left|3\eta_2(F_{k-1,l}(x))\frac{\partial F_{k-1,l}(x)}{\partial x_j}-3\eta_2(G_{k-1,l}(x))\frac{\partial G_{k-1,l}(x)}{\partial x_j}\right| \\
&\leq T_1+T_2\leq2\times\delta W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}2^{\frac{3^k-3}{2}-3k+6}3^{2k-3}.
\end{aligned} j = 1 ∑ d 3 η 2 ( F k − 1 , l ( x )) ∂ x j ∂ F k − 1 , l ( x ) − 3 η 2 ( G k − 1 , l ( x )) ∂ x j ∂ G k − 1 , l ( x ) ≤ T 1 + T 2 ≤ 2 × δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 2 2 3 k − 3 − 3 k + 6 3 2 k − 3 .
By taking supremum with respect to 1 ≤ l ≤ W 1\leq l\leq W 1 ≤ l ≤ W on the LHS yields:
(A.32)
B W ∥ J [ η 3 ∘ F k − 1 ] ( x ) − J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ ≤ δ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − 3 k + 6 3 2 k − 3 . BW\lVert J[\eta_3\circ F_{k-1}](x)-J[\eta_3\circ G_{k-1}](x)\rVert_\infty\leq\delta W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-1}{2}-3k+6}3^{2k-3}. B W ∥ J [ η 3 ∘ F k − 1 ] ( x ) − J [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ ≤ δ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − 3 k + 6 3 2 k − 3 .
By adding the two upper bounds in A.31 and A.32, we can deduce that:
∥ J [ F k ] ( x ) − J [ G k ] ( x ) ∥ ∞ ≤ δ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 − 1 2 3 k − 1 2 − 3 k + 5 3 k − 1 + δ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − 3 k + 6 3 2 k − 3 ≤ δ W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 2 − k + 1 3 2 k − 2 . \begin{aligned}
\lVert J[F_k](x)-J[G_k](x)\rVert_\infty\leq\delta W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}-1}2^{\frac{3^k-1}{2}-3k+5}3^{k-1} \\
&+\delta W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-1}{2}-3k+6}3^{2k-3} \\
&\leq\delta W^{\frac{3^k-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}2^{\frac{3^k-1}{2}-k+1}3^{2k-2}.
\end{aligned} ∥ J [ F k ] ( x ) − J [ G k ] ( x ) ∥ ∞ ≤ δ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 − 1 2 2 3 k − 1 − 3 k + 5 3 k − 1 + δ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − 3 k + 6 3 2 k − 3 ≤ δ W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 2 2 3 k − 1 − k + 1 3 2 k − 2 .
where the last inequality above follows from k ≥ 2 k \ge2 k ≥ 2 . Taking supremum with respect to x ∈ Ω x \in\Omega x ∈ Ω on the LHS implies the given upper bound also holds for k k k . By induction, the claim is proved. In particular, when k = L k = L k = L , we have ∇ F L ( x ) = J [ F L ] ( x ) T \nabla F_L(x) = J[F_L](x)^T ∇ F L ( x ) = J [ F L ] ( x ) T for any x ∈ Ω x \in\Omega x ∈ Ω . Applying Lemma A.20 then yields:
sup x ∈ Ω ∣ ∥ ∇ F L ( x ) ∥ − ∥ ∇ G L ( x ) ∥ ∣ = sup x ∈ Ω ∣ ∥ ∇ J [ F L ] ( x ) T ∥ − ∥ ∇ J [ G L ] ( x ) T ∥ ∣ ≤ sup x ∈ Ω ∥ J [ F L ] ( x ) − J [ G L ] ( x ) ∥ ∞ ≤ δ W 3 L − 1 2 ( B ∨ d ) 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 2 − k + 1 3 2 L − 2 . \begin{aligned}
\sup_{x\in\Omega}\left|\|\nabla F_L(x)\|-\|\nabla G_L(x)\|\right| = \sup_{x\in\Omega}\left|\|\nabla J[F_L](x)^T\|-\|\nabla J[G_L](x)^T\|\right| \\
&\le\sup_{x\in\Omega}\|J[F_L](x)-J[G_L](x)\|_{\infty} \\
&\le\delta W^{\frac{3^L-1}{2}}(B\vee d)^{\frac{5\cdot3^{L-1}-1}{2}}2^{\frac{3^L-1}{2}-k+1}3^{2L-2}.
\end{aligned} x ∈ Ω sup ∣ ∥∇ F L ( x ) ∥ − ∥∇ G L ( x ) ∥ ∣ = x ∈ Ω sup ∥∇ J [ F L ] ( x ) T ∥ − ∥∇ J [ G L ] ( x ) T ∥ ≤ x ∈ Ω sup ∥ J [ F L ] ( x ) − J [ G L ] ( x ) ∥ ∞ ≤ δ W 2 3 L − 1 ( B ∨ d ) 2 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 − k + 1 3 2 L − 2 .
This finishes our proof of the Lemma.
Theorem A.22. (Bounding the DNN Gradient space covering number ) Fix some sufficiently large N ∈ Z + N \in\mathbb{Z}^{+} N ∈ Z + . Consider a Deep Neural Network space Φ ( L , W , S , B ) \Phi(L,W,S,B) Φ ( L , W , S , B ) with L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( N ) W = O(N) W = O ( N ) , S = O ( N ) S = O(N) S = O ( N ) and B = O ( N ) B = O(N) B = O ( N ) . Then the log value of the covering number of the DNN Gradient space with respect to the ∥ ⋅ ∥ ∞ \|\cdot\|_{\infty} ∥ ⋅ ∥ ∞ norm ∥ F ( x ) ∥ ∞ : = sup x ∈ Ω ∣ F ( x ) ∣ \|F(x)\|_{\infty} := \sup_{x\in\Omega}|F(x)| ∥ F ( x ) ∥ ∞ := sup x ∈ Ω ∣ F ( x ) ∣ , which is denoted by N ( δ , ∇ Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) \mathcal{N}(\delta,\nabla\Phi(L,W,S,B),\|\cdot\|_{\infty}) N ( δ , ∇Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) , can be upper bounded by:
log N ( δ , ∇ Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) = O ( S [ log ( δ − 1 ) + 3 L log ( W B ) ] ) . (73) \log\mathcal{N}(\delta,\nabla\Phi(L,W,S,B),\|\cdot\|_{\infty}) = O\left(S\left[\log(\delta^{-1})+3^L\log(WB)\right]\right). \tag*{(73)} log N ( δ , ∇Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) = O ( S [ log ( δ − 1 ) + 3 L log ( W B ) ] ) . (73)
Proof. We firstly fix a sparsity pattern (i.e, the locations of the non-zero entries are fixed). Using equation A.29 in Lemma A.21 , yields the following upper bound on the covering number with respect to ∥ ⋅ ∥ ∞ \|\cdot\|_{\infty} ∥ ⋅ ∥ ∞ :
( δ W 3 L − 1 2 ( B ∨ d ) 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 2 − L + 1 3 2 L − 2 ) − S . \left(\frac{\delta}{W^{\frac{3^L-1}{2}}(B\vee d)^{\frac{5\cdot3^{L-1}-1}{2}}2^{\frac{3^L-1}{2}-L+1}3^{2L-2}}\right)^{-S}. ( W 2 3 L − 1 ( B ∨ d ) 2 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 − L + 1 3 2 L − 2 δ ) − S .
Furthermore, note that the number of feasible configurations is upper bounded by: ( ( W + 1 ) L S ) ≤ ( W + 1 ) L S \binom{(W+1)^L}{S} \le(W+1)^{LS} ( S ( W + 1 ) L ) ≤ ( W + 1 ) L S . [63 , 18 ] Plug this inequality into the previous estimation then yields:
log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≤ log [ ( W + 1 ) L S ( δ W 3 L − 1 2 ( B ∨ d ) 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 2 − L + 1 3 2 L − 2 ) − S ] ≤ S log [ δ − 1 ( W + 1 ) L 3 2 L − 2 W 3 L − 1 2 ( B ∨ d ) 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 2 − L + 1 ] \begin{aligned}
\log\mathcal{N}(\delta,\Phi(L,W,S,B),\|\cdot\|_{\infty}) \le\log\left[(W+1)^{LS}\left(\frac{\delta}{W^{\frac{3^L-1}{2}}(B\vee d)^{\frac{5\cdot3^{L-1}-1}{2}}2^{\frac{3^L-1}{2}-L+1}3^{2L-2}}\right)^{-S}\right] \\
&\le S\log\left[\delta^{-1}(W+1)^L3^{2L-2}W^{\frac{3^L-1}{2}}(B\vee d)^{\frac{5\cdot3^{L-1}-1}{2}}2^{\frac{3^L-1}{2}-L+1}\right]
\end{aligned} log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≤ log ( W + 1 ) L S ( W 2 3 L − 1 ( B ∨ d ) 2 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 − L + 1 3 2 L − 2 δ ) − S ≤ S log [ δ − 1 ( W + 1 ) L 3 2 L − 2 W 2 3 L − 1 ( B ∨ d ) 2 5 ⋅ 3 L − 1 − 1 2 2 3 L − 1 − L + 1 ]
≲ S [ log ( δ − 1 ) + 2 L log ( 3 W ) + 3 L log ( W ( B ∨ d ) ) + 3 L log 2 ] . \lesssim S\left[\log(\delta^{-1})+2L\log(3W)+3^L\log(W(B\vee d))+3^L\log2\right]. ≲ S [ log ( δ − 1 ) + 2 L log ( 3 W ) + 3 L log ( W ( B ∨ d )) + 3 L log 2 ] .
Note that here the dimension d d d is some constant. Thus, by plugging in the given magnitudes L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( N ) W = O(N) W = O ( N ) , S = O ( N ) S = O(N) S = O ( N ) and B = O ( N ) B = O(N) B = O ( N ) , we can further deduce that:
log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≲ S [ log ( δ − 1 ) + 3 L log ( W B ) ] . \log\mathcal{N}(\delta,\Phi(L,W,S,B),\|\cdot\|_{\infty}) \lesssim S\left[\log(\delta^{-1})+3^L\log(WB)\right]. log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≲ S [ log ( δ − 1 ) + 3 L log ( W B ) ] .
This finishes our proof.
Now let’s consider upper bounding the covering number of the Laplacian of the sparse Deep Neural Networks. Note that for any 1 ≤ k ≤ L − 1 1 \le k \le L - 1 1 ≤ k ≤ L − 1 , any k k k –ReLU3 Deep Neural Network F k ∈ Φ k ( L , W , S , B ) F_k \in\Phi_k(L, W, S, B) F k ∈ Φ k ( L , W , S , B ) is a vector-valued function mapping from R d \mathbb{R}^d R d to R W \mathbb{R}^W R W . Moreover, we define the Laplacian of F k ( x ) F_k(x) F k ( x ) , which is denoted by Δ [ F k ] ( x ) \Delta[F_k](x) Δ [ F k ] ( x ) , as follows:
Δ [ F k ] ( x ) = [ Δ F k , 1 ( x ) , Δ F k , 2 ( x ) , ⋯ , Δ F k , W ( x ) ] T ∈ R W , \Delta[F_k](x) = [\Delta F_{k,1}(x), \Delta F_{k,2}(x), \cdots, \Delta F_{k,W}(x)]^T \in\mathbb{R}^W, Δ [ F k ] ( x ) = [ Δ F k , 1 ( x ) , Δ F k , 2 ( x ) , ⋯ , Δ F k , W ( x ) ] T ∈ R W ,
where for any 1 ≤ l ≤ W 1 \le l \le W 1 ≤ l ≤ W , we have:
Δ F k , l ( x ) = ∑ j = 1 d ∂ 2 ∂ x j 2 F k , l ( x ) . \Delta F_{k,l}(x) = \sum_{j=1}^{d} \frac{\partial^2}{\partial x_j^2}F_{k,l}(x). Δ F k , l ( x ) = j = 1 ∑ d ∂ x j 2 ∂ 2 F k , l ( x ) .
In particular, when k = L k = L k = L , we have that any F L ∈ Φ L ( L , W , S , B ) = Φ ( L , W , S , B ) F_L \in\Phi_L(L, W, S, B) = \Phi(L, W, S, B) F L ∈ Φ L ( L , W , S , B ) = Φ ( L , W , S , B ) is a scalar-valued function mapping from R d \mathbb{R}^d R d to R \mathbb{R} R . Thus, its Laplacian can be explicitly written as:
Δ [ F L ] ( x ) = Δ F L ( x ) = ∑ j = 1 d ∂ 2 ∂ x j 2 F L ( x ) . \Delta[F_L](x) = \Delta F_L(x) = \sum_{j=1}^{d} \frac{\partial^2}{\partial x_j^2}F_L(x). Δ [ F L ] ( x ) = Δ F L ( x ) = j = 1 ∑ d ∂ x j 2 ∂ 2 F L ( x ) .
For both Lemma A.23 and Lemma A.24 below, we consider a fixed Deep Neural Network space Φ ( L , W , S , B ) \Phi(L, W, S, B) Φ ( L , W , S , B ) with L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( N ) W = O(N) W = O ( N ) , S = O ( N ) S = O(N) S = O ( N ) and B = O ( N ) B = O(N) B = O ( N ) , where N ∈ Z + N \in\mathbb{Z}^+ N ∈ Z + is fixed and sufficiently large.
Lemma A.23. (Upper bound on ∞ \infty ∞ -norm of Laplacian of elements in the DNN space ) For any 1 ≤ k ≤ L 1 \le k \le L 1 ≤ k ≤ L , we have the following upper bound:
sup x ∈ Ω , F k ∈ Φ k ( L , W , S , B ) ∥ Δ [ F k ] ( x ) ∥ ∞ = O ( W 3 k − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 ) . \sup_{x \in\Omega,F_k \in\Phi_k(L,W,S,B)} \|\Delta[F_k](x)\|_\infty= O\left(W^{\frac{3^k-1}{2}}(B \vee d)^{\frac{5\cdot3^{k-1}-1}{2}}\right). x ∈ Ω , F k ∈ Φ k ( L , W , S , B ) sup ∥Δ [ F k ] ( x ) ∥ ∞ = O ( W 2 3 k − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 ) . Proof. We use induction on k k k to prove the claim.
Base case: k = 1 k = 1 k = 1 . Note that any F 1 ∈ Φ 1 ( L , W , S , B ) F_1 \in\Phi_1(L, W, S, B) F 1 ∈ Φ 1 ( L , W , S , B ) is a linear transform, so the Laplacian Δ [ F 1 ] ( x ) \Delta[F_1](x) Δ [ F 1 ] ( x ) must be the zero vector for any x ∈ Ω x \in\Omega x ∈ Ω . This implies:
∥ Δ [ F 1 ] ( x ) ∥ ∞ = 0 ≲ ( B ∨ d ) 2 . \|\Delta[F_1](x)\|_\infty= 0 \lesssim(B \vee d)^2. ∥Δ [ F 1 ] ( x ) ∥ ∞ = 0 ≲ ( B ∨ d ) 2 . Inductive Step: Assume that the claim has been proved for k − 1 k - 1 k − 1 , where 2 ≤ k ≤ L 2 \le k \le L 2 ≤ k ≤ L . For any x ∈ Ω x \in\Omega x ∈ Ω and any F k ∈ Φ k ( L , W , S , B ) F_k \in\Phi_k(L, W, S, B) F k ∈ Φ k ( L , W , S , B ) , using linearity of the Laplacian operator implies:
Δ [ F k ] ( x ) = W F ( k ) Δ [ η 3 ∘ F k − 1 ] ( x ) . \Delta[F_k](x) = \mathcal{W}_F^{(k)}\Delta[\eta_3 \circ F_{k-1}](x). Δ [ F k ] ( x ) = W F ( k ) Δ [ η 3 ∘ F k − 1 ] ( x ) . Taking the inf-norm on both sides of the identity above implies:
∥ Δ [ F k ] ( x ) ∥ ∞ ≤ ∥ W F ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ W B ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ . \|\Delta[F_k](x)\|_\infty\le\|\mathcal{W}_F^{(k)}\|_\infty\|\Delta[\eta_3 \circ F_{k-1}](x)\|_\infty\le WB\|\Delta[\eta_3 \circ F_{k-1}](x)\|_\infty. ∥Δ [ F k ] ( x ) ∥ ∞ ≤ ∥ W F ( k ) ∥ ∞ ∥Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≤ W B ∥Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ . It now remains to upper bound the term ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ \|\Delta[\eta_3 \circ F_{k-1}](x)\|_\infty ∥Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ . For any 1 ≤ l ≤ W 1 \le l \le W 1 ≤ l ≤ W , we will use the Chain Rule to write the l l l -th component ( Δ [ η 3 ∘ F k − 1 ] ( x ) ) l \left(\Delta[\eta_3 \circ F_{k-1}](x)\right)_l ( Δ [ η 3 ∘ F k − 1 ] ( x ) ) l in an explicit form. For any 1 ≤ j ≤ d 1 \le j \le d 1 ≤ j ≤ d , we have:
∂ ∂ x j η 3 [ F k − 1 , l ( x ) ] = 3 η 2 [ F k − 1 , l ( x ) ] ∂ ∂ x j F k − 1 , l ( x ) . \frac{\partial}{\partial x_j}\eta_3[F_{k-1,l}(x)] = 3\eta_2[F_{k-1,l}(x)]\frac{\partial}{\partial x_j}F_{k-1,l}(x). ∂ x j ∂ η 3 [ F k − 1 , l ( x )] = 3 η 2 [ F k − 1 , l ( x )] ∂ x j ∂ F k − 1 , l ( x ) .
Differentiating with respect to x j x_j x j on both sides above yields:
∂ 2 ∂ x j 2 η 3 [ F k − 1 , l ( x ) ] = 6 η 1 [ F k − 1 , l ( x ) ] ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 + 3 η 2 [ F k − 1 , l ( x ) ] ∂ 2 ∂ x j 2 F k − 1 , l ( x ) . (74) \frac{\partial^2}{\partial x_j^2}\eta_3[F_{k-1,l}(x)] = 6\eta_1[F_{k-1,l}(x)]\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2 + 3\eta_2[F_{k-1,l}(x)]\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x). \tag*{(74)} ∂ x j 2 ∂ 2 η 3 [ F k − 1 , l ( x )] = 6 η 1 [ F k − 1 , l ( x )] ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 + 3 η 2 [ F k − 1 , l ( x )] ∂ x j 2 ∂ 2 F k − 1 , l ( x ) . (74)
Summing the expression above from j = 1 j=1 j = 1 to j = d j=d j = d implies:
∣ ( Δ [ η 3 ∘ F k − 1 ] ( x ) ) l ∣ = ∣ ∑ j = 1 d ∂ 2 ∂ x j 2 η 3 [ F k − 1 , l ( x ) ] ∣ = ∣ 6 η 1 [ F k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 + 3 η 2 [ F k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) ∣ ≤ 6 ∣ η 1 [ F k − 1 , l ( x ) ] ∣ ( ∑ j = 1 d ∂ ∂ x j F k − 1 , l ( x ) ) 2 + 3 ∣ η 2 [ F k − 1 , l ( x ) ] ∣ ∣ ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) ∣ . \begin{aligned}
\left|\left(\Delta[\eta_3\circ F_{k-1}](x)\right)_l\right|
&=\left|\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}\eta_3[F_{k-1,l}(x)]\right|\\
&=\left|6\eta_1[F_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2+3\eta_2[F_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)\right|\\
&\leq6\left|\eta_1[F_{k-1,l}(x)]\right|\left(\sum_{j=1}^{d}\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2+3\left|\eta_2[F_{k-1,l}(x)]\right|\left|\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)\right|.
\end{aligned} ∣ ( Δ [ η 3 ∘ F k − 1 ] ( x ) ) l ∣ = j = 1 ∑ d ∂ x j 2 ∂ 2 η 3 [ F k − 1 , l ( x )] = 6 η 1 [ F k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 + 3 η 2 [ F k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) ≤ 6 ∣ η 1 [ F k − 1 , l ( x )] ∣ ( j = 1 ∑ d ∂ x j ∂ F k − 1 , l ( x ) ) 2 + 3 ∣ η 2 [ F k − 1 , l ( x )] ∣ j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) .
We denote the two summations above by U 1 U_1 U 1 and U 2 U_2 U 2 , respectively:
U 1 : = 6 ∣ η 1 [ F k − 1 , l ( x ) ] ∣ ( ∑ j = 1 d ∣ ∂ ∂ x j F k − 1 , l ( x ) ∣ ) 2 . U_1:=6\left|\eta_1[F_{k-1,l}(x)]\right|\left(\sum_{j=1}^{d}\left|\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right|\right)^2. U 1 := 6 ∣ η 1 [ F k − 1 , l ( x )] ∣ ( j = 1 ∑ d ∂ x j ∂ F k − 1 , l ( x ) ) 2 .
U 2 : = 3 ∣ η 2 [ F k − 1 , l ( x ) ] ∣ ∣ ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) ∣ . U_2:=3\left|\eta_2[F_{k-1,l}(x)]\right|\left|\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)\right|. U 2 := 3 ∣ η 2 [ F k − 1 , l ( x )] ∣ j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) .
On the one hand, by applying Lemma A.15 and Lemma A.19, we can upper bound U 1 U_1 U 1 by:
U 1 ≤ 6 ( sup x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) ∥ F k − 1 ( x ) ∥ ∞ ) ( sup x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) ∥ J [ F k − 1 ] ( x ) ∥ ∞ ) 2 ≤ 6 × W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 2 − k + 2 × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 3 2 k − 4 ≲ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 , \begin{aligned}
U_1&\leq6\left(\sup_{x\in\Omega,\ F_{k-1}\in\Phi_{k-1}(L,W,S,B)}\left\|F_{k-1}(x)\right\|_\infty\right)\left(\sup_{x\in\Omega,\ F_{k-1}\in\Phi_{k-1}(L,W,S,B)}\left\|J[F_{k-1}](x)\right\|_\infty\right)^2\\
&\leq6\times W^{\frac{3^k-2-1}{2}}(B\vee d)^{\frac{5\cdot3^k-2-1}{2}}2^{\frac{3^k-1}{2}-k+2}\times W^{3^k-2-1}(B\vee d)^{5\cdot3^k-2-1}2^{3^k-1-1-2k+4}3^{2k-4}\\
&\lesssim W^{\frac{3^k-1-3}{2}}(B\vee d)^{\frac{5\cdot3^k-1-3}{2}},
\end{aligned} U 1 ≤ 6 ( x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) sup ∥ F k − 1 ( x ) ∥ ∞ ) ( x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) sup ∥ J [ F k − 1 ] ( x ) ∥ ∞ ) 2 ≤ 6 × W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − k + 2 × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 3 2 k − 4 ≲ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 ,
where the last step above follows from k ≤ L k\leq L k ≤ L and L = O ( 1 ) L=O(1) L = O ( 1 ) .
On the other hand, by applying Lemma A.15 and the inductive hypothesis, we have:
U 2 ≤ 3 ( sup x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) ∥ F k − 1 ( x ) ∥ ∞ ) 2 ∥ Δ [ F k − 1 ] ( x ) ∥ ∞ ≤ 3 × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 ≲ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 , \begin{aligned}
U_2&\leq3\left(\sup_{x\in\Omega,\ F_{k-1}\in\Phi_{k-1}(L,W,S,B)}\left\|F_{k-1}(x)\right\|_\infty\right)^2\left\|\Delta[F_{k-1}](x)\right\|_\infty\\
&\leq3\times W^{3^k-2-1}(B\vee d)^{5\cdot3^k-2-1}2^{3^k-1-1-2k+4}\times W^{\frac{3^k-2-1}{2}}(B\vee d)^{\frac{5\cdot3^k-2-1}{2}}\\
&\lesssim W^{\frac{3^k-1-3}{2}}(B\vee d)^{\frac{5\cdot3^k-1-3}{2}},
\end{aligned} U 2 ≤ 3 ( x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) sup ∥ F k − 1 ( x ) ∥ ∞ ) 2 ∥ Δ [ F k − 1 ] ( x ) ∥ ∞ ≤ 3 × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 ≲ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 ,
where the last step above follows from k ≤ L k\leq L k ≤ L and L = O ( 1 ) L=O(1) L = O ( 1 ) .
Summing the two bounds on U 1 U_1 U 1 and U 2 U_2 U 2 implies that for any 1 ≤ l ≤ W 1\leq l\leq W 1 ≤ l ≤ W , we have:
∣ ( Δ [ η 3 ∘ F k − 1 ] ( x ) ) l ∣ ≤ U 1 + U 2 ≲ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 . (75) \left|\left(\Delta[\eta_3\circ F_{k-1}](x)\right)_l\right|\leq U_1+U_2\lesssim W^{\frac{3^k-1-3}{2}}(B\vee d)^{\frac{5\cdot3^k-1-3}{2}}. \tag*{(75)} ∣ ( Δ [ η 3 ∘ F k − 1 ] ( x ) ) l ∣ ≤ U 1 + U 2 ≲ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 . (75)
Taking supremum with respect to 1 ≤ l ≤ W 1 \le l \le W 1 ≤ l ≤ W then yields:
∥ Δ [ F k ] ( x ) ∥ ∞ ≤ W B ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≲ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 . \lVert\Delta[F_k](x) \rVert_{\infty} \le W B \lVert\Delta[\eta_3 \circ F_{k-1}](x) \rVert_{\infty} \lesssim W^{\frac{3^{k-1}-1}{2}} (B \vee d)^{\frac{5\cdot3^{k-1}-1}{2}}. ∥ Δ [ F k ] ( x ) ∥ ∞ ≤ W B ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≲ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 .
Taking supremum with respect to x ∈ Ω x \in\Omega x ∈ Ω and F k ∈ Φ k ( L , W , S , B ) F_k \in\Phi_k(L,W,S,B) F k ∈ Φ k ( L , W , S , B ) on the LHS implies that the given upper bound also holds for k k k . By induction, the claim is proved.
Lemma A.24. (Relation between the covering number of the DNN Laplacian space and parameter space ) For any 1 ≤ k ≤ L 1 \le k \le L 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L , W , S , B ) F_k,G_k \in\Phi_k(L,W,S,B) F k , G k ∈ Φ k ( L , W , S , B ) are given by:
F k ( x ) : = ( W F ( k ) η 3 ( ⋅ ) + b F ( k ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) , F_k(x) := \left(\mathcal{W}^{(k)}_F\eta_3(\cdot)+b^{(k)}_F\right)\cdots\left(\mathcal{W}^{(1)}_F x+b^{(1)}_F\right), F k ( x ) := ( W F ( k ) η 3 ( ⋅ ) + b F ( k ) ) ⋯ ( W F ( 1 ) x + b F ( 1 ) ) , G k ( x ) : = ( W G ( k ) η 3 ( ⋅ ) + b G ( k ) ) ⋯ ( W G ( 1 ) x + b G ( 1 ) ) . G_k(x) := \left(\mathcal{W}^{(k)}_G\eta_3(\cdot)+b^{(k)}_G\right)\cdots\left(\mathcal{W}^{(1)}_G x+b^{(1)}_G\right). G k ( x ) := ( W G ( k ) η 3 ( ⋅ ) + b G ( k ) ) ⋯ ( W G ( 1 ) x + b G ( 1 ) ) . Furthermore, assume that the ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_{\infty} ∥ ⋅ ∥ ∞ norm of the distance between the parameter spaces is uniformly upper bounded by δ \delta δ , i.e
∥ W F ( l ) − W G ( l ) ∥ ∞ , ∞ ≤ δ , ∥ b F ( l ) − b G ( l ) ∥ ∞ ≤ δ , ( ∀ 1 ≤ l ≤ k ) . (76) \lVert\mathcal{W}^{(l)}_F-\mathcal{W}^{(l)}_G\rVert_{\infty,\infty}\le\delta,\quad\lVert b^{(l)}_F-b^{(l)}_G\rVert_{\infty}\le\delta,\quad(\forall\ 1\le l\le k). \tag*{(76)} ∥ W F ( l ) − W G ( l ) ∥ ∞ , ∞ ≤ δ , ∥ b F ( l ) − b G ( l ) ∥ ∞ ≤ δ , ( ∀ 1 ≤ l ≤ k ) . (76) Then we have:
sup x ∈ Ω ∥ Δ [ F k ] ( x ) − Δ [ G k ] ( x ) ∥ ∞ = O ( δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 ) . (77) \sup_{x\in\Omega}\lVert\Delta[F_k](x)-\Delta[G_k](x)\rVert_{\infty}
=O\left(\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}\right). \tag*{(77)} x ∈ Ω sup ∥ Δ [ F k ] ( x ) − Δ [ G k ] ( x ) ∥ ∞ = O ( δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 ) . (77)
Proof. We use induction on k k k to prove the claim.
Base case: k = 1 k=1 k = 1 . Note that any F 1 ∈ Φ 1 ( L , W , S , B ) F_1\in\Phi_1(L,W,S,B) F 1 ∈ Φ 1 ( L , W , S , B ) is a linear transform, so the Laplacian Δ [ F 1 ] ( x ) \Delta[F_1](x) Δ [ F 1 ] ( x ) must be the zero vector for any x ∈ Ω x\in\Omega x ∈ Ω . Hence, for any x ∈ Ω x\in\Omega x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L , W , S , B ) F_1,G_1\in\Phi_1(L,W,S,B) F 1 , G 1 ∈ Φ 1 ( L , W , S , B ) , we have:
∥ Δ [ F 1 ] ( x ) − Δ [ G 1 ] ( x ) ∥ ∞ = 0 ≲ δ ( B ∨ d ) 2 . \lVert\Delta[F_1](x)-\Delta[G_1](x)\rVert_{\infty}=0\lesssim\delta(B\vee d)^2. ∥ Δ [ F 1 ] ( x ) − Δ [ G 1 ] ( x ) ∥ ∞ = 0 ≲ δ ( B ∨ d ) 2 .
Inductive Step: assume that the claim has been proved for k − 1 k-1 k − 1 , where 2 ≤ k ≤ L 2\le k\le L 2 ≤ k ≤ L . Then for any x ∈ Ω x\in\Omega x ∈ Ω and F k , G k ∈ Φ k ( L , W , S , B ) F_k,G_k\in\Phi_k(L,W,S,B) F k , G k ∈ Φ k ( L , W , S , B ) satisfying constraint A.36, applying linearity of the Laplacian operator indicates:
∥ Δ [ F k ] ( x ) − Δ [ G k ] ( x ) ∥ ∞ = ∥ W F ( k ) Δ [ η 3 ∘ F k − 1 ] ( x ) − W G ( k ) Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ = ∥ ( W F ( k ) − W G ( k ) ) Δ [ η 3 ∘ F k − 1 ] ( x ) + W G ( k ) ( Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ) ∥ ∞ ≤ ∥ W F ( k ) − W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ + ∥ W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ . \begin{aligned}
\lVert\Delta[F_k](x)-\Delta[G_k](x)\rVert_{\infty}
&=\lVert\mathcal{W}^{(k)}_F\Delta[\eta_3\circ F_{k-1}](x)-\mathcal{W}^{(k)}_G\Delta[\eta_3\circ G_{k-1}](x)\rVert_{\infty}\\
&=\left\lVert\left(\mathcal{W}^{(k)}_F-\mathcal{W}^{(k)}_G\right)\Delta[\eta_3\circ F_{k-1}](x)\right.\\
&\qquad\left.+\mathcal{W}^{(k)}_G\left(\Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x)\right)\right\rVert_{\infty}\\
&\le\lVert\mathcal{W}^{(k)}_F-\mathcal{W}^{(k)}_G\rVert_{\infty}\lVert\Delta[\eta_3\circ F_{k-1}](x)\rVert_{\infty}\\
&\qquad+\lVert\mathcal{W}^{(k)}_G\rVert_{\infty}\lVert\Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x)\rVert_{\infty}.
\end{aligned} ∥ Δ [ F k ] ( x ) − Δ [ G k ] ( x ) ∥ ∞ = ∥ W F ( k ) Δ [ η 3 ∘ F k − 1 ] ( x ) − W G ( k ) Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ = ( W F ( k ) − W G ( k ) ) Δ [ η 3 ∘ F k − 1 ] ( x ) + W G ( k ) ( Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ) ∞ ≤ ∥ W F ( k ) − W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ + ∥ W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ .
For the first term ∥ W F ( k ) − W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ \lVert\mathcal{W}^{(k)}_F-\mathcal{W}^{(k)}_G\rVert_{\infty}\lVert\Delta[\eta_3\circ F_{k-1}](x)\rVert_{\infty} ∥ W F ( k ) − W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ , applying the bound in equation A.35 and equation A.36 yields:
∥ W F ( k ) − W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≲ δ W × W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 = δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 . (78) \begin{aligned}
\lVert\mathcal{W}^{(k)}_F-\mathcal{W}^{(k)}_G\rVert_{\infty}\lVert\Delta[\eta_3\circ F_{k-1}](x)\rVert_{\infty}
\lesssim\delta W\times W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}} \\
&=\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}. \tag*{(78)}
\end{aligned} ∥ W F ( k ) − W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) ∥ ∞ ≲ δ W × W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 = δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 . (78)
For the second term ∥ W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ \lVert\mathcal{W}_{G}^{(k)}\rVert_{\infty}\lVert\Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x)\rVert_{\infty} ∥ W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ , we need to upper bound the norm ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ \lVert\Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x)\rVert_{\infty} ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ at first. Note that for any 1 ≤ l ≤ W 1\le l\le W 1 ≤ l ≤ W , we can use equation A.34 to write the l l l -th component of Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) \Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x) Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) as:
( Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ) l = ∑ j = 1 d ∂ 2 ∂ x j 2 η 3 [ F k − 1 , l ( x ) ] − ∑ j = 1 d ∂ 2 ∂ x j 2 η 3 [ G k − 1 , l ( x ) ] = 6 η 1 [ F k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j G k − 1 , l ( x ) ) 2 + 3 η 2 [ F k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 G k − 1 , l ( x ) = 6 η 1 [ F k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 + 6 η 1 [ G k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j G k − 1 , l ( x ) ) 2 + 3 η 2 [ F k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) + 3 η 2 [ G k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 G k − 1 , l ( x ) . \begin{aligned}
\left(\Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x)\right)_l
&=\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}\eta_3[F_{k-1,l}(x)]-\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}\eta_3[G_{k-1,l}(x)]\\
&=6\eta_1[F_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-6\eta_1[G_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right)^2\\
&\quad+3\eta_2[F_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)-3\eta_2[G_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}G_{k-1,l}(x)\\
&=6\eta_1[F_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-6\eta_1[G_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2\\
&\quad+6\eta_1[G_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-6\eta_1[G_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right)^2\\
&\quad+3\eta_2[F_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)-3\eta_2[G_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)\\
&\quad+3\eta_2[G_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)-3\eta_2[G_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}G_{k-1,l}(x).
\end{aligned} ( Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ) l = j = 1 ∑ d ∂ x j 2 ∂ 2 η 3 [ F k − 1 , l ( x )] − j = 1 ∑ d ∂ x j 2 ∂ 2 η 3 [ G k − 1 , l ( x )] = 6 η 1 [ F k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ G k − 1 , l ( x ) ) 2 + 3 η 2 [ F k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 G k − 1 , l ( x ) = 6 η 1 [ F k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 + 6 η 1 [ G k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ G k − 1 , l ( x ) ) 2 + 3 η 2 [ F k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) + 3 η 2 [ G k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 G k − 1 , l ( x ) .
We denote the four summations above by V 1 , V 2 , V 3 V_1,V_2,V_3 V 1 , V 2 , V 3 and V 4 V_4 V 4 , respectively:
V 1 : = 6 η 1 [ F k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 , V_1:=6\eta_1[F_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-6\eta_1[G_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2, V 1 := 6 η 1 [ F k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 ,
V 2 : = 6 η 1 [ G k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x ) ] ∑ j = 1 d ( ∂ ∂ x j G k − 1 , l ( x ) ) 2 , V_2:=6\eta_1[G_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-6\eta_1[G_{k-1,l}(x)]\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right)^2, V 2 := 6 η 1 [ G k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − 6 η 1 [ G k − 1 , l ( x )] j = 1 ∑ d ( ∂ x j ∂ G k − 1 , l ( x ) ) 2 ,
V 3 : = 3 η 2 [ F k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) , V_3:=3\eta_2[F_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)-3\eta_2[G_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x), V 3 := 3 η 2 [ F k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) ,
V 4 : = 3 η 2 [ G k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x ) ] ∑ j = 1 d ∂ 2 ∂ x j 2 G k − 1 , l ( x ) . V_4:=3\eta_2[G_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)-3\eta_2[G_{k-1,l}(x)]\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}G_{k-1,l}(x). V 4 := 3 η 2 [ G k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) − 3 η 2 [ G k − 1 , l ( x )] j = 1 ∑ d ∂ x j 2 ∂ 2 G k − 1 , l ( x ) .
By applying Lemma A.15 , Lemma A.16, Lemma A.17 and Lemma A.19 , we can upper bound V 1 V_1 V 1 by:
V 1 = 6 ( η 1 [ F k − 1 , l ( x ) ] − η 1 [ G k − 1 , l ( x ) ] ) ∑ j = 1 d ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 ≤ 6 ∣ F k − 1 , l ( x ) − G k − 1 , l ( x ) ∣ ( ∑ j = 1 d ∣ ∂ ∂ x j F k − 1 , l ( x ) ∣ ) 2 ≤ 6 ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ∥ J [ F k − 1 ] ( x ) ∥ ∞ 2 ≲ δ W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 3 k − 2 × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 3 2 k − 4 ≲ δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 . \begin{aligned}
V_1&=6\left(\eta_1[F_{k-1,l}(x)]-\eta_1[G_{k-1,l}(x)]\right)\sum_{j=1}^{d}\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2\\
&\leq6\lvert F_{k-1,l}(x)-G_{k-1,l}(x)\rvert\left(\sum_{j=1}^{d}\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right\rvert\right)^2\leq6\lVert F_{k-1}(x)-G_{k-1}(x)\rVert_\infty\lVert J[F_{k-1}](x)\rVert_\infty^2\\
&\lesssim\delta W^{\frac{3k-2-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}3^{k-2}\times W^{3^{k-2}-1}(B\vee d)^{5\cdot3^{k-2}-1}2^{3^{k-1}-1-2k+4}3^{2k-4}\\
&\lesssim\delta W^{\frac{3k-1-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}.
\end{aligned} V 1 = 6 ( η 1 [ F k − 1 , l ( x )] − η 1 [ G k − 1 , l ( x )] ) j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 ≤ 6 ∣ F k − 1 , l ( x ) − G k − 1 , l ( x )∣ ( j = 1 ∑ d ∂ x j ∂ F k − 1 , l ( x ) ) 2 ≤ 6 ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ∥ J [ F k − 1 ] ( x ) ∥ ∞ 2 ≲ δ W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 3 k − 2 × W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 3 2 k − 4 ≲ δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 .
where the last step above follows from k ≤ L k\leq L k ≤ L and L = O ( 1 ) L=O(1) L = O ( 1 ) .
Furthermore, note that for any 1 ≤ j ≤ d 1\leq j\leq d 1 ≤ j ≤ d , we can upper bound the difference ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − ( ∂ ∂ x j G k − 1 , l ( x ) ) 2 \left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-\left(\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right)^2 ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − ( ∂ x j ∂ G k − 1 , l ( x ) ) 2 as follows:
( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − ( ∂ ∂ x j G k − 1 , l ( x ) ) 2 ≤ ∣ ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − ( ∂ ∂ x j G k − 1 , l ( x ) ) 2 ∣ = ∣ ∂ ∂ x j F k − 1 , l ( x ) + ∂ ∂ x j G k − 1 , l ( x ) ∣ ∣ ∂ ∂ x j F k − 1 , l ( x ) − ∂ ∂ x j G k − 1 , l ( x ) ∣ ≤ ( ∣ ∂ ∂ x j F k − 1 , l ( x ) ∣ + ∣ ∂ ∂ x j G k − 1 , l ( x ) ∣ ) ∣ ∂ ∂ x j F k − 1 , l ( x ) − ∂ ∂ x j G k − 1 , l ( x ) ∣ . (79) \begin{aligned}
\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-\left(\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right)^2
&\leq\left\lvert\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-\left(\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right)^2\right\rvert\\
&=\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)+\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right\rvert\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)-\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right\rvert\\
&\leq\left(\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right\rvert+\left\lvert\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right\rvert\right)\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)-\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right\rvert. \tag*{(79)}
\end{aligned} ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − ( ∂ x j ∂ G k − 1 , l ( x ) ) 2 ≤ ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − ( ∂ x j ∂ G k − 1 , l ( x ) ) 2 = ∂ x j ∂ F k − 1 , l ( x ) + ∂ x j ∂ G k − 1 , l ( x ) ∂ x j ∂ F k − 1 , l ( x ) − ∂ x j ∂ G k − 1 , l ( x ) ≤ ( ∂ x j ∂ F k − 1 , l ( x ) + ∂ x j ∂ G k − 1 , l ( x ) ) ∂ x j ∂ F k − 1 , l ( x ) − ∂ x j ∂ G k − 1 , l ( x ) . (79)
Note that η 1 ( G k − 1 , l ( x ) ) ≥ 0 \eta_1(G_{k-1,l}(x))\geq0 η 1 ( G k − 1 , l ( x )) ≥ 0 . Combining the non-negativity with equation A.39, Lemma A.15 , Lemma A.19 and Lemma A.21 helps us upper bound V 2 V_2 V 2 by:
V 2 = 6 η 1 [ G k − 1 , l ( x ) ] ∑ j = 1 d [ ( ∂ ∂ x j F k − 1 , l ( x ) ) 2 − ( ∂ ∂ x j G k − 1 , l ( x ) ) 2 ] ≤ 6 ∥ G k − 1 ( x ) ∥ ∞ ∑ j = 1 d ( ∣ ∂ ∂ x j F k − 1 , l ( x ) ∣ + ∣ ∂ ∂ x j G k − 1 , l ( x ) ∣ ) ∣ ∂ ∂ x j F k − 1 , l ( x ) − ∂ ∂ x j G k − 1 , l ( x ) ∣ ≤ 6 ∥ G k − 1 ( x ) ∥ ∞ ( ∑ j = 1 d ∣ ∂ ∂ x j F k − 1 , l ( x ) ∣ + ∑ j = 1 d ∣ ∂ ∂ x j G k − 1 , l ( x ) ∣ ) ( ∑ j = 1 d ∣ ∂ ∂ x j F k − 1 , l ( x ) − ∂ ∂ x j G k − 1 , l ( x ) ∣ ) ≤ 6 ∥ G k − 1 ( x ) ∥ ∞ ( ∥ J [ F k − 1 ] ( x ) ∥ ∞ + ∥ J [ G k − 1 ] ( x ) ∥ ∞ ) ∥ J [ F k − 1 ] ( x ) − J [ G k − 1 ] ( x ) ∥ ∞ ≤ 6 W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 × 2 W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 2 3 k − 2 × δ W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 2 − k + 1 3 2 k − 4 ≲ δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 . \begin{aligned}
V_2&=6\eta_1[G_{k-1,l}(x)]\sum_{j=1}^{d}\left[\left(\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right)^2-\left(\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right)^2\right]\\
&\leq6\lVert G_{k-1}(x)\rVert_\infty\sum_{j=1}^{d}\left(\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right\rvert+\left\lvert\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right\rvert\right)\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)-\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right\rvert\\
&\leq6\lVert G_{k-1}(x)\rVert_\infty\left(\sum_{j=1}^{d}\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)\right\rvert+\sum_{j=1}^{d}\left\lvert\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right\rvert\right)\left(\sum_{j=1}^{d}\left\lvert\frac{\partial}{\partial x_j}F_{k-1,l}(x)-\frac{\partial}{\partial x_j}G_{k-1,l}(x)\right\rvert\right)\\
&\leq6\lVert G_{k-1}(x)\rVert_\infty\left(\lVert J[F_{k-1}](x)\rVert_\infty+\lVert J[G_{k-1}](x)\rVert_\infty\right)\lVert J[F_{k-1}](x)-J[G_{k-1}](x)\rVert_\infty\\
&\leq6W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}\times2W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+2}3^{k-2}\\
&\quad\times\delta W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}2^{\frac{3^{k-1}-1}{2}-k+1}3^{2k-4}\lesssim\delta W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}.
\end{aligned} V 2 = 6 η 1 [ G k − 1 , l ( x )] j = 1 ∑ d [ ( ∂ x j ∂ F k − 1 , l ( x ) ) 2 − ( ∂ x j ∂ G k − 1 , l ( x ) ) 2 ] ≤ 6 ∥ G k − 1 ( x ) ∥ ∞ j = 1 ∑ d ( ∂ x j ∂ F k − 1 , l ( x ) + ∂ x j ∂ G k − 1 , l ( x ) ) ∂ x j ∂ F k − 1 , l ( x ) − ∂ x j ∂ G k − 1 , l ( x ) ≤ 6 ∥ G k − 1 ( x ) ∥ ∞ ( j = 1 ∑ d ∂ x j ∂ F k − 1 , l ( x ) + j = 1 ∑ d ∂ x j ∂ G k − 1 , l ( x ) ) ( j = 1 ∑ d ∂ x j ∂ F k − 1 , l ( x ) − ∂ x j ∂ G k − 1 , l ( x ) ) ≤ 6 ∥ G k − 1 ( x ) ∥ ∞ ( ∥ J [ F k − 1 ] ( x ) ∥ ∞ + ∥ J [ G k − 1 ] ( x ) ∥ ∞ ) ∥ J [ F k − 1 ] ( x ) − J [ G k − 1 ] ( x ) ∥ ∞ ≤ 6 W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 × 2 W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 2 3 k − 2 × δ W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 2 2 3 k − 1 − 1 − k + 1 3 2 k − 4 ≲ δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 .
where the last step above follows from k ≤ L k\leq L k ≤ L and L = O ( 1 ) L=O(1) L = O ( 1 ) .
Moreover, using Lemma A.15 , Lemma A.16 and Lemma A.23 helps us upper bound V 3 V_3 V 3 by:
V 3 = ( 3 η 2 [ F k − 1 , l ( x ) ] − 3 η 2 [ G k − 1 , l ( x ) ] ) ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) ≤ ∣ 3 η 2 [ F k − 1 , l ( x ) ] − 3 η 2 [ G k − 1 , l ( x ) ] ∣ ∣ ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) ∣ ≤ 6 ( sup x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) ∥ F k − 1 ( x ) ∥ ∞ ) ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ∥ Δ [ F k − 1 ] ( x ) ∥ ∞ ≲ 6 W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − k + 2 × δ W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 3 k − 2 × W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 ≲ δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 . \begin{aligned}
V_3 &= \left(3\eta_2[F_{k-1,l}(x)]-3\eta_2[G_{k-1,l}(x)]\right)\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)\\
&\le\left|3\eta_2[F_{k-1,l}(x)]-3\eta_2[G_{k-1,l}(x)]\right|\left|\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)\right|\\
&\le6\left(\sup_{x\in\Omega,\ F_{k-1}\in\Phi_{k-1}(L,W,S,B)}\|F_{k-1}(x)\|_\infty\right)\|F_{k-1}(x)-G_{k-1}(x)\|_\infty\|\Delta[F_{k-1}](x)\|_\infty\\
&\lesssim6W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}3^{k-1-k+2}\times\delta W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}3^{k-2}\\
&\quad\times W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}\lesssim\delta W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}.
\end{aligned} V 3 = ( 3 η 2 [ F k − 1 , l ( x )] − 3 η 2 [ G k − 1 , l ( x )] ) j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) ≤ ∣ 3 η 2 [ F k − 1 , l ( x )] − 3 η 2 [ G k − 1 , l ( x )] ∣ j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) ≤ 6 ( x ∈ Ω , F k − 1 ∈ Φ k − 1 ( L , W , S , B ) sup ∥ F k − 1 ( x ) ∥ ∞ ) ∥ F k − 1 ( x ) − G k − 1 ( x ) ∥ ∞ ∥Δ [ F k − 1 ] ( x ) ∥ ∞ ≲ 6 W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 3 k − 1 − k + 2 × δ W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 3 k − 2 × W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 ≲ δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 .
where the last step above follows from k ≤ L k\le L k ≤ L and L = O ( 1 ) L=O(1) L = O ( 1 ) .
Finally, applying Lemma A.15 and inductive hypothesis helps us upper bound V 4 V_4 V 4 by:
V 4 = 3 η 2 [ G k − 1 , l ( x ) ] ( ∑ j = 1 d ∂ 2 ∂ x j 2 F k − 1 , l ( x ) − ∑ j = 1 d ∂ 2 ∂ x j 2 G k − 1 , l ( x ) ) ≤ 3 ∥ G k − 1 ( x ) ∥ ∞ 2 ∥ Δ [ F k − 1 ] ( x ) − Δ [ G k − 1 ] ( x ) ∥ ∞ ≲ 3 W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × δ W 3 k − 2 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 ≲ δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 . \begin{aligned}
V_4 &= 3\eta_2[G_{k-1,l}(x)]\left(\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}F_{k-1,l}(x)-\sum_{j=1}^{d}\frac{\partial^2}{\partial x_j^2}G_{k-1,l}(x)\right)\\
&\le3\|G_{k-1}(x)\|_\infty^2\|\Delta[F_{k-1}](x)-\Delta[G_{k-1}](x)\|_\infty\\
&\lesssim3W^{3^{k-2}-1}(B\vee d)^{5\cdot3^{k-2}-1}2^{3^{k-1}-1-2k+4}\times\delta W^{\frac{3^{k-2}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-2}-1}{2}}\\
&\lesssim\delta W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}.
\end{aligned} V 4 = 3 η 2 [ G k − 1 , l ( x )] ( j = 1 ∑ d ∂ x j 2 ∂ 2 F k − 1 , l ( x ) − j = 1 ∑ d ∂ x j 2 ∂ 2 G k − 1 , l ( x ) ) ≤ 3∥ G k − 1 ( x ) ∥ ∞ 2 ∥Δ [ F k − 1 ] ( x ) − Δ [ G k − 1 ] ( x ) ∥ ∞ ≲ 3 W 3 k − 2 − 1 ( B ∨ d ) 5 ⋅ 3 k − 2 − 1 2 3 k − 1 − 1 − 2 k + 4 × δ W 2 3 k − 2 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 2 − 1 ≲ δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 .
where the last step above follows from k ≤ L k\le L k ≤ L and L = O ( 1 ) L=O(1) L = O ( 1 ) .
Combining the four bounds on V 1 V_1 V 1 , V 2 V_2 V 2 , V 3 V_3 V 3 and V 4 V_4 V 4 implies:
( Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ) l = ∑ i = 1 4 V i ≲ δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 \left(\Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x)\right)_l
=\sum_{i=1}^{4}V_i\lesssim\delta W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}} ( Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ) l = i = 1 ∑ 4 V i ≲ δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3
Taking supremum with respect to 1 ≤ l ≤ W 1\le l\le W 1 ≤ l ≤ W gives us an upper bound on the second term ∥ W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ \|W_G^{(k)}\|_\infty\|\Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x)\|_\infty ∥ W G ( k ) ∥ ∞ ∥Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ :
∥ W G ( k ) ∥ ∞ ∥ Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ ≲ W B × δ W 3 k − 1 − 3 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 = δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 . (80) \begin{aligned}
\|W_G^{(k)}\|_\infty\|\Delta[\eta_3\circ F_{k-1}](x)-\Delta[\eta_3\circ G_{k-1}](x)\|_\infty
&\lesssim WB\times\delta W^{\frac{3^{k-1}-3}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}\\
&=\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}. \tag*{(80)}
\end{aligned} ∥ W G ( k ) ∥ ∞ ∥Δ [ η 3 ∘ F k − 1 ] ( x ) − Δ [ η 3 ∘ G k − 1 ] ( x ) ∥ ∞ ≲ W B × δ W 2 3 k − 1 − 3 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 = δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 . (80)
Combining the two bounds derived in equation A.38 and equation A.40 then implies:
∥ Δ [ F k ] ( x ) − Δ [ G k ] ( x ) ∥ ∞ ≲ δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 3 2 + δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 ≲ δ W 3 k − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 k − 1 − 1 2 . \begin{aligned}
\|\Delta[F_k](x)-\Delta[G_k](x)\|_\infty
&\lesssim\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-3}{2}}+\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}\\
&\lesssim\delta W^{\frac{3^{k-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{k-1}-1}{2}}.
\end{aligned} ∥Δ [ F k ] ( x ) − Δ [ G k ] ( x ) ∥ ∞ ≲ δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 3 + δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 ≲ δ W 2 3 k − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 k − 1 − 1 .
Taking supremum with respect to x ∈ Ω x\in\Omega x ∈ Ω on the LHS implies that the given upper bound also holds for k k k . By induction, the claim is proved.
Given a Neural Network function space Φ ( L , W , S , B ) \Phi(L, W, S, B) Φ ( L , W , S , B ) , we define a corresponding Neural Network Laplacian space Δ Φ ( L , W , S , B ) \Delta\Phi(L, W, S, B) ΔΦ ( L , W , S , B ) as:
Δ Φ ( L , W , S , B ) : = { Δ F ∣ F ∈ Φ ( L , W , S , B ) } . (81) \Delta\Phi(L, W, S, B) := \{\Delta F \mid F \in\Phi(L, W, S, B)\}. \tag*{(81)} ΔΦ ( L , W , S , B ) := { Δ F ∣ F ∈ Φ ( L , W , S , B )} . (81)
Theorem A.25. (Bounding the Neural Network Laplacian space covering number) Fix some sufficiently large N ∈ Z + N \in\mathbb{Z}^{+} N ∈ Z + . Consider a Deep Neural Network space Φ ( L , W , S , B ) \Phi(L, W, S, B) Φ ( L , W , S , B ) with L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( N ) W = O(N) W = O ( N ) , S = O ( N ) S = O(N) S = O ( N ) and B = O ( N ) B = O(N) B = O ( N ) . Then the log value of the covering number of the DNN Laplacian space with respect to the ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_{\infty} ∥ ⋅ ∥ ∞ norm ∥ F ( x ) ∥ ∞ : = sup x ∈ Ω ∣ F ( x ) ∣ \lVert F(x)\rVert_{\infty} := \sup_{x\in\Omega}|F(x)| ∥ F ( x ) ∥ ∞ := sup x ∈ Ω ∣ F ( x ) ∣ , which is denoted by N ( δ , Δ Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) \mathcal{N}(\delta, \Delta\Phi(L, W, S, B), \lVert\cdot\rVert_{\infty}) N ( δ , ΔΦ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) , can be upper bounded by:
log N ( δ , Δ Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) = O ( S [ log ( δ − 1 ) + 3 L log ( W B ) ] ) (82) \log\mathcal{N}(\delta, \Delta\Phi(L, W, S, B), \lVert\cdot\rVert_{\infty}) = O\left(S\left[\log(\delta^{-1}) + 3^{L}\log(WB)\right]\right) \tag*{(82)} log N ( δ , ΔΦ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) = O ( S [ log ( δ − 1 ) + 3 L log ( W B ) ] ) (82)
Proof. We firstly fix a sparsity pattern (i.e, the locations of the non-zero entries are fixed). Applying Lemma A.24 yields that there exists some constant C = O ( 1 ) C = O(1) C = O ( 1 ) , such that the covering number with respect to ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_{\infty} ∥ ⋅ ∥ ∞ can be upper bounded by:
( δ C W 3 L − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 L − 1 − 1 2 ) − S \left(\frac{\delta}{CW^{\frac{3^{L-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{L-1}-1}{2}}}\right)^{-S} ( C W 2 3 L − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 L − 1 − 1 δ ) − S
Furthermore, note that the number of feasible configurations is upper bounded by ( ( W + 1 ) L S ) ≤ ( W + 1 ) L S \binom{(W+1)^{L}}{S} \le(W+1)^{LS} ( S ( W + 1 ) L ) ≤ ( W + 1 ) L S [63 , 18 ] . Then we plug this into the previous estimation and yields:
log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≤ log [ ( W + 1 ) L S ( δ C W 3 L − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 L − 1 − 1 2 ) − S ] ≤ S log [ δ − 1 ( W + 1 ) L W 3 L − 1 − 1 2 ( B ∨ d ) 5 ⋅ 3 L − 1 − 1 2 ] ≤ S [ log ( δ − 1 ) + L log ( W ) + 3 L log ( W ( B ∨ d ) ) ] \begin{aligned}
\log\mathcal{N}(\delta, \Phi(L, W, S, B), \lVert\cdot\rVert_{\infty}) \le\log\left[(W+1)^{LS}\left(\frac{\delta}{CW^{\frac{3^{L-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{L-1}-1}{2}}}\right)^{-S}\right] \\
&\le S\log\left[\delta^{-1}(W+1)^{L}W^{\frac{3^{L-1}-1}{2}}(B\vee d)^{\frac{5\cdot3^{L-1}-1}{2}}\right] \\
&\le S\left[\log(\delta^{-1})+L\log(W)+3^{L}\log(W(B\vee d))\right]
\end{aligned} log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≤ log ( W + 1 ) L S ( C W 2 3 L − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 L − 1 − 1 δ ) − S ≤ S log [ δ − 1 ( W + 1 ) L W 2 3 L − 1 − 1 ( B ∨ d ) 2 5 ⋅ 3 L − 1 − 1 ] ≤ S [ log ( δ − 1 ) + L log ( W ) + 3 L log ( W ( B ∨ d )) ]
Note that here the dimension d d d is some constant. Thus, by plugging in the given magnitudes L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( N ) W = O(N) W = O ( N ) , S = O ( N ) S = O(N) S = O ( N ) and B = O ( N ) B = O(N) B = O ( N ) , we can further deduce that:
log N ( δ , Δ Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≲ S [ log ( δ − 1 ) + 3 L log ( W B ) ] \log\mathcal{N}(\delta, \Delta\Phi(L, W, S, B), \lVert\cdot\rVert_{\infty}) \lesssim S\left[\log(\delta^{-1})+3^{L}\log(WB)\right] log N ( δ , ΔΦ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) ≲ S [ log ( δ − 1 ) + 3 L log ( W B ) ]
This finishes our proof.
Lemma A.26 (Local Rademacher Complexity Bound for Deep Ritz Method). Consider a Deep Neural Network space F ( Ω ) = Φ ( L , W , S , B ) \mathcal{F}(\Omega) = \Phi(L, W, S, B) F ( Ω ) = Φ ( L , W , S , B ) with L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( N ) W = O(N) W = O ( N ) , S = O ( N ) S = O(N) S = O ( N ) and B = O ( N ) B = O(N) B = O ( N ) , where N ∈ Z + N \in\mathbb{Z}^{+} N ∈ Z + is fixed to be sufficiently large. Moreover, assume that the gradients and function value of F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , V V V and f f f are uniformly bounded
max { sup u ∈ F ( Ω ) ∥ u ∥ L ∞ ( Ω ) , sup u ∈ F ( Ω ) ∥ ∇ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ ∇ u ∗ ∥ L ∞ ( Ω ) , V max , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (83) \max\left\{\sup_{u\in\mathcal{F}(\Omega)}\lVert u\rVert_{L^{\infty}(\Omega)},\sup_{u\in\mathcal{F}(\Omega)}\lVert\nabla u\rVert_{L^{\infty}(\Omega)},\lVert u^{*}\rVert_{L^{\infty}(\Omega)},\lVert\nabla u^{*}\rVert_{L^{\infty}(\Omega)},V_{\max},\lVert f\rVert_{L^{\infty}(\Omega)}\right\}\le C. \tag*{(83)} max { u ∈ F ( Ω ) sup ∥ u ∥ L ∞ ( Ω ) , u ∈ F ( Ω ) sup ∥ ∇ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ ∇ u ∗ ∥ L ∞ ( Ω ) , V m a x , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (83)
For any ρ > 0 \rho> 0 ρ > 0 , we consider a localized set L ρ L_\rho L ρ defined by:
L ρ ( Ω ) : = { u : u ∈ F ( Ω ) , ∥ u − u ∗ ∥ H 1 2 ≤ ρ } . L_\rho(\Omega) := \{u : u \in\mathcal{F}(\Omega), \|u-u^*\|_{H^1}^2 \le\rho\}. L ρ ( Ω ) := { u : u ∈ F ( Ω ) , ∥ u − u ∗ ∥ H 1 2 ≤ ρ } .
Then for any ρ ≳ n − 2 \rho\gtrsim n^{-2} ρ ≳ n − 2 , the Rademacher complexity of a localized function space S ρ ( Ω ) : = { h : = ∣ Ω ∣ [ 1 2 ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) + 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] | u ∈ L ρ ( Ω ) } \mathcal{S}_\rho(\Omega) := \left\{h := |\Omega|\left[\frac{1}{2}\left(\|\nabla u\|^2-\|\nabla u^*\|^2\right)+\frac{1}{2}V(|u|^2-|u^*|^2)-f(u-u^*)\right]\ \middle|\ u\in L_\rho(\Omega)\right\} S ρ ( Ω ) := { h := ∣Ω∣ [ 2 1 ( ∥∇ u ∥ 2 − ∥∇ u ∗ ∥ 2 ) + 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] u ∈ L ρ ( Ω ) } can be upper bounded by a sub-root function
ϕ ( ρ ) : = O ( S 3 L ρ n log ( B W n ) ) . \phi(\rho) := O\left(\sqrt{\frac{S^3L\rho}{n}\log(BWn)}\right). ϕ ( ρ ) := O ( n S 3 L ρ log ( B W n ) ) .
i.e. we have
ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( S ρ ( Ω ) ) ≤ ϕ ( ρ ) . (84) \phi(4\rho) \le2\phi(\rho)\ \text{and}\ R_n(\mathcal{S}_\rho(\Omega)) \le\phi(\rho). \tag*{(84)} ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( S ρ ( Ω )) ≤ ϕ ( ρ ) . (84)
holds for all ρ ≳ n − 2 \rho\gtrsim n^{-2} ρ ≳ n − 2 .
Proof. Firstly, we will check that for any u ∈ L ρ ( Ω ) u\in L_\rho(\Omega) u ∈ L ρ ( Ω ) , the corresponding function h h h in S ρ ( Ω ) \mathcal{S}_\rho(\Omega) S ρ ( Ω ) is Lipschitz with respect to u − u ∗ u-u^* u − u ∗ and ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ \|\nabla u\|-\|\nabla u^*\| ∥∇ u ∥ − ∥∇ u ∗ ∥ . Note that for any u 1 , u 2 ∈ L ρ ( Ω ) u_1,u_2\in L_\rho(\Omega) u 1 , u 2 ∈ L ρ ( Ω ) with corresponding functions h 1 , h 2 ∈ S ρ ( Ω ) h_1,h_2\in\mathcal{S}_\rho(\Omega) h 1 , h 2 ∈ S ρ ( Ω ) , applying boundedness condition 4.10 yields:
∣ h 1 ( x ) − h 2 ( x ) ∣ ≤ 1 2 ∣ ∥ ∇ u 1 ( x ) ∥ 2 − ∥ ∇ u 2 ( x ) ∥ 2 ∣ + 1 2 ∣ V ( x ) ∣ ∣ u 1 ( x ) 2 − u 2 ( x ) 2 ∣ + ∣ f ( x ) ∣ ∣ u 1 ( x ) − u 2 ( x ) ∣ ≤ C ∣ ∥ ∇ u 1 ( x ) ∥ − ∥ ∇ u 2 ( x ) ∥ ∣ + ( C 2 + C ) ∣ u 1 ( x ) − u 2 ( x ) ∣ = C ∣ ( ∥ ∇ u 1 ( x ) ∥ − ∥ ∇ u ∗ ( x ) ∥ ) − ( ∥ ∇ u 2 ( x ) ∥ − ∥ ∇ u ∗ ( x ) ∥ ) ∣ + ( C 2 + C ) ∣ ( u 1 ( x ) − u ∗ ( x ) ) − ( u 2 ( x ) − u ∗ ( x ) ) ∣ . \begin{aligned}
|h_1(x)-h_2(x)| &\le\frac{1}{2}\left|\|\nabla u_1(x)\|^2-\|\nabla u_2(x)\|^2\right|+\frac{1}{2}|V(x)||u_1(x)^2-u_2(x)^2|+|f(x)||u_1(x)-u_2(x)|\\
&\le C\left|\|\nabla u_1(x)\|-\|\nabla u_2(x)\|\right|+(C^2+C)|u_1(x)-u_2(x)|\\
&= C\left|\left(\|\nabla u_1(x)\|-\|\nabla u^*(x)\|\right)-\left(\|\nabla u_2(x)\|-\|\nabla u^*(x)\|\right)\right|\\
&\quad+(C^2+C)\left|(u_1(x)-u^*(x))-(u_2(x)-u^*(x))\right|.
\end{aligned} ∣ h 1 ( x ) − h 2 ( x ) ∣ ≤ 2 1 ∥∇ u 1 ( x ) ∥ 2 − ∥∇ u 2 ( x ) ∥ 2 + 2 1 ∣ V ( x ) ∣∣ u 1 ( x ) 2 − u 2 ( x ) 2 ∣ + ∣ f ( x ) ∣∣ u 1 ( x ) − u 2 ( x ) ∣ ≤ C ∣ ∥∇ u 1 ( x ) ∥ − ∥∇ u 2 ( x ) ∥ ∣ + ( C 2 + C ) ∣ u 1 ( x ) − u 2 ( x ) ∣ = C ∣ ( ∥∇ u 1 ( x ) ∥ − ∥∇ u ∗ ( x ) ∥ ) − ( ∥∇ u 2 ( x ) ∥ − ∥∇ u ∗ ( x ) ∥ ) ∣ + ( C 2 + C ) ∣ ( u 1 ( x ) − u ∗ ( x )) − ( u 2 ( x ) − u ∗ ( x )) ∣ .
Let’s pick L = C 2 + C > C L=C^2+C>C L = C 2 + C > C . Applying the Talagrand Contraction Lemma A.4 helps us upper bound the local Rademacher complexity R n ( S ρ ( Ω ) ) R_n(\mathcal{S}_\rho(\Omega)) R n ( S ρ ( Ω )) by
R n ( S ρ ( Ω ) ) = E x E σ [ sup u ∈ L ρ ( Ω ) 1 n ∑ i = 1 n σ i [ 1 2 ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) + 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] ] ≤ 2 L E x E σ [ sup u ∈ L ρ ( Ω ) 1 n ∑ i = 1 n σ i ( u ( x i ) − u ∗ ( x i ) ) ] + 2 L E x ′ E σ ′ [ sup u ∈ L ρ ( Ω ) 1 n ∑ i = 1 n σ i ′ ( ∥ ∇ u ( x i ′ ) ∥ − ∥ ∇ u ∗ ( x i ′ ) ∥ ) ] ≲ R n ( { u − u ∗ : u ∈ L ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ L ρ } ) . \begin{aligned}
R_n(\mathcal{S}_\rho(\Omega)) &= \mathbb{E}_{x}\mathbb{E}_{\sigma}\left[\sup_{u\in L_\rho(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\left[\frac{1}{2}\left(\|\nabla u\|^2-\|\nabla u^*\|^2\right)+\frac{1}{2}V(|u|^2-|u^*|^2)-f(u-u^*)\right]\right]\\
&\le2L\mathbb{E}_{x}\mathbb{E}_{\sigma}\left[\sup_{u\in L_\rho(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i(u(x_i)-u^*(x_i))\right]\\
&\quad+2L\mathbb{E}_{x'}\mathbb{E}_{\sigma'}\left[\sup_{u\in L_\rho(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i'(\|\nabla u(x_i')\|-\|\nabla u^*(x_i')\|)\right]\\
&\lesssim R_n\left(\{u-u^*:u\in L_\rho\}\right)+R_n\left(\{\|\nabla u\|-\|\nabla u^*\|:u\in L_\rho\}\right).
\end{aligned} R n ( S ρ ( Ω )) = E x E σ [ u ∈ L ρ ( Ω ) sup n 1 i = 1 ∑ n σ i [ 2 1 ( ∥∇ u ∥ 2 − ∥∇ u ∗ ∥ 2 ) + 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] ] ≤ 2 L E x E σ [ u ∈ L ρ ( Ω ) sup n 1 i = 1 ∑ n σ i ( u ( x i ) − u ∗ ( x i )) ] + 2 L E x ′ E σ ′ [ u ∈ L ρ ( Ω ) sup n 1 i = 1 ∑ n σ i ′ ( ∥∇ u ( x i ′ ) ∥ − ∥∇ u ∗ ( x i ′ ) ∥ ) ] ≲ R n ( { u − u ∗ : u ∈ L ρ } ) + R n ( { ∥∇ u ∥ − ∥∇ u ∗ ∥ : u ∈ L ρ } ) .
From the localization constraint ρ ≥ ∥ u − u ∗ ∥ H 1 ( Ω ) 2 = ∥ u − u ∗ ∥ L 2 ( Ω ) 2 + ∥ ∇ u − ∇ u ∗ ∥ L 2 ( Ω ) 2 \rho\ge\|u-u^*\|_{H^1(\Omega)}^2=\|u-u^*\|_{L^2(\Omega)}^2+\|\nabla u-\nabla u^*\|_{L^2(\Omega)}^2 ρ ≥ ∥ u − u ∗ ∥ H 1 ( Ω ) 2 = ∥ u − u ∗ ∥ L 2 ( Ω ) 2 + ∥∇ u − ∇ u ∗ ∥ L 2 ( Ω ) 2 , we can deduce that
∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ and ∥ ∇ u − ∇ u ∗ ∥ L 2 ( Ω ) ≤ ρ (85) \|u-u^*\|_{L^2(\Omega)}\le\sqrt{\rho}\ \text{and}\ \|\nabla u-\nabla u^*\|_{L^2(\Omega)}\le\sqrt{\rho} \tag*{(85)} ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ and ∥∇ u − ∇ u ∗ ∥ L 2 ( Ω ) ≤ ρ (85)
Moreover, note that Ω ⊂ [ 0 , 1 ] d \Omega\subset[0,1]^d Ω ⊂ [ 0 , 1 ] d . Applying triangle inequality yields:
∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ L 2 ( Ω ) 2 = ∫ Ω ∣ ∥ ∇ u ( x ) ∥ − ∥ ∇ u ∗ ( x ) ∥ ∣ 2 d x ≤ ∫ Ω ∥ ∇ u ( x ) − ∇ u ∗ ( x ) ∥ 2 d x = ∥ ∇ u − ∇ u ∗ ∥ L 2 ( Ω ) 2 ≤ ρ ⇒ ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ L 2 ( Ω ) ≤ ρ (86) \begin{aligned}
\left\lVert\lVert\nabla u\rVert-\lVert\nabla u^*\rVert\right\rVert_{L^2(\Omega)}^2
&=\int_{\Omega}\left|\lVert\nabla u(x)\rVert-\lVert\nabla u^*(x)\rVert\right|^2\,dx
\le\int_{\Omega}\lVert\nabla u(x)-\nabla u^*(x)\rVert^2\,dx\\
&=\lVert\nabla u-\nabla u^*\rVert_{L^2(\Omega)}^2\le\rho
\Rightarrow\left\lVert\lVert\nabla u\rVert-\lVert\nabla u^*\rVert\right\rVert_{L^2(\Omega)}
\le\sqrt{\rho} \tag*{(86)}
\end{aligned} ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ L 2 ( Ω ) 2 = ∫ Ω ∣ ∥ ∇ u ( x )∥ − ∥ ∇ u ∗ ( x )∥ ∣ 2 d x ≤ ∫ Ω ∥ ∇ u ( x ) − ∇ u ∗ ( x ) ∥ 2 d x = ∥ ∇ u − ∇ u ∗ ∥ L 2 ( Ω ) 2 ≤ ρ ⇒ ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ L 2 ( Ω ) ≤ ρ (86)
Using inequality A.45 and inequality A.46, we have:
R n ( S ρ ( Ω ) ) ≲ R n ( { u − u ∗ : u ∈ L ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ L ρ } ) ≤ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ Φ ( L , W , S , B ) , ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ L 2 ( Ω ) ≤ ρ } ) \begin{aligned}
R_n(S_\rho(\Omega)) &\lesssim R_n\left(\left\{u-u^*:u\in L_\rho\right\}\right)+R_n\left(\left\{\lVert\nabla u\rVert-\lVert\nabla u^*\rVert:u\in L_\rho\right\}\right)\\
&\le R_n\left(\left\{u-u^*:u\in\Phi(L,W,S,B),\lVert u-u^*\rVert_{L^2(\Omega)}\le\sqrt{\rho}\right\}\right)\\
&\quad+R_n\left(\left\{\lVert\nabla u\rVert-\lVert\nabla u^*\rVert:u\in\Phi(L,W,S,B),\left\lVert\lVert\nabla u\rVert-\lVert\nabla u^*\rVert\right\rVert_{L^2(\Omega)}\le\sqrt{\rho}\right\}\right)
\end{aligned} R n ( S ρ ( Ω )) ≲ R n ( { u − u ∗ : u ∈ L ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ L ρ } ) ≤ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ Φ ( L , W , S , B ) , ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ L 2 ( Ω ) ≤ ρ } )
[20 , 59 ] showed a “upper isometry” property, where the metric ∥ ⋅ ∥ L 2 \lVert\cdot\rVert_{L_2} ∥ ⋅ ∥ L 2 is equivalent to ∥ ⋅ ∥ n , 2 \lVert\cdot\rVert_{n,2} ∥ ⋅ ∥ n , 2 with high probability. Combining this fact with Theorem A.5 , we can bound the local Rademacher complexities using Dudley integral:
R n ( S ρ ( Ω ) ) ≤ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ Φ ( L , W , S , B ) , ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ L 2 ( Ω ) ≤ ρ } ) ≤ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ n , 2 ≤ 2 ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ Φ ( L , W , S , B ) , ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ n , 2 ≤ 2 ρ } ) ≲ inf 0 < α < 2 ρ { 4 α + 12 n ∫ α 2 ρ log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ n ) d δ } + inf 0 < α < 2 ρ { 4 α + 12 n ∫ α 2 ρ log N ( δ , ∇ Φ ( L , W , S , B ) , ∥ ⋅ ∥ n , 2 ) d δ } ≲ inf 0 < α < 2 ρ { 4 α + 12 n ∫ α 2 ρ log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) d δ } + inf 0 < α < 2 ρ { 4 α + 12 n ∫ α 2 ρ log N ( δ , ∇ Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) d δ } \begin{aligned}
R_n(S_\rho(\Omega)) &\le R_n\left(\left\{u-u^*:u\in\Phi(L,W,S,B),\lVert u-u^*\rVert_{L^2(\Omega)}\le\sqrt{\rho}\right\}\right)\\
&\quad+R_n\left(\left\{\lVert\nabla u\rVert-\lVert\nabla u^*\rVert:u\in\Phi(L,W,S,B),\left\lVert\lVert\nabla u\rVert-\lVert\nabla u^*\rVert\right\rVert_{L^2(\Omega)}\le\sqrt{\rho}\right\}\right)\\
&\le R_n\left(\left\{u-u^*:u\in\Phi(L,W,S,B),\lVert u-u^*\rVert_{n,2}\le2\sqrt{\rho}\right\}\right)\\
&\quad+R_n\left(\left\{\lVert\nabla u\rVert-\lVert\nabla u^*\rVert:u\in\Phi(L,W,S,B),\left\lVert\lVert\nabla u\rVert-\lVert\nabla u^*\rVert\right\rVert_{n,2}\le2\sqrt{\rho}\right\}\right)\\
&\lesssim\inf_{0<\alpha<2\sqrt{\rho}}\left\{4\alpha+\frac{12}{\sqrt{n}}\int_{\alpha}^{2\sqrt{\rho}}\sqrt{\log\mathcal{N}(\delta,\Phi(L,W,S,B),\lVert\cdot\rVert_n)}\,d\delta\right\}\\
&\quad+\inf_{0<\alpha<2\sqrt{\rho}}\left\{4\alpha+\frac{12}{\sqrt{n}}\int_{\alpha}^{2\sqrt{\rho}}\sqrt{\log\mathcal{N}(\delta,\nabla\Phi(L,W,S,B),\lVert\cdot\rVert_{n,2})}\,d\delta\right\}\\
&\lesssim\inf_{0<\alpha<2\sqrt{\rho}}\left\{4\alpha+\frac{12}{\sqrt{n}}\int_{\alpha}^{2\sqrt{\rho}}\sqrt{\log\mathcal{N}(\delta,\Phi(L,W,S,B),\lVert\cdot\rVert_\infty)}\,d\delta\right\}\\
&\quad+\inf_{0<\alpha<2\sqrt{\rho}}\left\{4\alpha+\frac{12}{\sqrt{n}}\int_{\alpha}^{2\sqrt{\rho}}\sqrt{\log\mathcal{N}(\delta,\nabla\Phi(L,W,S,B),\lVert\cdot\rVert_\infty)}\,d\delta\right\}
\end{aligned} R n ( S ρ ( Ω )) ≤ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ Φ ( L , W , S , B ) , ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ L 2 ( Ω ) ≤ ρ } ) ≤ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ n , 2 ≤ 2 ρ } ) + R n ( { ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ : u ∈ Φ ( L , W , S , B ) , ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ n , 2 ≤ 2 ρ } ) ≲ 0 < α < 2 ρ inf { 4 α + n 12 ∫ α 2 ρ log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ n ) d δ } + 0 < α < 2 ρ inf { 4 α + n 12 ∫ α 2 ρ log N ( δ , ∇Φ ( L , W , S , B ) , ∥ ⋅ ∥ n , 2 ) d δ } ≲ 0 < α < 2 ρ inf { 4 α + n 12 ∫ α 2 ρ log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) d δ } + 0 < α < 2 ρ inf { 4 α + n 12 ∫ α 2 ρ log N ( δ , ∇Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) d δ }
For any ρ ≥ 1 n 2 \rho\ge\frac{1}{n^2} ρ ≥ n 2 1 , we pick α = 1 n ≤ ρ \alpha=\frac{1}{n}\le\sqrt{\rho} α = n 1 ≤ ρ and plug in the upper bounds proved in Theorem A.18 and Theorem A.22 , which implies:
R n ( S ρ ( Ω ) ) ≲ 1 n + 1 n ∫ 1 n 2 ρ S [ log ( δ − 1 ) + 3 L log ( W B ) ] d δ + 1 n ∫ 1 n 2 ρ S [ log ( δ − 1 ) + 3 L log ( W B ) ] d δ ≲ S 3 L ρ n log ( B W n ) \begin{aligned}
R_n(S_\rho(\Omega))&\lesssim\frac{1}{n}+\frac{1}{\sqrt{n}}\int_{\frac{1}{n}}^{2\sqrt{\rho}}\sqrt{S\left[\log(\delta^{-1})+3L\log(WB)\right]}\,d\delta\\
&\quad+\frac{1}{\sqrt{n}}\int_{\frac{1}{n}}^{2\sqrt{\rho}}\sqrt{S\left[\log(\delta^{-1})+3L\log(WB)\right]}\,d\delta\\
&\lesssim\sqrt{\frac{S3L\rho}{n}\log(BWn)}
\end{aligned} R n ( S ρ ( Ω )) ≲ n 1 + n 1 ∫ n 1 2 ρ S [ log ( δ − 1 ) + 3 L log ( W B ) ] d δ + n 1 ∫ n 1 2 ρ S [ log ( δ − 1 ) + 3 L log ( W B ) ] d δ ≲ n S 3 L ρ log ( B W n )
Lemma A.27 (Local Rademacher Complexity Bound for Physics Informed Neural Network). Consider a Deep Neural Network space F ( Ω ) = Φ ( L , W , S , B ) \mathcal{F}(\Omega) = \Phi(L,W,S,B) F ( Ω ) = Φ ( L , W , S , B ) with L = O ( 1 ) L = O(1) L = O ( 1 ) , W = O ( N ) W = O(N) W = O ( N ) , S = O ( N ) S = O(N) S = O ( N ) and B = O ( N ) B = O(N) B = O ( N ) , where N ∈ Z + N \in\mathbb{Z}^{+} N ∈ Z + is fixed to be sufficiently large. Moreover, assume that the gradients and function value of F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , V V V and f f f are uniformly bounded
max { sup u ∈ F ( Ω ) ∥ u ∥ L ∞ ( Ω ) , sup u ∈ F ( Ω ) ∥ Δ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ Δ u ∗ ∥ L ∞ ( Ω ) , V max , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (87) \max\left\{\sup_{u\in\mathcal{F}(\Omega)}\lVert u\rVert_{L^\infty(\Omega)},\sup_{u\in\mathcal{F}(\Omega)}\lVert\Delta u\rVert_{L^\infty(\Omega)},\lVert u^*\rVert_{L^\infty(\Omega)},\lVert\Delta u^*\rVert_{L^\infty(\Omega)},V_{\max},\lVert f\rVert_{L^\infty(\Omega)}\right\}\le C. \tag*{(87)} max { u ∈ F ( Ω ) sup ∥ u ∥ L ∞ ( Ω ) , u ∈ F ( Ω ) sup ∥ Δ u ∥ L ∞ ( Ω ) , ∥ u ∗ ∥ L ∞ ( Ω ) , ∥ Δ u ∗ ∥ L ∞ ( Ω ) , V m a x , ∥ f ∥ L ∞ ( Ω ) } ≤ C . (87) For any ρ > 0 \rho>0 ρ > 0 , we consider a localized set M ρ M_\rho M ρ defined by:
M ρ ( Ω ) : = { u : u ∈ F ( Ω ) , ∥ u − u ∗ ∥ H 2 2 ≤ ρ } . M_\rho(\Omega):=\left\{u:u\in\mathcal{F}(\Omega),\lVert u-u^*\rVert_{H^2}^2\le\rho\right\}. M ρ ( Ω ) := { u : u ∈ F ( Ω ) , ∥ u − u ∗ ∥ H 2 2 ≤ ρ } . Then for any ρ ≳ n − 2 \rho\gtrsim n^{-2} ρ ≳ n − 2 , the Rademacher complexity of a localized function space T ρ ( Ω ) : = { h : = ∣ Ω ∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] ∣ u ∈ M ρ ( Ω ) } T_\rho(\Omega):=\left\{h:=|\Omega|\cdot\left[(\Delta u-Vu+f)^2-(\Delta u^*-Vu^*+f)^2\right]\mid u\in M_\rho(\Omega)\right\} T ρ ( Ω ) := { h := ∣Ω∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] ∣ u ∈ M ρ ( Ω ) } can be upper bounded by a sub-root function
ϕ ( ρ ) : = O ( S 3 L ρ n log ( B W n ) ) . \phi(\rho):=O\left(\sqrt{\frac{S^3L\rho}{n}}\log(BWn)\right). ϕ ( ρ ) := O ( n S 3 L ρ log ( B W n ) ) . i.e. we have
ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( T ρ ( Ω ) ) ≤ ϕ ( ρ ) . (88) \phi(4\rho)\le2\phi(\rho)\ \text{and}\ R_n(T_\rho(\Omega))\le\phi(\rho). \tag*{(88)} ϕ ( 4 ρ ) ≤ 2 ϕ ( ρ ) and R n ( T ρ ( Ω )) ≤ ϕ ( ρ ) . (88) holds for all ρ ≳ n − 2 \rho\gtrsim n^{-2} ρ ≳ n − 2 .
Proof. Firstly, we will check that for any u ∈ L ρ ( Ω ) u\in L_\rho(\Omega) u ∈ L ρ ( Ω ) , the corresponding function h h h in S ρ ( Ω ) S_\rho(\Omega) S ρ ( Ω ) is Lipschitz with respect to u − u ∗ u-u^* u − u ∗ and Δ u − Δ u ∗ \Delta u-\Delta u^* Δ u − Δ u ∗ . Note that for any u 1 , u 2 ∈ L ρ ( Ω ) u_1,u_2\in L_\rho(\Omega) u 1 , u 2 ∈ L ρ ( Ω ) with corresponding functions h 1 , h 2 ∈ S ρ ( Ω ) h_1,h_2\in S_\rho(\Omega) h 1 , h 2 ∈ S ρ ( Ω ) , applying boundedness condition 4.12 yields:
∣ h 1 ( x ) − h 2 ( x ) ∣ ≤ ∣ Δ u 1 − Δ u 2 − V ( u 1 − u 2 ) ∣ ∣ Δ u 1 − V u 1 + Δ u 2 − V u 2 + 2 f ∣ ≤ ( 2 C 2 + 4 C ) ( ∣ Δ u 1 − Δ u 2 ∣ + C ∣ u 1 − u 2 ∣ ) = ( 2 C 2 + 4 C ) ∣ ( Δ u 1 ( x ) − Δ u ∗ ( x ) ) − ( Δ u 2 ( x ) − Δ u ∗ ( x ) ) ∣ + ( 2 C 3 + 4 C 2 ) ∣ ( u 1 ( x ) − u ∗ ( x ) ) − ( u 2 ( x ) − u ∗ ( x ) ) ∣ . \begin{aligned}
|h_1(x)-h_2(x)|&\le|\Delta u_1-\Delta u_2-V(u_1-u_2)||\Delta u_1-Vu_1+\Delta u_2-Vu_2+2f|\\
&\le(2C^2+4C)(|\Delta u_1-\Delta u_2|+C|u_1-u_2|)\\
&=(2C^2+4C)\left|(\Delta u_1(x)-\Delta u^*(x))-(\Delta u_2(x)-\Delta u^*(x))\right|\\
&\quad+(2C^3+4C^2)\left|(u_1(x)-u^*(x))-(u_2(x)-u^*(x))\right|.
\end{aligned} ∣ h 1 ( x ) − h 2 ( x ) ∣ ≤ ∣Δ u 1 − Δ u 2 − V ( u 1 − u 2 ) ∣∣Δ u 1 − V u 1 + Δ u 2 − V u 2 + 2 f ∣ ≤ ( 2 C 2 + 4 C ) ( ∣Δ u 1 − Δ u 2 ∣ + C ∣ u 1 − u 2 ∣ ) = ( 2 C 2 + 4 C ) ∣ ( Δ u 1 ( x ) − Δ u ∗ ( x )) − ( Δ u 2 ( x ) − Δ u ∗ ( x )) ∣ + ( 2 C 3 + 4 C 2 ) ∣ ( u 1 ( x ) − u ∗ ( x )) − ( u 2 ( x ) − u ∗ ( x )) ∣ .
Let’s pick L = max { 2 C 2 + 4 C , 2 C 3 + 4 C 2 } L=\max\{2C^2+4C,2C^3+4C^2\} L = max { 2 C 2 + 4 C , 2 C 3 + 4 C 2 } . Applying the Talagrand Contraction Lemma A.4 helps us upper bound the local Rademacher complexity R n ( T ρ ( Ω ) ) R_n(\mathcal{T}_{\rho}(\Omega)) R n ( T ρ ( Ω )) by
R n ( T ρ ( Ω ) ) = E x E σ [ sup u ∈ M ρ ( Ω ) 1 n ∑ i = 1 n σ i [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] ] ≤ 2 L E x E σ [ sup u ∈ M ρ ( Ω ) 1 n ∑ i = 1 n σ i ( u ( x i ) − u ∗ ( x i ) ) ] + 2 L E x ′ , σ ′ [ sup u ∈ M ρ ( Ω ) 1 n ∑ i = 1 n σ i ′ ( Δ u ( x i ′ ) − Δ u ∗ ( x i ′ ) ) ] ≲ R n ( { u − u ∗ : u ∈ M ρ } ) + R n ( { Δ u − Δ u ∗ : u ∈ M ρ } ) ≲ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) + R n ( { Δ u − Δ u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ Δ u − Δ u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) ≤ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ n , 2 ≤ 2 ρ } ) + R n ( { Δ u − Δ u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ Δ u − Δ u ∗ ∥ n , 2 ≤ 2 ρ } ) ≲ inf 0 < α < 2 ρ { 4 α + 12 n ∫ α 2 ρ log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ n ) d δ } + inf 0 < α < 2 ρ { 4 α + 12 n ∫ α 2 ρ log N ( δ , Δ Φ ( L , W , S , B ) , ∥ ⋅ ∥ n , 2 ) d δ } ≲ inf 0 < α < 2 ρ { 4 α + 12 n ∫ α 2 ρ log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) d δ } + inf 0 < α < 2 ρ { 4 α + 12 n ∫ α 2 ρ log N ( δ , Δ Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) d δ } . \begin{aligned}
R_n(\mathcal{T}_{\rho}(\Omega)) &= \mathbb{E}_{\mathbf{x}}\mathbb{E}_{\boldsymbol{\sigma}}\left[\sup_{u\in\mathcal{M}_{\rho}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\left[(\Delta u-Vu+f)^2-(\Delta u^{*}-Vu^{*}+f)^2\right]\right]\\
&\le2L\mathbb{E}_{\mathbf{x}}\mathbb{E}_{\boldsymbol{\sigma}}\left[\sup_{u\in\mathcal{M}_{\rho}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i\left(u(x_i)-u^{*}(x_i)\right)\right]\\
&\quad+2L\mathbb{E}_{\mathbf{x}',\boldsymbol{\sigma}'}\left[\sup_{u\in\mathcal{M}_{\rho}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\sigma_i'\left(\Delta u(x_i')-\Delta u^{*}(x_i')\right)\right]\\
&\lesssim R_n\left(\left\{u-u^{*}:u\in\mathcal{M}_{\rho}\right\}\right)+R_n\left(\left\{\Delta u-\Delta u^{*}:u\in\mathcal{M}_{\rho}\right\}\right)\\
&\lesssim R_n\left(\left\{u-u^{*}:u\in\Phi(L,W,S,B),\ \|u-u^{*}\|_{L^2(\Omega)}\le\sqrt{\rho}\right\}\right)\\
&\quad+R_n\left(\left\{\Delta u-\Delta u^{*}:u\in\Phi(L,W,S,B),\ \|\Delta u-\Delta u^{*}\|_{L^2(\Omega)}\le\sqrt{\rho}\right\}\right)\\
&\le R_n\left(\left\{u-u^{*}:u\in\Phi(L,W,S,B),\ \|u-u^{*}\|_{n,2}\le2\sqrt{\rho}\right\}\right)\\
&\quad+R_n\left(\left\{\Delta u-\Delta u^{*}:u\in\Phi(L,W,S,B),\ \|\Delta u-\Delta u^{*}\|_{n,2}\le2\sqrt{\rho}\right\}\right)\\
&\lesssim\inf_{0<\alpha<2\sqrt{\rho}}\left\{4\alpha+\frac{12}{\sqrt{n}}\int_{\alpha}^{2\sqrt{\rho}}\sqrt{\log\mathcal{N}(\delta,\Phi(L,W,S,B),\|\cdot\|_n)}\,d\delta\right\}\\
&\quad+\inf_{0<\alpha<2\sqrt{\rho}}\left\{4\alpha+\frac{12}{\sqrt{n}}\int_{\alpha}^{2\sqrt{\rho}}\sqrt{\log\mathcal{N}(\delta,\Delta\Phi(L,W,S,B),\|\cdot\|_{n,2})}\,d\delta\right\}\\
&\lesssim\inf_{0<\alpha<2\sqrt{\rho}}\left\{4\alpha+\frac{12}{\sqrt{n}}\int_{\alpha}^{2\sqrt{\rho}}\sqrt{\log\mathcal{N}(\delta,\Phi(L,W,S,B),\|\cdot\|_{\infty})}\,d\delta\right\}\\
&\quad+\inf_{0<\alpha<2\sqrt{\rho}}\left\{4\alpha+\frac{12}{\sqrt{n}}\int_{\alpha}^{2\sqrt{\rho}}\sqrt{\log\mathcal{N}(\delta,\Delta\Phi(L,W,S,B),\|\cdot\|_{\infty})}\,d\delta\right\}.
\end{aligned} R n ( T ρ ( Ω )) = E x E σ [ u ∈ M ρ ( Ω ) sup n 1 i = 1 ∑ n σ i [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] ] ≤ 2 L E x E σ [ u ∈ M ρ ( Ω ) sup n 1 i = 1 ∑ n σ i ( u ( x i ) − u ∗ ( x i ) ) ] + 2 L E x ′ , σ ′ [ u ∈ M ρ ( Ω ) sup n 1 i = 1 ∑ n σ i ′ ( Δ u ( x i ′ ) − Δ u ∗ ( x i ′ ) ) ] ≲ R n ( { u − u ∗ : u ∈ M ρ } ) + R n ( { Δ u − Δ u ∗ : u ∈ M ρ } ) ≲ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) + R n ( { Δ u − Δ u ∗ : u ∈ Φ ( L , W , S , B ) , ∥Δ u − Δ u ∗ ∥ L 2 ( Ω ) ≤ ρ } ) ≤ R n ( { u − u ∗ : u ∈ Φ ( L , W , S , B ) , ∥ u − u ∗ ∥ n , 2 ≤ 2 ρ } ) + R n ( { Δ u − Δ u ∗ : u ∈ Φ ( L , W , S , B ) , ∥Δ u − Δ u ∗ ∥ n , 2 ≤ 2 ρ } ) ≲ 0 < α < 2 ρ inf { 4 α + n 12 ∫ α 2 ρ log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ n ) d δ } + 0 < α < 2 ρ inf { 4 α + n 12 ∫ α 2 ρ log N ( δ , ΔΦ ( L , W , S , B ) , ∥ ⋅ ∥ n , 2 ) d δ } ≲ 0 < α < 2 ρ inf { 4 α + n 12 ∫ α 2 ρ log N ( δ , Φ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) d δ } + 0 < α < 2 ρ inf { 4 α + n 12 ∫ α 2 ρ log N ( δ , ΔΦ ( L , W , S , B ) , ∥ ⋅ ∥ ∞ ) d δ } .
For any ρ ≳ 1 n 2 \rho\gtrsim\frac{1}{n^2} ρ ≳ n 2 1 , we pick α = 1 n ≲ ρ \alpha=\frac{1}{n}\lesssim\sqrt{\rho} α = n 1 ≲ ρ and plug in the upper bounds proved in Theorem A.18 and Theorem A.22, which implies:
R n ( T ρ ( Ω ) ) ≲ 1 n + 1 n ∫ 1 n 2 ρ S [ log ( δ − 1 ) + 3 L log ( W B ) ] d δ + 1 n ∫ 1 n 2 ρ S [ log ( δ − 1 ) + 3 L log ( W B ) ] d δ ≲ S 3 L ρ n log ( B W n ) . \begin{aligned}
R_n(\mathcal{T}_{\rho}(\Omega))&\lesssim\frac{1}{n}+\frac{1}{\sqrt{n}}\int_{\frac{1}{n}}^{2\sqrt{\rho}}\sqrt{S\left[\log(\delta^{-1})+3^L\log(WB)\right]}\,d\delta\\
&\quad+\frac{1}{\sqrt{n}}\int_{\frac{1}{n}}^{2\sqrt{\rho}}\sqrt{S\left[\log(\delta^{-1})+3^L\log(WB)\right]}\,d\delta\\
&\lesssim\sqrt{\frac{S3^L\rho}{n}\log(BWn)}.
\end{aligned} R n ( T ρ ( Ω )) ≲ n 1 + n 1 ∫ n 1 2 ρ S [ log ( δ − 1 ) + 3 L log ( W B ) ] d δ + n 1 ∫ n 1 2 ρ S [ log ( δ − 1 ) + 3 L log ( W B ) ] d δ ≲ n S 3 L ρ log ( B W n ) .
Proof of The Meta-Theorem for PINN
Proof. To upper bound the excess risk Δ E ( n ) \Delta\mathcal{E}^{(n)} Δ E ( n ) , following [78 , 47 , 14 ] , we decompose the excess risk into approximation error and generalization error with probability 1 − e − t 1-e^{-t} 1 − e − t .
Δ E ( n ) ( u ^ P I N N ) = E ( u ^ P I N N ) − E ( u ∗ ) = [ E ( u ^ P I N N ) − E n ( u ^ P I N N ) ] + [ E n ( u ^ P I N N ) − E n ( u F ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u ^ P I N N ) − E n ( u ^ P I N N ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( E ^ ( u ^ P I N N ) − E ( u ∗ ) ) + E n ( u ∗ ) − E n ( u ^ P I N N ) ] + 3 2 [ E ( u F ) − E ( u ∗ ) ] + t 2 n . (89) \Delta\mathcal{E}^{(n)}(\hat{u}_{\mathrm{PINN}})=\mathcal{E}(\hat{u}_{\mathrm{PINN}})-\mathcal{E}(u^{*})=\left[\mathcal{E}(\hat{u}_{\mathrm{PINN}})-\mathcal{E}_{n}(\hat{u}_{\mathrm{PINN}})\right]+\left[\mathcal{E}_{n}(\hat{u}_{\mathrm{PINN}})-\mathcal{E}_{n}(u_{\mathcal{F}})\right]
+\left[\mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}(u_{\mathcal{F}})\right]+\left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})\right]
\leq\left[\mathcal{E}(\hat{u}_{\mathrm{PINN}})-\mathcal{E}_{n}(\hat{u}_{\mathrm{PINN}})\right]+\left[\mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}(u_{\mathcal{F}})\right]+\left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})\right]
\leq\left[\mathbb{E}(\hat{\mathcal{E}}(\hat{u}_{\mathrm{PINN}})-\mathcal{E}(u^{*}))+\mathcal{E}_{n}(u^{*})-\mathcal{E}_{n}(\hat{u}_{\mathrm{PINN}})\right]
+\frac{3}{2}\left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})\right]+\frac{t}{2n}. \tag*{(89)} Δ E ( n ) ( u ^ PINN ) = E ( u ^ PINN ) − E ( u ∗ ) = [ E ( u ^ PINN ) − E n ( u ^ PINN ) ] + [ E n ( u ^ PINN ) − E n ( u F ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u ^ PINN ) − E n ( u ^ PINN ) ] + [ E n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( E ^ ( u ^ PINN ) − E ( u ∗ )) + E n ( u ∗ ) − E n ( u ^ PINN ) ] + 2 3 [ E ( u F ) − E ( u ∗ ) ] + 2 n t . (89)
where the expectation is on all sampled data. The inequality of the third line is because the u u u is the minimizer of the empirical loss E n \mathcal{E}_{n} E n in the solution set F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , so we have E n ( u ) ≤ E n ( u F ) \mathcal{E}_{n}(u)\leq\mathcal{E}_{n}(u_{\mathcal{F}}) E n ( u ) ≤ E n ( u F ) . The last inequality is based on the Bernstein inequality. The variance of h = ∣ Ω ∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] h=|\Omega|\cdot\left[(\Delta u-\mathcal{V}u+f)^{2}-(\Delta u^{*}-\mathcal{V}u^{*}+f)^{2}\right] h = ∣Ω∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] can be bounded by [ E ( u F ) − E ( u ∗ ) ] \left[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})\right] [ E ( u F ) − E ( u ∗ ) ] due to the strong convexity of the variation objective (A.51). According to the Brenstein inequality, we know with probability 1 − e − t 1-e^{-t} 1 − e − t we have
E n ( u F ) − E n ( u ∗ ) − E ( u F ) + E ( u ∗ ) ≤ t [ E ( u F ) − E ( u ∗ ) ] n ≤ 1 2 [ E ( u F ) − E ( u ∗ ) ] + t 2 n . \mathcal{E}_{n}(u_{\mathcal{F}})-\mathcal{E}_{n}(u^{*})-\mathcal{E}(u_{\mathcal{F}})+\mathcal{E}(u^{*})\leq\sqrt{\frac{t[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})]}{n}}\leq\frac{1}{2}[\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})]+\frac{t}{2n}. E n ( u F ) − E n ( u ∗ ) − E ( u F ) + E ( u ∗ ) ≤ n t [ E ( u F ) − E ( u ∗ )] ≤ 2 1 [ E ( u F ) − E ( u ∗ )] + 2 n t .
Note that A.53 holds for all function lies in the function space F \mathcal{F} F . Thus, we can take u F : = arg min u 0 ∈ F ( Ω ) ( E ( u 0 ) − E ( u ∗ ) ) u_{\mathcal{F}}:=\operatorname*{arg\,min}_{u_{0}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{0})-\mathcal{E}(u^{*})\right) u F := arg min u 0 ∈ F ( Ω ) ( E ( u 0 ) − E ( u ∗ ) ) and finally get
Δ E ( n ) ≤ E ( u ^ P I N N ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ P I N N ) ⏟ Δ E g e n + 3 2 inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) ⏟ Δ E a p p + t 2 n . \Delta\mathcal{E}^{(n)}\leq\underbrace{\mathcal{E}(\hat{u}_{\mathrm{PINN}})-\mathcal{E}(u^{*})+\mathcal{E}_{n}(u^{*})-\mathcal{E}_{n}(\hat{u}_{\mathrm{PINN}})}_{\Delta\mathcal{E}_{\mathrm{gen}}}+\frac{3}{2}\underbrace{\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})\right)}_{\Delta\mathcal{E}_{\mathrm{app}}}+\frac{t}{2n}. Δ E ( n ) ≤ Δ E gen E ( u ^ PINN ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ PINN ) + 2 3 Δ E app u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + 2 n t .
This inequality decompose the excess risk to the generalization error Δ E g e n : = E ( u ^ P I N N ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ P I N N ) \Delta\mathcal{E}_{\mathrm{gen}}:=\mathcal{E}(\hat{u}_{\mathrm{PINN}})-\mathcal{E}(u^{*})+\mathcal{E}_{n}(u^{*})-\mathcal{E}_{n}(\hat{u}_{\mathrm{PINN}}) Δ E gen := E ( u ^ PINN ) − E ( u ∗ ) + E n ( u ∗ ) − E n ( u ^ PINN ) and the approximation error Δ E a p p = inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) \Delta\mathcal{E}_{\mathrm{app}}=\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^{*})\right) Δ E app = inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) . We’ll focus on providing fast rate upper bounds of the generalization error for the two estimators using the localization technique[3 , 78 ] . To achieve the fast generalization bound, we focus on the following normalized empirical process
T ~ r ( Ω ) : = { h ~ ( x ) : = E [ h ] − h ( x ) E [ h ] + r ∣ h ∈ T ( Ω ) } ( r > 0 ) . \widetilde{\mathcal{T}}_{r}(\Omega):=\left\{\widetilde{h}(x):=\frac{\mathbb{E}[h]-h(x)}{\mathbb{E}[h]+r}\mid h\in\mathcal{T}(\Omega)\right\}\ (r>0). T r ( Ω ) := { h ( x ) := E [ h ] + r E [ h ] − h ( x ) ∣ h ∈ T ( Ω ) } ( r > 0 ) .
First, we try to bound the expectation of the normalized empirical process. Applying the Symmetrization Lemma A.3, we can first bound the expectation as
sup h ~ ∈ T ~ r ( Ω ) E x ′ [ 1 n ∑ i = 1 n h ~ ( x i ′ ) ] ≤ E x ′ [ sup h ∈ T ( Ω ) 1 n ∑ i = 1 n h ( x i ′ ) − E [ h ] E [ h ] + r ] ≤ 2 R n ( T ^ r ( Ω ) ) . \sup_{\widetilde{h}\in\widetilde{\mathcal{T}}_{r}(\Omega)}\mathbb{E}_{x'}\left[\frac{1}{n}\sum_{i=1}^{n}\widetilde{h}(x_{i}')\right]\leq\mathbb{E}_{x'}\left[\sup_{h\in\mathcal{T}(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\frac{h(x_{i}')-\mathbb{E}[h]}{\mathbb{E}[h]+r}\right]\leq2R_{n}(\widehat{\mathcal{T}}_{r}(\Omega)). h ∈ T r ( Ω ) sup E x ′ [ n 1 i = 1 ∑ n h ( x i ′ ) ] ≤ E x ′ [ h ∈ T ( Ω ) sup n 1 i = 1 ∑ n E [ h ] + r h ( x i ′ ) − E [ h ] ] ≤ 2 R n ( T r ( Ω )) .
where the function class S ^ r ( Ω ) \widehat{\mathcal{S}}_{r}(\Omega) S r ( Ω ) is defined as:
T ^ r ( Ω ) : = { h ^ ( x ) : = h ( x ) E [ h ] + r ∣ h ∈ T ( Ω ) } , \widehat{\mathcal{T}}_{r}(\Omega):=\left\{\widehat{h}(x):=\frac{h(x)}{\mathbb{E}[h]+r}\mid h\in\mathcal{T}(\Omega)\right\}, T r ( Ω ) := { h ( x ) := E [ h ] + r h ( x ) ∣ h ∈ T ( Ω ) } ,
where T ( Ω ) = { h : = ∣ Ω ∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] } \mathcal{T}(\Omega)=\left\{h:=|\Omega|\cdot\left[(\Delta u-Vu+f)^2-(\Delta u^*-Vu^*+f)^2\right]\right\} T ( Ω ) = { h := ∣Ω∣ ⋅ [ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ] } . Then Applying the Peeling Lemma to any function h ∈ T ( Ω ) h\in\mathcal{T}(\Omega) h ∈ T ( Ω ) helps us upper bound the local Rademacher complexity R n ( T ^ r ( Ω ) ) R_n(\widehat{\mathcal{T}}_r(\Omega)) R n ( T r ( Ω )) with the function ϕ \phi ϕ defined in equation (35) :
R n ( T ^ r ( Ω ) ) = E σ [ E x [ sup h ∈ T ( Ω ) 1 n ∑ i = 1 n σ i h ( x i ) E [ h ] + r ] ] ≤ 4 ϕ ( r ) r . R_n(\widehat{\mathcal{T}}_r(\Omega))=\mathbb{E}_{\sigma}\left[\mathbb{E}_{x}\left[\sup_{h\in\mathcal{T}(\Omega)}\frac{\frac{1}{n}\sum_{i=1}^{n}\sigma_i h(x_i)}{\mathbb{E}[h]+r}\right]\right]\leq\frac{4\phi(r)}{r}. R n ( T r ( Ω )) = E σ [ E x [ h ∈ T ( Ω ) sup E [ h ] + r n 1 ∑ i = 1 n σ i h ( x i ) ] ] ≤ r 4 ϕ ( r ) .
Combining all inequalities derived above yields:
sup h ~ ∈ T ~ r ( Ω ) E x ′ [ 1 n ∑ i = 1 n h ~ ( x i ′ ) ] ≤ 2 R n ( T ^ r ( Ω ) ) ≤ 8 ϕ ( r ) r ( r > 0 ) . (90) \sup_{\widetilde{h}\in\widetilde{\mathcal{T}}_r(\Omega)}\mathbb{E}_{x'}\left[\frac{1}{n}\sum_{i=1}^{n}\widetilde{h}(x_i')\right]\leq2R_n(\widehat{\mathcal{T}}_r(\Omega))\leq\frac{8\phi(r)}{r}\quad(r>0). \tag*{(90)} h ∈ T r ( Ω ) sup E x ′ [ n 1 i = 1 ∑ n h ( x i ′ ) ] ≤ 2 R n ( T r ( Ω )) ≤ r 8 ϕ ( r ) ( r > 0 ) . (90)
Secondly we’ll apply the Talagrand concentration inequality, which requires us to verify the condition needed. We will first check that the expectation value E [ h ] \mathbb{E}[h] E [ h ] is always non-negative for any h ∈ S ( Ω ) h\in\mathcal{S}(\Omega) h ∈ S ( Ω ) :
E [ h ] = 1 ∣ Ω ∣ ∫ Ω ∣ Ω ∣ ⋅ ( 1 2 ∣ ∇ u ( x ) ∣ 2 + 1 2 V ( x ) ∣ u ( x ) ∣ 2 − f ( x ) u ( x ) ) d x − 1 ∣ Ω ∣ ∫ Ω ∣ Ω ∣ ⋅ ( 1 2 ∣ ∇ u ∗ ( x ) ∣ 2 + 1 2 V ( x ) ∣ u ∗ ( x ) ∣ 2 − f ( x ) u ∗ ( x ) ) d x = E ( u ) − E ( u ∗ ) ≥ 0 ⇒ E [ h ] ≥ 0. \begin{aligned}
\mathbb{E}[h]&=\frac{1}{|\Omega|}\int_{\Omega}|\Omega|\cdot\left(\frac{1}{2}|\nabla u(x)|^2+\frac{1}{2}V(x)|u(x)|^2-f(x)u(x)\right)\,dx\\
&\quad-\frac{1}{|\Omega|}\int_{\Omega}|\Omega|\cdot\left(\frac{1}{2}|\nabla u^*(x)|^2+\frac{1}{2}V(x)|u^*(x)|^2-f(x)u^*(x)\right)\,dx\\
&=\mathcal{E}(u)-\mathcal{E}(u^*)\geq0\Rightarrow\mathbb{E}[h]\geq0.
\end{aligned} E [ h ] = ∣Ω∣ 1 ∫ Ω ∣Ω∣ ⋅ ( 2 1 ∣∇ u ( x ) ∣ 2 + 2 1 V ( x ) ∣ u ( x ) ∣ 2 − f ( x ) u ( x ) ) d x − ∣Ω∣ 1 ∫ Ω ∣Ω∣ ⋅ ( 2 1 ∣∇ u ∗ ( x ) ∣ 2 + 2 1 V ( x ) ∣ u ∗ ( x ) ∣ 2 − f ( x ) u ∗ ( x ) ) d x = E ( u ) − E ( u ∗ ) ≥ 0 ⇒ E [ h ] ≥ 0.
We will proceed to verify that any h ~ = E [ h ] − h E [ h ] + r ∈ T ~ r ( Ω ) \widetilde{h}=\frac{\mathbb{E}[h]-h}{\mathbb{E}[h]+r}\in\widetilde{\mathcal{T}}_r(\Omega) h = E [ h ] + r E [ h ] − h ∈ T r ( Ω ) is of bounded inf-norm. We need to prove that any h ∈ T ( Ω ) h\in\mathcal{T}(\Omega) h ∈ T ( Ω ) is of bounded inf-norm beforehand. Using boundedness condition listed in equation (34) implies:
∥ h ∥ ∞ = ∣ Ω ∣ ⋅ ∥ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ∥ ∞ = ∣ Ω ∣ ⋅ ∥ ( Δ u − V u + f ) 2 ∥ ∞ ≤ ∣ Ω ∣ ⋅ ( ∥ Δ u ∥ ∞ + V max ∥ u ∥ ∞ + ∥ f ∥ ∞ ) 2 ≤ ∣ Ω ∣ ( V max + 2 ) 2 C 2 \begin{aligned}
\|h\|_{\infty}&=|\Omega|\cdot\left\|(\Delta u-Vu+f)^2-(\Delta u^*-Vu^*+f)^2\right\|_{\infty}=|\Omega|\cdot\left\|(\Delta u-Vu+f)^2\right\|_{\infty}\\
&\leq|\Omega|\cdot\left(\|\Delta u\|_{\infty}+V_{\max}\|u\|_{\infty}+\|f\|_{\infty}\right)^2\leq|\Omega|(V_{\max}+2)^2C^2
\end{aligned} ∥ h ∥ ∞ = ∣Ω∣ ⋅ ( Δ u − V u + f ) 2 − ( Δ u ∗ − V u ∗ + f ) 2 ∞ = ∣Ω∣ ⋅ ( Δ u − V u + f ) 2 ∞ ≤ ∣Ω∣ ⋅ ( ∥Δ u ∥ ∞ + V m a x ∥ u ∥ ∞ + ∥ f ∥ ∞ ) 2 ≤ ∣Ω∣ ( V m a x + 2 ) 2 C 2
By taking M : = ∣ Ω ∣ ( V max + 2 ) 2 C 2 M:=|\Omega|(V_{\max}+2)^2C^2 M := ∣Ω∣ ( V m a x + 2 ) 2 C 2 , we then have ∥ h ∥ ∞ ≤ M \|h\|_{\infty}\leq M ∥ h ∥ ∞ ≤ M for all h ∈ T ( Ω ) h\in\mathcal{T}(\Omega) h ∈ T ( Ω ) . Note that the denominator can be lower bounded by ∣ E [ h ] + r ∣ ≥ r > 0 |\mathbb{E}[h]+r|\geq r>0 ∣ E [ h ] + r ∣ ≥ r > 0 . Combining these two inequalities help us upper bound the inf-norm ∥ h ~ ∥ ∞ = sup x ∈ Ω ∣ h ~ ( x ) ∣ \|\widetilde{h}\|_{\infty}=\sup_{x\in\Omega}|\widetilde{h}(x)| ∥ h ∥ ∞ = sup x ∈ Ω ∣ h ( x ) ∣ as follows:
∥ h ~ ∥ ∞ = ∥ E [ h ] − h ∥ ∞ ∣ E [ h ] + r ∣ ≤ 2 ∥ h ∥ ∞ r ≤ 2 M r = : β . \|\widetilde{h}\|_{\infty}=\frac{\|\mathbb{E}[h]-h\|_{\infty}}{|\mathbb{E}[h]+r|}\leq\frac{2\|h\|_{\infty}}{r}\leq\frac{2M}{r}=:\beta. ∥ h ∥ ∞ = ∣ E [ h ] + r ∣ ∥ E [ h ] − h ∥ ∞ ≤ r 2∥ h ∥ ∞ ≤ r 2 M =: β .
We will then check the normalized functions E [ h ] − h ( x ) E [ h ] + r \frac{\mathbb{E}[h]-h(x)}{\mathbb{E}[h]+r} E [ h ] + r E [ h ] − h ( x ) in T ~ r ( Ω ) \widetilde{\mathcal{T}}_r(\Omega) T r ( Ω ) have bounded second moment, which is satisfied because of the regularity results of the PDE. We aim to show that there exist some constants α , α ′ > 0 \alpha,\alpha'>0 α , α ′ > 0 , such that for any h ∈ T ( Ω ) h\in\mathcal{T}(\Omega) h ∈ T ( Ω ) , the following inequality holds:
α E [ h ] 2 ≤ ∥ u − u ∗ ∥ H 2 ( Ω ) 2 ≤ α ′ E [ h ] . (91) \alpha\mathbb{E}[h]^2\leq\|u-u^*\|_{H^2(\Omega)}^2\leq\alpha'\mathbb{E}[h]. \tag*{(91)} α E [ h ] 2 ≤ ∥ u − u ∗ ∥ H 2 ( Ω ) 2 ≤ α ′ E [ h ] . (91)
The RHS of the inequality follows from strong convexity of the PINN objective function proved in Theorem A.2:
E [ h ] = E ( u ) − E ( u ∗ ) ≥ 1 min { 1 , C min } ∥ u − u ∗ ∥ H 2 ( Ω ) 2 \mathbb{E}[h]=\mathcal{E}(u)-\mathcal{E}(u^*)\geq\frac{1}{\min\{1,C_{\min}\}}\|u-u^*\|_{H^2(\Omega)}^2 E [ h ] = E ( u ) − E ( u ∗ ) ≥ min { 1 , C m i n } 1 ∥ u − u ∗ ∥ H 2 ( Ω ) 2
The LHS of the inequality follows from boundedness condition listed in equation (34) and the QM-AM inequality:
E [ h 2 ] = ∫ Ω [ ( Δ u − ∇ u + f ) 2 − ( Δ u ∗ − ∇ u ∗ + f ) 2 ] 2 d x = ∫ Ω ( Δ u − ∇ u + f ) 4 d x ≤ M 2 ∫ Ω ( Δ u − ∇ u − Δ u ∗ + ∇ u ∗ ) 2 d x ≤ 2 M 2 ∫ Ω [ ( Δ u − Δ u ∗ ) 2 + V 2 ( u − u ∗ ) 2 ] d x \begin{aligned}
\mathbb{E}[h^2] = \int_{\Omega} [(\Delta u - \nabla u + f)^2 - (\Delta u^* - \nabla u^* + f)^2]^2\,dx = \int_{\Omega}(\Delta u-\nabla u+f)^4dx \\
&\le M^2\int_{\Omega}(\Delta u-\nabla u-\Delta u^*+\nabla u^*)^2dx\le2M^2\int_{\Omega}[(\Delta u-\Delta u^*)^2+V^2(u-u^*)^2]dx
\end{aligned} E [ h 2 ] = ∫ Ω [( Δ u − ∇ u + f ) 2 − ( Δ u ∗ − ∇ u ∗ + f ) 2 ] 2 d x = ∫ Ω ( Δ u − ∇ u + f ) 4 d x ≤ M 2 ∫ Ω ( Δ u − ∇ u − Δ u ∗ + ∇ u ∗ ) 2 d x ≤ 2 M 2 ∫ Ω [( Δ u − Δ u ∗ ) 2 + V 2 ( u − u ∗ ) 2 ] d x
≤ 2 M 2 max { 1 , V max 2 } ∥ u − u ∗ ∥ H 2 ( Ω ) 2 \le2M^2\max\{1,V_{\max}^2\}\|u-u^*\|_{H^2(\Omega)}^2 ≤ 2 M 2 max { 1 , V m a x 2 } ∥ u − u ∗ ∥ H 2 ( Ω ) 2
By picking α ′ = 1 min { 1 , C min } \alpha'=\frac{1}{\min\{1,C_{\min}\}} α ′ = m i n { 1 , C m i n } 1 and α = 1 M 2 max { 1 , V max 2 } \alpha=\frac{1}{M^2\max\{1,V_{\max}^2\}} α = M 2 m a x { 1 , V m a x 2 } 1 , we have finished proving inequality A.51. Then we can can upper bound the expectation E [ h ~ 2 ] \mathbb{E}[\tilde{h}^2] E [ h ~ 2 ] as:
E [ h ~ 2 ] = E [ ( h − E [ h ] ) 2 ] ∣ E [ h ] + r ∣ 2 = E [ h 2 ] − E [ h ] 2 ∣ E [ h ] + r ∣ 2 ≤ E [ h 2 ] ∣ E [ h ] + r ∣ 2 . \mathbb{E}[\tilde{h}^2]=\frac{\mathbb{E}[(h-\mathbb{E}[h])^2]}{|\mathbb{E}[h]+r|^2}=\frac{\mathbb{E}[h^2]-\mathbb{E}[h]^2}{|\mathbb{E}[h]+r|^2}\le\frac{\mathbb{E}[h^2]}{|\mathbb{E}[h]+r|^2}. E [ h ~ 2 ] = ∣ E [ h ] + r ∣ 2 E [( h − E [ h ] ) 2 ] = ∣ E [ h ] + r ∣ 2 E [ h 2 ] − E [ h ] 2 ≤ ∣ E [ h ] + r ∣ 2 E [ h 2 ] .
Using the fact that E [ h ] ≥ 0 \mathbb{E}[h]\ge0 E [ h ] ≥ 0 and inequality A.51, we can lower bound the denominator ∣ E [ h ] + r ∣ 2 |\mathbb{E}[h]+r|^2 ∣ E [ h ] + r ∣ 2 as follows:
∣ E [ h ] + r ∣ 2 ≥ 2 E [ h ] r ≥ 2 r α α ′ E [ h 2 ] . |\mathbb{E}[h]+r|^2\ge2\mathbb{E}[h]r\ge\frac{2r\alpha}{\alpha'}\mathbb{E}[h^2]. ∣ E [ h ] + r ∣ 2 ≥ 2 E [ h ] r ≥ α ′ 2 r α E [ h 2 ] .
Therefore, we can deduce that:
E [ h ~ 2 ] ≤ E [ h 2 ] ∣ E [ h ] + r ∣ 2 ≤ E [ h 2 ] 2 r α α ′ E [ h 2 ] = α ′ 2 r α = : σ 2 . \mathbb{E}[\tilde{h}^2]\le\frac{\mathbb{E}[h^2]}{|\mathbb{E}[h]+r|^2}\le\frac{\mathbb{E}[h^2]}{\frac{2r\alpha}{\alpha'}\mathbb{E}[h^2]}=\frac{\alpha'}{2r\alpha}=:\sigma^2. E [ h ~ 2 ] ≤ ∣ E [ h ] + r ∣ 2 E [ h 2 ] ≤ α ′ 2 r α E [ h 2 ] E [ h 2 ] = 2 r α α ′ =: σ 2 .
Hence, any function in the localized class T ~ r ( Ω ) \tilde{T}_r(\Omega) T ~ r ( Ω ) is of bounded second moment.
It is easy to check that for any h ~ ∈ T ~ r ( Ω ) \tilde{h}\in\tilde{T}_r(\Omega) h ~ ∈ T ~ r ( Ω ) , we have
E [ h ~ ] = E [ h ] − E [ h ] ∣ E [ h ] + r ∣ = 0 , \mathbb{E}[\tilde{h}]=\frac{\mathbb{E}[h]-\mathbb{E}[h]}{|\mathbb{E}[h]+r|}=0, E [ h ~ ] = ∣ E [ h ] + r ∣ E [ h ] − E [ h ] = 0 ,
i.e. any function in the localized class S ~ r ( Ω ) \tilde{S}_r(\Omega) S ~ r ( Ω ) is of zero mean.
Now we have verified that any function h ~ ∈ S ~ r ( Ω ) \tilde{h}\in\tilde{S}_r(\Omega) h ~ ∈ S ~ r ( Ω ) satisfies all the required conditions. By taking μ \mu μ to be the uniform distribution on the domain Ω \Omega Ω and applying Talagrand’s Concentration inequality given in Lemma A.6, we have:
P x [ sup h ~ ∈ T ~ r ( Ω ) 1 n ∑ i = 1 n h ~ ( x i ) ≥ 2 sup h ~ ∈ T ~ r ( Ω ) E x ′ [ 1 n ∑ i = 1 n h ~ ( x i ′ ) ] + 2 t σ 2 n + 2 t β n ] ≤ e − t . \mathbb{P}_x\left[\sup_{\tilde{h}\in\tilde{T}_r(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x_i)\ge2\sup_{\tilde{h}\in\tilde{T}_r(\Omega)}\mathbb{E}_{x'}\left[\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x'_i)\right]+\sqrt{\frac{2t\sigma^2}{n}}+\frac{2t\beta}{n}\right]\le e^{-t}. P x [ h ~ ∈ T ~ r ( Ω ) sup n 1 i = 1 ∑ n h ~ ( x i ) ≥ 2 h ~ ∈ T ~ r ( Ω ) sup E x ′ [ n 1 i = 1 ∑ n h ~ ( x i ′ ) ] + n 2 t σ 2 + n 2 tβ ] ≤ e − t .
By using the upper bound deduced above and plugging in the expressions of β \beta β and σ \sigma σ , we can rewrite Talagrand’s Concentration Inequality in the following way. With probability at least 1 − e − t 1-e^{-t} 1 − e − t , the inequality below holds:
1 n ∑ i = 1 n h ~ ( x i ) ≤ sup h ~ ∈ S ~ r ( Ω ) 1 n ∑ i = 1 n h ~ ( x i ) ≤ 2 sup h ~ ∈ S ~ r ( Ω ) E x ′ [ 1 n ∑ i = 1 n h ~ ( x i ′ ) ] + 2 t σ 2 n + 2 t β n ≤ 16 ϕ ( r ) r + t α ′ n α r + 4 M t n r = : ψ ( r ) \begin{aligned}
\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x_i)\le\sup_{\tilde{h}\in\tilde{S}_r(\Omega)}\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x_i)\le2\sup_{\tilde{h}\in\tilde{S}_r(\Omega)}\mathbb{E}_{x'}\left[\frac{1}{n}\sum_{i=1}^{n}\tilde{h}(x'_i)\right]+\sqrt{\frac{2t\sigma^2}{n}}+\frac{2t\beta}{n} \\
&\le\frac{16\phi(r)}{r}+\sqrt{\frac{t\alpha'}{n\alpha r}}+\frac{4Mt}{nr}=: \psi(r)
\end{aligned} n 1 i = 1 ∑ n h ~ ( x i ) ≤ h ~ ∈ S ~ r ( Ω ) sup n 1 i = 1 ∑ n h ~ ( x i ) ≤ 2 h ~ ∈ S ~ r ( Ω ) sup E x ′ [ n 1 i = 1 ∑ n h ~ ( x i ′ ) ] + n 2 t σ 2 + n 2 tβ ≤ r 16 ϕ ( r ) + n α r t α ′ + n r 4 M t =: ψ ( r )
Let’s pick the critical radius r 0 r_0 r 0 to be:
r 0 = max { 2 14 r ∗ , 24 M t n , 36 α ′ t α n } . (92) r_0 = \max\left\{2^{14}r^*, \frac{24Mt}{n}, \frac{36\alpha't}{\alpha n}\right\}. \tag*{(92)} r 0 = max { 2 14 r ∗ , n 24 M t , α n 36 α ′ t } . (92)
Note that concavity of the function ϕ \phi ϕ implies that ϕ ( r ) ≤ r \phi(r) \le r ϕ ( r ) ≤ r for any r ≥ r ∗ r \ge r^* r ≥ r ∗ . Combining this with the first inequality listed in 4.26 yields:
16 ϕ ( r ) r ≤ 2 11 ϕ ( r 0 2 14 ) 2 14 r 0 2 14 = 1 8 × ϕ ( r 0 2 14 ) r 0 2 14 ≤ 1 8 . \frac{16\phi(r)}{r} \le\frac{2^{11}\phi\left(\frac{r_0}{2^{14}}\right)}{2^{14}\frac{r_0}{2^{14}}} = \frac{1}{8}\times\frac{\phi\left(\frac{r_0}{2^{14}}\right)}{\frac{r_0}{2^{14}}} \le\frac{1}{8}. r 16 ϕ ( r ) ≤ 2 14 2 14 r 0 2 11 ϕ ( 2 14 r 0 ) = 8 1 × 2 14 r 0 ϕ ( 2 14 r 0 ) ≤ 8 1 .
On the other hand, applying equation A.52 yields:
α ′ t n α r 0 ≤ α ′ t n α α n 36 α ′ t = 1 6 , 4 M t n r 0 ≤ 4 M t n × n 24 M t = 1 6 . \sqrt{\frac{\alpha't}{n\alpha r_0}} \le\sqrt{\frac{\alpha't}{n\alpha\frac{\alpha n}{36\alpha't}}} = \frac{1}{6},
\qquad
\frac{4Mt}{nr_0} \le\frac{4Mt}{n}\times\frac{n}{24Mt} = \frac{1}{6}. n α r 0 α ′ t ≤ n α 36 α ′ t α n α ′ t = 6 1 , n r 0 4 M t ≤ n 4 M t × 24 M t n = 6 1 .
Summing the three inequalities above implies:
ψ ( r 0 ) = 16 ϕ ( r 0 ) r 0 + t α ′ n α r 0 + 4 M t n r 0 ≤ 1 8 + 1 6 + 1 6 < 1 2 . \psi(r_0) = \frac{16\phi(r_0)}{r_0} + \sqrt{\frac{t\alpha'}{n\alpha r_0}} + \frac{4Mt}{nr_0} \le\frac{1}{8}+\frac{1}{6}+\frac{1}{6}<\frac{1}{2}. ψ ( r 0 ) = r 0 16 ϕ ( r 0 ) + n α r 0 t α ′ + n r 0 4 M t ≤ 8 1 + 6 1 + 6 1 < 2 1 .
By picking r = r 0 r=r_0 r = r 0 , we can further deduce that for any function u ∈ F ( Ω ) u\in\mathcal{F}(\Omega) u ∈ F ( Ω ) , the following inequality holds with probability 1 − e − t 1-e^{-t} 1 − e − t :
E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) E ( u ) − E ( u ∗ ) + r 0 = 1 n ∑ i = 1 n h ~ ( x i ) ≤ ψ ( r 0 ) < 1 2 . \frac{\mathcal{E}(u)-\mathcal{E}(u^*)-\mathcal{E}_n(u)+\mathcal{E}_n(u^*)}{\mathcal{E}(u)-\mathcal{E}(u^*)+r_0}
= \frac{1}{n}\sum_{i=1}^{n}\widetilde{h}(x_i)\le\psi(r_0)<\frac{1}{2}. E ( u ) − E ( u ∗ ) + r 0 E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) = n 1 i = 1 ∑ n h ( x i ) ≤ ψ ( r 0 ) < 2 1 .
Multiplying the denominator on both sides indicates:
Δ E g e n = E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) ≤ 1 2 [ E ( u ) − E ( u ∗ ) ] + 1 2 r 0 = 1 2 Δ E ( n ) + 1 2 r 0 . \Delta\mathcal{E}_{\mathrm{gen}}=\mathcal{E}(u)-\mathcal{E}(u^*)-\mathcal{E}_n(u)+\mathcal{E}_n(u^*)\le\frac{1}{2}\left[\mathcal{E}(u)-\mathcal{E}(u^*)\right]+\frac{1}{2}r_0=\frac{1}{2}\Delta\mathcal{E}^{(n)}+\frac{1}{2}r_0. Δ E gen = E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) ≤ 2 1 [ E ( u ) − E ( u ∗ ) ] + 2 1 r 0 = 2 1 Δ E ( n ) + 2 1 r 0 .
Substituting the upper bound above into the decomposition Δ E ( n ) ≤ Δ E g e n + 3 2 Δ E a p p + t 2 n \Delta\mathcal{E}^{(n)}\le\Delta\mathcal{E}_{\mathrm{gen}}+\frac{3}{2}\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{2n} Δ E ( n ) ≤ Δ E gen + 2 3 Δ E app + 2 n t yields that with probability 1 − e − t 1-e^{-t} 1 − e − t , we have:
Δ E ( n ) ≤ Δ E g e n + 3 2 Δ E a p p + t 2 n ≤ 1 2 Δ E ( n ) + 1 2 r 0 + 3 2 Δ E a p p + t 2 n . \Delta\mathcal{E}^{(n)}\le\Delta\mathcal{E}_{\mathrm{gen}}+\frac{3}{2}\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{2n}\le\frac{1}{2}\Delta\mathcal{E}^{(n)}+\frac{1}{2}r_0+\frac{3}{2}\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{2n}. Δ E ( n ) ≤ Δ E gen + 2 3 Δ E app + 2 n t ≤ 2 1 Δ E ( n ) + 2 1 r 0 + 2 3 Δ E app + 2 n t .
Simplifying the inequality above yields that with probability 1 − e − t 1-e^{-t} 1 − e − t , we have:
Δ E ( n ) ≤ r 0 + 3 Δ E a p p + t n = 3 inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { 2 14 r ∗ , 24 M t n , 36 α ′ t α n } + t n \Delta\mathcal{E}^{(n)}\le r_0+3\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{n}
=3\inf_{u_F\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_F)-\mathcal{E}(u^*)\right)+\max\left\{2^{14}r^*,\frac{24Mt}{n},\frac{36\alpha't}{\alpha n}\right\}+\frac{t}{n} Δ E ( n ) ≤ r 0 + 3Δ E app + n t = 3 u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { 2 14 r ∗ , n 24 M t , α n 36 α ′ t } + n t
≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \lesssim\inf_{u_F\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_F)-\mathcal{E}(u^*)\right)+\max\left\{r^*,\frac{t}{n}\right\}. ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Moreover, using strong convexity of the PINN objective function proved in Theorem A.1 implies:
Δ E ( n ) = E ( u ^ P I N N ) − E ( u ∗ ) ≥ 1 { 1 , C min } ∥ u ^ P I N N − u ∗ ∥ H 1 ( Ω ) 2 . \Delta\mathcal{E}^{(n)}=\mathcal{E}(\hat{u}_{\mathrm{PINN}})-\mathcal{E}(u^*)\ge\frac{1}{\{1,C_{\min}\}}\left\|\hat{u}_{\mathrm{PINN}}-u^*\right\|_{H^1(\Omega)}^2. Δ E ( n ) = E ( u ^ PINN ) − E ( u ∗ ) ≥ { 1 , C m i n } 1 ∥ u ^ PINN − u ∗ ∥ H 1 ( Ω ) 2 .
Combining the two bounds above yields that with probability 1 − e − t 1-e^{-t} 1 − e − t , we have:
∥ u ^ P I N N − u ∗ ∥ H 1 ( Ω ) 2 ≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \left\|\hat{u}_{\mathrm{PINN}}-u^*\right\|_{H^1(\Omega)}^2\lesssim\inf_{u_F\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_F)-\mathcal{E}(u^*)\right)+\max\left\{r^*,\frac{t}{n}\right\}. ∥ u ^ PINN − u ∗ ∥ H 1 ( Ω ) 2 ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Proof of The Meta-Theorem for MDRM
Proof. To upper bound the excess risk Δ E ( N , n ) : = E ( u ^ M D R M ) − E ( u ∗ ) \Delta\mathcal{E}^{(N,n)} := \mathcal{E}(\hat{u}_{\mathrm{MDRM}}) - \mathcal{E}(u^{*}) Δ E ( N , n ) := E ( u ^ MDRM ) − E ( u ∗ ) , following [78 , 47 , 14 ] , we decompose the excess risk into approximation error and generalization error with probability 1 − e − t 1-e^{-t} 1 − e − t :
Δ E ( N , n ) = [ E ( u ^ M D R M ) − E ( u ∗ ) ] = [ E ( u ^ M D R M ) − E N , n ( u ^ M D R M ) ] + [ E N , n ( u ^ M D R M ) − E N , n ( u F ) ] + [ E N , n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u ^ M D R M ) − E N , n ( u ^ M D R M ) ] + [ E N , n ( u F ) − E ( u F ) ] + [ E ( u F ) − E ( u ∗ ) ] ≤ [ E ( u ^ M D R M ) − E ( u ∗ ) + E N , n ( u ∗ ) − E N , n ( u ^ M D R M ) ] + 2 [ E ( u F ) − E ( u ∗ ) ] + 4 t min { N , n } , (93) \begin{aligned}
\Delta\mathcal{E}^{(N,n)}
&= [\mathcal{E}(\hat{u}_{\mathrm{MDRM}})-\mathcal{E}(u^{*})] \\
&= [\mathcal{E}(\hat{u}_{\mathrm{MDRM}})-\mathcal{E}_{N,n}(\hat{u}_{\mathrm{MDRM}})] + [\mathcal{E}_{N,n}(\hat{u}_{\mathrm{MDRM}})-\mathcal{E}_{N,n}(u_{F})] \\
&\quad+ [\mathcal{E}_{N,n}(u_{F})-\mathcal{E}(u_{F})] + [\mathcal{E}(u_{F})-\mathcal{E}(u^{*})] \\
&\leq[\mathcal{E}(\hat{u}_{\mathrm{MDRM}})-\mathcal{E}_{N,n}(\hat{u}_{\mathrm{MDRM}})] + [\mathcal{E}_{N,n}(u_{F})-\mathcal{E}(u_{F})] + [\mathcal{E}(u_{F})-\mathcal{E}(u^{*})] \\
&\leq[\mathcal{E}(\hat{u}_{\mathrm{MDRM}})-\mathcal{E}(u^{*})+\mathcal{E}_{N,n}(u^{*})-\mathcal{E}_{N,n}(\hat{u}_{\mathrm{MDRM}})] \\
&\quad+ 2[\mathcal{E}(u_{F})-\mathcal{E}(u^{*})] + \frac{4t}{\min\{N,n\}}, \tag*{(93)}
\end{aligned} Δ E ( N , n ) = [ E ( u ^ MDRM ) − E ( u ∗ )] = [ E ( u ^ MDRM ) − E N , n ( u ^ MDRM )] + [ E N , n ( u ^ MDRM ) − E N , n ( u F )] + [ E N , n ( u F ) − E ( u F )] + [ E ( u F ) − E ( u ∗ )] ≤ [ E ( u ^ MDRM ) − E N , n ( u ^ MDRM )] + [ E N , n ( u F ) − E ( u F )] + [ E ( u F ) − E ( u ∗ )] ≤ [ E ( u ^ MDRM ) − E ( u ∗ ) + E N , n ( u ∗ ) − E N , n ( u ^ MDRM )] + 2 [ E ( u F ) − E ( u ∗ )] + min { N , n } 4 t , (93)
where the expectation is on all sampled data. The inequality of the third line is because u ^ M D R M \hat{u}_{\mathrm{MDRM}} u ^ MDRM is the minimizer of the empirical loss E n \mathcal{E}_{n} E n in the solution set F ( Ω ) \mathcal{F}(\Omega) F ( Ω ) , so we have E N , n ( u ^ M D R M ) ≤ E N , n ( u F ) \mathcal{E}_{N,n}(\hat{u}_{\mathrm{MDRM}}) \leq\mathcal{E}_{N,n}(u_{F}) E N , n ( u ^ MDRM ) ≤ E N , n ( u F ) . The last inequality is based on the Bernstein inequality. For any u F ∈ F ( Ω ) u_{F} \in\mathcal{F}(\Omega) u F ∈ F ( Ω ) , we use h F , 1 , h F , 2 h_{F,1}, h_{F,2} h F , 1 , h F , 2 to denote the following two functions:
h F , 1 : = 1 2 ( ∥ ∇ u F ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) , h_{F,1}:=\frac{1}{2}\left(\|\nabla u_{F}\|^{2}-\|\nabla u^{*}\|^{2}\right), h F , 1 := 2 1 ( ∥∇ u F ∥ 2 − ∥∇ u ∗ ∥ 2 ) ,
h F , 2 : = 1 2 V ( ∣ u F ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u F − u ∗ ) . h_{F,2}:=\frac{1}{2}V(|u_{F}|^{2}-|u^{*}|^{2})-f(u_{F}-u^{*}). h F , 2 := 2 1 V ( ∣ u F ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u F − u ∗ ) .
Applying Bernstein’s inequality twice to h F , 1 h_{F,1} h F , 1 and h F , 2 h_{F,2} h F , 2 implies that there exists some constant C q C_{q} C q , such that with probability 1 − 2 e − C q t 1-2e^{-C_{q}t} 1 − 2 e − C q t , the following two inequalities hold simultaneously:
E N ( h F , 1 ) − E ( h F , 1 ) ≤ t α ′ E [ h F , 1 2 ] N , \mathcal{E}_{N}(h_{F,1})-\mathcal{E}(h_{F,1})\leq\sqrt{\frac{t}{\alpha'}\frac{\mathcal{E}[h_{F,1}^{2}]}{N}}, E N ( h F , 1 ) − E ( h F , 1 ) ≤ α ′ t N E [ h F , 1 2 ] ,
E n ( h F , 2 ) − E ( h F , 2 ) ≤ t α ′ E [ h F , 2 2 ] n . \mathcal{E}_{n}(h_{F,2})-\mathcal{E}(h_{F,2})\leq\sqrt{\frac{t}{\alpha'}\frac{\mathcal{E}[h_{F,2}^{2}]}{n}}. E n ( h F , 2 ) − E ( h F , 2 ) ≤ α ′ t n E [ h F , 2 2 ] .
Note that the variance sum E [ h F , 1 2 ] + E [ h F , 2 2 ] \mathcal{E}[h_{F,1}^{2}]+\mathcal{E}[h_{F,2}^{2}] E [ h F , 1 2 ] + E [ h F , 2 2 ] can be upper bounded by α ′ α [ E ( u F ) − E ( u ∗ ) ] \frac{\alpha'}{\alpha}[\mathcal{E}(u_{F})-\mathcal{E}(u^{*})] α α ′ [ E ( u F ) − E ( u ∗ )] due to the strong convexity of the variation objective (A.57). Adding the two inequalities above implies with probability 1 − 2 e − C q t 1-2e^{-C_{q}t} 1 − 2 e − C q t we have:
E N , n ( u F ) − E N , n ( u ∗ ) − E ( u F ) + E ( u ∗ ) = E N ( h F , 1 ) − E ( h F , 1 ) + E n ( h F , 2 ) − E ( h F , 2 ) ≤ t α ′ E [ h F , 1 2 ] N + t α ′ E [ h F , 2 2 ] n ≤ 2 t α ′ E [ h F , 1 2 ] + E [ h F , 1 2 ] min { N , n } ≤ 2 t [ E ( u F ) − E ( u ∗ ) ] min { N , n } ≤ [ E ( u F ) − E ( u ∗ ) ] + 4 t min { N , n } . \begin{aligned}
\mathcal{E}_{N,n}(u_{F})-\mathcal{E}_{N,n}(u^{*})-\mathcal{E}(u_{F})+\mathcal{E}(u^{*})
&=\mathcal{E}_{N}(h_{F,1})-\mathcal{E}(h_{F,1})+\mathcal{E}_{n}(h_{F,2})-\mathcal{E}(h_{F,2}) \\
&\leq\sqrt{\frac{t}{\alpha'}\frac{\mathcal{E}[h_{F,1}^{2}]}{N}}+\sqrt{\frac{t}{\alpha'}\frac{\mathcal{E}[h_{F,2}^{2}]}{n}} \\
&\leq\sqrt{\frac{2t}{\alpha'}\frac{\mathcal{E}[h_{F,1}^{2}]+\mathcal{E}[h_{F,1}^{2}]}{\min\{N,n\}}} \\
&\leq\sqrt{\frac{2t[\mathcal{E}(u_{F})-\mathcal{E}(u^{*})]}{\min\{N,n\}}} \\
&\leq[\mathcal{E}(u_{F})-\mathcal{E}(u^{*})]+\frac{4t}{\min\{N,n\}}.
\end{aligned} E N , n ( u F ) − E N , n ( u ∗ ) − E ( u F ) + E ( u ∗ ) = E N ( h F , 1 ) − E ( h F , 1 ) + E n ( h F , 2 ) − E ( h F , 2 ) ≤ α ′ t N E [ h F , 1 2 ] + α ′ t n E [ h F , 2 2 ] ≤ α ′ 2 t min { N , n } E [ h F , 1 2 ] + E [ h F , 1 2 ] ≤ min { N , n } 2 t [ E ( u F ) − E ( u ∗ )] ≤ [ E ( u F ) − E ( u ∗ )] + min { N , n } 4 t .
Note that A.53 holds for all function lies in the function space F \mathcal{F} F . Thus, we can take u F : = arg min u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) u_{\mathcal{F}} := \operatorname*{arg\,min}_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)} \left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right) u F := arg min u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) and finally get:
Δ E ( N , n ) ≤ E ( u ^ M D R M ) − E ( u ∗ ) + E N , n ( u ∗ ) − E N , n ( u ^ M D R M ) ⏟ Δ E g e n + 2 inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) ⏟ Δ E a p p + 4 t n . \Delta\mathcal{E}^{(N,n)} \leq\underbrace{\mathcal{E}(\hat{u}_{\mathrm{MDRM}})-\mathcal{E}(u^*)+\mathcal{E}_{N,n}(u^*)-\mathcal{E}_{N,n}(\hat{u}_{\mathrm{MDRM}})}_{\Delta\mathcal{E}_{\mathrm{gen}}}+2\underbrace{\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right)}_{\Delta\mathcal{E}_{\mathrm{app}}}+\frac{4t}{n}. Δ E ( N , n ) ≤ Δ E gen E ( u ^ MDRM ) − E ( u ∗ ) + E N , n ( u ∗ ) − E N , n ( u ^ MDRM ) + 2 Δ E app u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + n 4 t .
This inequality decomposes the excess risk to the generalization error Δ E g e n : = E ( u ^ M D R M ) − E ( u ∗ ) + E N , n ( u ∗ ) − E N , n ( u ^ M D R M ) \Delta\mathcal{E}_{\mathrm{gen}} := \mathcal{E}(\hat{u}_{\mathrm{MDRM}})-\mathcal{E}(u^*)+\mathcal{E}_{N,n}(u^*)-\mathcal{E}_{N,n}(\hat{u}_{\mathrm{MDRM}}) Δ E gen := E ( u ^ MDRM ) − E ( u ∗ ) + E N , n ( u ∗ ) − E N , n ( u ^ MDRM ) and the approximation error Δ E a p p = inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) \Delta\mathcal{E}_{\mathrm{app}} = \inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right) Δ E app = inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) . From the lemmata proved in Section ??, we already have an estimation of the approximation error’s convergence rate. So now we’ll focus on providing fast rate upper bounds of the generalization error for the two estimators using the localization techinque[3 , 78 ] . To achieve the fast generalization bound, we focus on the following two normalized empirical processes:
S ~ r , 1 ( Ω ) : = { h ~ 1 ( x ) : = E [ h 1 ] − h 1 ( x ) E [ h 1 ] + E [ h 2 ] + r | ( h 1 , h 2 ) ∈ S ( Ω ) } ( r > 0 ) , \widetilde{S}_{r,1}(\Omega):=\left\{\widetilde{h}_1(x):=\frac{\mathbb{E}[h_1]-h_1(x)}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\,\middle|\,(h_1,h_2)\in S(\Omega)\right\}\quad(r>0), S r , 1 ( Ω ) := { h 1 ( x ) := E [ h 1 ] + E [ h 2 ] + r E [ h 1 ] − h 1 ( x ) ( h 1 , h 2 ) ∈ S ( Ω ) } ( r > 0 ) ,
S ~ r , 2 ( Ω ) : = { h ~ 2 ( x ) : = E [ h 2 ] − h 2 ( x ) E [ h 1 ] + E [ h 2 ] + r | ( h 1 , h 2 ) ∈ S ( Ω ) } ( r > 0 ) . \widetilde{S}_{r,2}(\Omega):=\left\{\widetilde{h}_2(x):=\frac{\mathbb{E}[h_2]-h_2(x)}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\,\middle|\,(h_1,h_2)\in S(\Omega)\right\}\quad(r>0). S r , 2 ( Ω ) := { h 2 ( x ) := E [ h 1 ] + E [ h 2 ] + r E [ h 2 ] − h 2 ( x ) ( h 1 , h 2 ) ∈ S ( Ω ) } ( r > 0 ) .
where the space S ( Ω ) S(\Omega) S ( Ω ) is defined as:
S(\Omega):=\{(h_1,h_2)\middle|h_1:=|\Omega|\cdot[\frac{1}{2}(\|\nabla u\|^2-\|\nabla u^*\|^2)],
h 2 : = ∣ Ω ∣ ⋅ [ 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] , u ∈ F ( Ω ) } . h_2:=|\Omega|\cdot[\frac{1}{2}\mathcal{V}(|u|^2-|u^*|^2)-f(u-u^*)],\,u\in\mathcal{F}(\Omega)\}. h 2 := ∣Ω∣ ⋅ [ 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ )] , u ∈ F ( Ω )} .
First, we try to bound the expectation of the two normalized empirical processes. Applying the Symmetrization Lemma A.3, we can first bound the two expectations as:
sup h ~ 1 ∈ S ~ r , 1 ( Ω ) E y ′ [ 1 N ∑ i = 1 N h ~ 1 ( y i ′ ) ] ≤ E y ′ [ sup h 1 ∈ S 1 ( Ω ) ∣ 1 N ∑ i = 1 N h 1 ( y i ′ ) − E [ h 1 ] E [ h 1 ] + E [ h 2 ] + r ∣ ] ≤ 2 R N ( S ^ r , 1 ( Ω ) ) , \sup_{\widetilde{h}_1\in\widetilde{S}_{r,1}(\Omega)}\mathbb{E}_{y'}\left[\frac{1}{N}\sum_{i=1}^{N}\widetilde{h}_1(y_i')\right]\leq\mathbb{E}_{y'}\left[\sup_{h_1\in S_1(\Omega)}\left|\frac{1}{N}\sum_{i=1}^{N}\frac{h_1(y_i')-\mathbb{E}[h_1]}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\right|\right]\leq2R_N(\widehat{S}_{r,1}(\Omega)), h 1 ∈ S r , 1 ( Ω ) sup E y ′ [ N 1 i = 1 ∑ N h 1 ( y i ′ ) ] ≤ E y ′ [ h 1 ∈ S 1 ( Ω ) sup N 1 i = 1 ∑ N E [ h 1 ] + E [ h 2 ] + r h 1 ( y i ′ ) − E [ h 1 ] ] ≤ 2 R N ( S r , 1 ( Ω )) ,
sup h ~ 2 ∈ S ~ r , 2 ( Ω ) E y [ 1 n ∑ j = 1 n h ~ 2 ( y j ) ] ≤ E y [ sup h 2 ∈ S 2 ( Ω ) ∣ 1 n ∑ j = 1 n h 2 ( y j ) − E [ h 2 ] E [ h 1 ] + E [ h 2 ] + r ∣ ] ≤ 2 R n ( S ^ r , 2 ( Ω ) ) . \sup_{\widetilde{h}_2\in\widetilde{S}_{r,2}(\Omega)}\mathbb{E}_{y}\left[\frac{1}{n}\sum_{j=1}^{n}\widetilde{h}_2(y_j)\right]\leq\mathbb{E}_{y}\left[\sup_{h_2\in S_2(\Omega)}\left|\frac{1}{n}\sum_{j=1}^{n}\frac{h_2(y_j)-\mathbb{E}[h_2]}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\right|\right]\leq2R_n(\widehat{S}_{r,2}(\Omega)). h 2 ∈ S r , 2 ( Ω ) sup E y [ n 1 j = 1 ∑ n h 2 ( y j ) ] ≤ E y [ h 2 ∈ S 2 ( Ω ) sup n 1 j = 1 ∑ n E [ h 1 ] + E [ h 2 ] + r h 2 ( y j ) − E [ h 2 ] ] ≤ 2 R n ( S r , 2 ( Ω )) .
where the function classes S ^ r , k ( Ω ) \widehat{S}_{r,k}(\Omega) S r , k ( Ω ) ( 1 ≤ k ≤ 2 ) (1\leq k\leq2) ( 1 ≤ k ≤ 2 ) are defined as:
S ^ r , 1 ( Ω ) : = { h ^ 1 ( x ) : = h 1 ( x ) E [ h 1 ] + E [ h 2 ] + r | ( h 1 , h 2 ) ∈ S ( Ω ) } , \widehat{S}_{r,1}(\Omega):=\left\{\widehat{h}_1(x):=\frac{h_1(x)}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\,\middle|\,(h_1,h_2)\in S(\Omega)\right\}, S r , 1 ( Ω ) := { h 1 ( x ) := E [ h 1 ] + E [ h 2 ] + r h 1 ( x ) ( h 1 , h 2 ) ∈ S ( Ω ) } ,
S ^ r , 2 ( Ω ) : = { h ^ 2 ( x ) : = h 2 ( x ) E [ h 1 ] + E [ h 2 ] + r | ( h 1 , h 2 ) ∈ S ( Ω ) } . \widehat{S}_{r,2}(\Omega):=\left\{\widehat{h}_2(x):=\frac{h_2(x)}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\,\middle|\,(h_1,h_2)\in S(\Omega)\right\}. S r , 2 ( Ω ) := { h 2 ( x ) := E [ h 1 ] + E [ h 2 ] + r h 2 ( x ) ( h 1 , h 2 ) ∈ S ( Ω ) } .
Applying the modified Peeling Lemma A.8 to any function h = ( h 1 , h 2 ) ∈ S ( Ω ) h=(h_1,h_2)\in S(\Omega) h = ( h 1 , h 2 ) ∈ S ( Ω ) helps us upper bound the sum of the two local Rademacher complexities R N ( S ^ r , 1 ( Ω ) ) + R n ( S ^ r , 2 ( Ω ) ) R_N(\widehat{S}_{r,1}(\Omega))+R_n(\widehat{S}_{r,2}(\Omega)) R N ( S r , 1 ( Ω )) + R n ( S r , 2 ( Ω )) with the function ϕ \phi ϕ defined in equation 5.5:
R N ( S ^ r , 1 ( Ω ) ) + R n ( S ^ r , 2 ( Ω ) ) = E σ [ E y [ sup h ∈ S ( Ω ) 1 N ∑ i = 1 N σ i h 1 ( y i ) E [ h 1 ] + E [ h 2 ] + r ] ] + E τ [ E y ′ [ sup h ∈ S ( Ω ) 1 n ∑ j = 1 n τ j h 2 ( y j ′ ) E [ h 1 ] + E [ h 2 ] + r ] ] = E σ [ E y , y ′ [ sup h ∈ S ( Ω ) 1 N ∑ i = 1 N σ i h 1 ( y i ) E [ h 1 ] + E [ h 2 ] + r + sup h ∈ S ( Ω ) 1 n ∑ j = 1 n τ j h 2 ( y j ′ ) E [ h 1 ] + E [ h 2 ] + r ] ] = R N , n ( S ^ r ( Ω ) ) ≤ 4 ϕ ( r ) r . \begin{aligned}
R_N(\widehat{S}_{r,1}(\Omega))+R_n(\widehat{S}_{r,2}(\Omega))
&=\mathbb{E}_{\sigma}\left[\mathbb{E}_{y}\left[\sup_{h\in S(\Omega)}\frac{\frac{1}{N}\sum_{i=1}^{N}\sigma_i h_1(y_i)}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\right]\right]\\
&\quad+\mathbb{E}_{\tau}\left[\mathbb{E}_{y'}\left[\sup_{h\in S(\Omega)}\frac{\frac{1}{n}\sum_{j=1}^{n}\tau_j h_2(y'_j)}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\right]\right]\\
&=\mathbb{E}_{\sigma}\left[\mathbb{E}_{y,y'}\left[\sup_{h\in S(\Omega)}\frac{\frac{1}{N}\sum_{i=1}^{N}\sigma_i h_1(y_i)}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}+\sup_{h\in S(\Omega)}\frac{\frac{1}{n}\sum_{j=1}^{n}\tau_j h_2(y'_j)}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\right]\right]\\
&=R_{N,n}(\widehat{S}_r(\Omega))\leq\frac{4\phi(r)}{r}.
\end{aligned} R N ( S r , 1 ( Ω )) + R n ( S r , 2 ( Ω )) = E σ [ E y [ h ∈ S ( Ω ) sup E [ h 1 ] + E [ h 2 ] + r N 1 ∑ i = 1 N σ i h 1 ( y i ) ] ] + E τ [ E y ′ [ h ∈ S ( Ω ) sup E [ h 1 ] + E [ h 2 ] + r n 1 ∑ j = 1 n τ j h 2 ( y j ′ ) ] ] = E σ [ E y , y ′ [ h ∈ S ( Ω ) sup E [ h 1 ] + E [ h 2 ] + r N 1 ∑ i = 1 N σ i h 1 ( y i ) + h ∈ S ( Ω ) sup E [ h 1 ] + E [ h 2 ] + r n 1 ∑ j = 1 n τ j h 2 ( y j ′ ) ] ] = R N , n ( S r ( Ω )) ≤ r 4 ϕ ( r ) .
Combining all inequalities derived above yields:
sup h ~ 1 ∈ S ~ r , 1 ( Ω ) E y ′ [ 1 N ∑ i = 1 N h ~ 1 ( y i ′ ) ] + sup h ~ 2 ∈ S ~ r , 2 ( Ω ) E y [ 1 n ∑ j = 1 n h ~ 2 ( y j ) ] ≤ 2 R N ( S ^ r , 1 ( Ω ) ) + 2 R n ( S ^ r , 2 ( Ω ) ) = 2 R N , n ( S ^ r ( Ω ) ) ≤ 8 ϕ ( r ) r ( r > 0 ) . (94) \sup_{\widetilde{h}_1\in\widetilde{S}_{r,1}(\Omega)}\mathbb{E}_{y'}\left[\frac{1}{N}\sum_{i=1}^{N}\widetilde{h}_1(y'_i)\right]+\sup_{\widetilde{h}_2\in\widetilde{S}_{r,2}(\Omega)}\mathbb{E}_{y}\left[\frac{1}{n}\sum_{j=1}^{n}\widetilde{h}_2(y_j)\right]
\leq2R_N(\widehat{S}_{r,1}(\Omega))+2R_n(\widehat{S}_{r,2}(\Omega))=2R_{N,n}(\widehat{S}_r(\Omega))\leq\frac{8\phi(r)}{r}\quad(r>0). \tag*{(94)} h 1 ∈ S r , 1 ( Ω ) sup E y ′ [ N 1 i = 1 ∑ N h 1 ( y i ′ ) ] + h 2 ∈ S r , 2 ( Ω ) sup E y [ n 1 j = 1 ∑ n h 2 ( y j ) ] ≤ 2 R N ( S r , 1 ( Ω )) + 2 R n ( S r , 2 ( Ω )) = 2 R N , n ( S r ( Ω )) ≤ r 8 ϕ ( r ) ( r > 0 ) . (94)
Secondly, we’ll apply the Talagrand concentration inequality to the two function classes S ~ r , 1 ( Ω ) \widetilde{S}_{r,1}(\Omega) S r , 1 ( Ω ) and S ~ r , 2 ( Ω ) \widetilde{S}_{r,2}(\Omega) S r , 2 ( Ω ) , which requires us to verify the conditions needed. We will first check that the expectation sum E [ h 1 ] + E [ h 2 ] \mathbb{E}[h_1]+\mathbb{E}[h_2] E [ h 1 ] + E [ h 2 ] is always non-negative for any ( h 1 , h 2 ) ∈ S ( Ω ) (h_1,h_2)\in S(\Omega) ( h 1 , h 2 ) ∈ S ( Ω ) :
E [ h 1 ] + E [ h 2 ] = 1 ∣ Ω ∣ ∫ Ω ( ∣ Ω ∣ ⋅ ( 1 2 ∥ ∇ u ( x ) ∥ 2 + 1 2 V ( x ) ∣ u ( x ) ∣ 2 − f ( x ) u ( x ) ) ) d x − 1 ∣ Ω ∣ ∫ Ω ( ∣ Ω ∣ ⋅ ( 1 2 ∥ ∇ u ∗ ( x ) ∥ 2 + 1 2 V ( x ) ∣ u ∗ ( x ) ∣ 2 − f ( x ) u ∗ ( x ) ) ) d x = E ( u ) − E ( u ∗ ) ≥ 0 ⇒ E [ h 1 ] + E [ h 2 ] ≥ 0. \begin{aligned}
\mathbb{E}[h_1]+\mathbb{E}[h_2]
&=\frac{1}{|\Omega|}\int_{\Omega}\left(|\Omega|\cdot\left(\frac{1}{2}\lVert\nabla u(x)\rVert^2+\frac{1}{2}V(x)|u(x)|^2-f(x)u(x)\right)\right)\,dx\\
&\quad-\frac{1}{|\Omega|}\int_{\Omega}\left(|\Omega|\cdot\left(\frac{1}{2}\lVert\nabla u^*(x)\rVert^2+\frac{1}{2}V(x)|u^*(x)|^2-f(x)u^*(x)\right)\right)\,dx\\
&=\mathcal{E}(u)-\mathcal{E}(u^*)\geq0\Rightarrow\mathbb{E}[h_1]+\mathbb{E}[h_2]\geq0.
\end{aligned} E [ h 1 ] + E [ h 2 ] = ∣Ω∣ 1 ∫ Ω ( ∣Ω∣ ⋅ ( 2 1 ∥ ∇ u ( x ) ∥ 2 + 2 1 V ( x ) ∣ u ( x ) ∣ 2 − f ( x ) u ( x ) ) ) d x − ∣Ω∣ 1 ∫ Ω ( ∣Ω∣ ⋅ ( 2 1 ∥ ∇ u ∗ ( x ) ∥ 2 + 2 1 V ( x ) ∣ u ∗ ( x ) ∣ 2 − f ( x ) u ∗ ( x ) ) ) d x = E ( u ) − E ( u ∗ ) ≥ 0 ⇒ E [ h 1 ] + E [ h 2 ] ≥ 0.
Next, We will verify that S ~ r , 1 ( Ω ) \widetilde{S}_{r,1}(\Omega) S r , 1 ( Ω ) satisfies all three requirements. At first, we will show that any h ~ 1 = E [ h 1 ] − h 1 E [ h 1 ] + E [ h 2 ] + r ∈ S ~ r , 1 ( Ω ) \widetilde{h}_1=\frac{\mathbb{E}[h_1]-h_1}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\in\widetilde{S}_{r,1}(\Omega) h 1 = E [ h 1 ] + E [ h 2 ] + r E [ h 1 ] − h 1 ∈ S r , 1 ( Ω ) is of bounded inf-norm. We need to prove that any h 1 ∈ S 1 ( Ω ) h_1\in S_1(\Omega) h 1 ∈ S 1 ( Ω ) is of bounded inf-norm beforehand. Using boundedness condition listed in equation 5.4 implies:
∥ h 1 ∥ ∞ = ∥ 1 2 ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) ∥ ∞ ≤ 1 2 ( ∥ ∇ u ∥ ∞ 2 + ∥ ∇ u ∗ ∥ ∞ 2 ) ≤ C 2 . \lVert h_1\rVert_{\infty}=\left\lVert\frac{1}{2}\left(\lVert\nabla u\rVert^2-\lVert\nabla u^*\rVert^2\right)\right\rVert_{\infty}\leq\frac{1}{2}\left(\lVert\nabla u\rVert_{\infty}^2+\lVert\nabla u^*\rVert_{\infty}^2\right)\leq C^2. ∥ h 1 ∥ ∞ = 2 1 ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) ∞ ≤ 2 1 ( ∥ ∇ u ∥ ∞ 2 + ∥ ∇ u ∗ ∥ ∞ 2 ) ≤ C 2 .
By taking M 1 : = C 2 M_1:=C^2 M 1 := C 2 , we then have ∥ h 1 ∥ ∞ ≤ M 1 \lVert h_1\rVert_{\infty}\leq M_1 ∥ h 1 ∥ ∞ ≤ M 1 for all h 1 ∈ S 1 ( Ω ) h_1\in S_1(\Omega) h 1 ∈ S 1 ( Ω ) . Note that the denominator of h ~ 1 \widetilde{h}_1 h 1 can be lower bounded by ∣ E [ h 1 ] + E [ h 2 ] + r ∣ ≥ r > 0 |\mathbb{E}[h_1]+\mathbb{E}[h_2]+r|\geq r>0 ∣ E [ h 1 ] + E [ h 2 ] + r ∣ ≥ r > 0 . Combining these two inequalities help us upper bound the inf-norm ∥ h ~ 1 ∥ ∞ = sup x ∈ Ω ∣ h ~ 1 ( x ) ∣ \lVert\widetilde{h}_1\rVert_{\infty}=\sup_{x\in\Omega}|\widetilde{h}_1(x)| ∥ h 1 ∥ ∞ = sup x ∈ Ω ∣ h 1 ( x ) ∣ as follows:
∥ h ~ 1 ∥ ∞ = ∥ E [ h 1 ] − h 1 ∥ ∞ ∣ E [ h 1 ] + E [ h 2 ] + r ∣ ≤ 2 ∥ h 1 ∥ ∞ r ≤ 2 M 1 r = : β 1 . \lVert\widetilde{h}_1\rVert_{\infty}=\frac{\lVert\mathbb{E}[h_1]-h_1\rVert_{\infty}}{|\mathbb{E}[h_1]+\mathbb{E}[h_2]+r|}\leq\frac{2\lVert h_1\rVert_{\infty}}{r}\leq\frac{2M_1}{r}=:\beta_1. ∥ h 1 ∥ ∞ = ∣ E [ h 1 ] + E [ h 2 ] + r ∣ ∥ E [ h 1 ] − h 1 ∥ ∞ ≤ r 2 ∥ h 1 ∥ ∞ ≤ r 2 M 1 =: β 1 .
Also, it is easy to check that for any h ~ 1 ∈ S ~ r , 1 ( Ω ) \widetilde{h}_1\in\widetilde{S}_{r,1}(\Omega) h 1 ∈ S r , 1 ( Ω ) , we have
E [ h ~ 1 ] = E [ h 1 ] − E [ h 1 ] E [ h 1 ] + E [ h 2 ] + r = 0 , \mathbb{E}[\widetilde{h}_1]=\frac{\mathbb{E}[h_1]-\mathbb{E}[h_1]}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}=0, E [ h 1 ] = E [ h 1 ] + E [ h 2 ] + r E [ h 1 ] − E [ h 1 ] = 0 ,
i.e. any function in the localized class S ~ r , 1 ( Ω ) \widetilde{\mathcal{S}}_{r,1}(\Omega) S r , 1 ( Ω ) is of zero mean. Moreover, we take σ 1 2 = sup h ~ 1 ∈ S ~ r , 1 ( Ω ) E [ h ~ 1 2 ] \sigma_1^2=\sup_{\widetilde{h}_1\in\widetilde{\mathcal{S}}_{r,1}(\Omega)}\mathbb{E}[\widetilde{h}_1^2] σ 1 2 = sup h 1 ∈ S r , 1 ( Ω ) E [ h 1 2 ] to be the upper bound on the second moment of functions in S ~ r , 1 ( Ω ) \widetilde{\mathcal{S}}_{r,1}(\Omega) S r , 1 ( Ω ) . Now we have verified that any function h ~ 1 ∈ S ~ r , 1 ( Ω ) \widetilde{h}_1\in\widetilde{\mathcal{S}}_{r,1}(\Omega) h 1 ∈ S r , 1 ( Ω ) satisfies all the required conditions. By taking μ \mu μ to be the uniform distribution on the domain Ω \Omega Ω and applying Talagrand’s Concentration inequality given in Lemma A.6, we have:
P x [ sup h ~ 1 ∈ S ~ r , 1 ( Ω ) 1 N ∑ i = 1 N h ~ 1 ( x i ) ≥ 2 sup h ~ 1 ∈ S ~ r , 1 ( Ω ) E y [ 1 N ∑ i = 1 N h ~ 1 ( y i ) ] + 2 t σ 1 2 N + 2 t β 1 N ] ≤ e − t . (95) \mathbb{P}_{x}\left[\sup_{\widetilde{h}_1\in\widetilde{\mathcal{S}}_{r,1}(\Omega)}\frac{1}{N}\sum_{i=1}^{N}\widetilde{h}_1(x_i)\geq2\sup_{\widetilde{h}_1\in\widetilde{\mathcal{S}}_{r,1}(\Omega)}\mathbb{E}_{y}\left[\frac{1}{N}\sum_{i=1}^{N}\widetilde{h}_1(y_i)\right]+\sqrt{\frac{2t\sigma_1^2}{N}+\frac{2t\beta_1}{N}}\right]\leq e^{-t}. \tag*{(95)} P x [ h 1 ∈ S r , 1 ( Ω ) sup N 1 i = 1 ∑ N h 1 ( x i ) ≥ 2 h 1 ∈ S r , 1 ( Ω ) sup E y [ N 1 i = 1 ∑ N h 1 ( y i ) ] + N 2 t σ 1 2 + N 2 t β 1 ] ≤ e − t . (95)
Moreover, We will verify that S ~ r , 2 ( Ω ) \widetilde{\mathcal{S}}_{r,2}(\Omega) S r , 2 ( Ω ) also satisfies all three requirements. At first, we will show that any h ~ 2 = E [ h 2 ] − h 2 E [ h 1 ] + E [ h 2 ] + r ∈ S ~ r , 2 ( Ω ) \widetilde{h}_2=\frac{\mathbb{E}[h_2]-h_2}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}\in\widetilde{\mathcal{S}}_{r,2}(\Omega) h 2 = E [ h 1 ] + E [ h 2 ] + r E [ h 2 ] − h 2 ∈ S r , 2 ( Ω ) is of bounded inf-norm. We need to prove that any h 2 ∈ S 2 ( Ω ) h_2\in\mathcal{S}_2(\Omega) h 2 ∈ S 2 ( Ω ) is of bounded inf-norm beforehand. Using boundedness condition listed in equation (40) implies:
∥ h 2 ∥ ∞ = ∥ 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ∥ ∞ ≤ 1 2 V max ( ∥ u ∥ ∞ 2 + ∥ u ∗ ∥ ∞ 2 ) + ∥ f ∥ ∞ ( ∥ u ∥ ∞ + ∥ u ∗ ∥ ∞ ) ≤ 1 2 V max × 2 C 2 + 2 C 2 = ( V max + 2 ) C 2 . \begin{aligned}
\lVert h_2\rVert_\infty
&=\left\lVert\frac{1}{2}V\left(\lvert u\rvert^2-\lvert u^*\rvert^2\right)-f(u-u^*)\right\rVert_\infty\\
&\leq\frac{1}{2}V_{\max}\left(\lVert u\rVert_\infty^2+\lVert u^*\rVert_\infty^2\right)+\lVert f\rVert_\infty\left(\lVert u\rVert_\infty+\lVert u^*\rVert_\infty\right)\\
&\leq\frac{1}{2}V_{\max}\times2C^2+2C^2=(V_{\max}+2)C^2.
\end{aligned} ∥ h 2 ∥ ∞ = 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ∞ ≤ 2 1 V m a x ( ∥ u ∥ ∞ 2 + ∥ u ∗ ∥ ∞ 2 ) + ∥ f ∥ ∞ ( ∥ u ∥ ∞ + ∥ u ∗ ∥ ∞ ) ≤ 2 1 V m a x × 2 C 2 + 2 C 2 = ( V m a x + 2 ) C 2 .
By taking M 2 : = ( V max + 2 ) C 2 M_2:=(V_{\max}+2)C^2 M 2 := ( V m a x + 2 ) C 2 , we then have ∥ h 2 ∥ ∞ ≤ M 2 \lVert h_2\rVert_\infty\leq M_2 ∥ h 2 ∥ ∞ ≤ M 2 for all h 2 ∈ S 2 ( Ω ) h_2\in\mathcal{S}_2(\Omega) h 2 ∈ S 2 ( Ω ) . Note that the denominator of h ~ 2 \widetilde{h}_2 h 2 can be lower bounded by ∣ E [ h 1 ] + E [ h 2 ] + r ∣ ≥ r > 0 \lvert\mathbb{E}[h_1]+\mathbb{E}[h_2]+r\rvert\geq r>0 ∣ E [ h 1 ] + E [ h 2 ] + r ∣ ≥ r > 0 . Combining these two inequalities help us upper bound the inf-norm ∥ h ~ 2 ∥ ∞ = sup x ∈ Ω ∣ h ~ 2 ( x ) ∣ \lVert\widetilde{h}_2\rVert_\infty=\sup_{x\in\Omega}\lvert\widetilde{h}_2(x)\rvert ∥ h 2 ∥ ∞ = sup x ∈ Ω ∣ h 2 ( x )∣ as follows:
∥ h ~ 2 ∥ ∞ = ∥ E [ h 2 ] − h 2 ∥ ∞ ∣ E [ h 1 ] + E [ h 2 ] + r ∣ ≤ 2 ∥ h 2 ∥ ∞ r ≤ 2 M 2 r = : β 2 . \lVert\widetilde{h}_2\rVert_\infty=\frac{\lVert\mathbb{E}[h_2]-h_2\rVert_\infty}{\lvert\mathbb{E}[h_1]+\mathbb{E}[h_2]+r\rvert}\leq\frac{2\lVert h_2\rVert_\infty}{r}\leq\frac{2M_2}{r}=:\beta_2. ∥ h 2 ∥ ∞ = ∣ E [ h 1 ] + E [ h 2 ] + r ∣ ∥ E [ h 2 ] − h 2 ∥ ∞ ≤ r 2 ∥ h 2 ∥ ∞ ≤ r 2 M 2 =: β 2 .
Also, it is easy to check that for any h ~ 2 ∈ S ~ r , 2 ( Ω ) \widetilde{h}_2\in\widetilde{\mathcal{S}}_{r,2}(\Omega) h 2 ∈ S r , 2 ( Ω ) , we have
E [ h ~ 2 ] = E [ h 2 ] − E [ h 2 ] E [ h 1 ] + E [ h 2 ] + r = 0 , \mathbb{E}[\widetilde{h}_2]=\frac{\mathbb{E}[h_2]-\mathbb{E}[h_2]}{\mathbb{E}[h_1]+\mathbb{E}[h_2]+r}=0, E [ h 2 ] = E [ h 1 ] + E [ h 2 ] + r E [ h 2 ] − E [ h 2 ] = 0 ,
i.e. any function in the localized class S ~ r , 2 ( Ω ) \widetilde{\mathcal{S}}_{r,2}(\Omega) S r , 2 ( Ω ) is of zero mean. Moreover, we take σ 2 2 = sup h ~ 2 ∈ S ~ r , 2 ( Ω ) E [ h ~ 2 2 ] \sigma_2^2=\sup_{\widetilde{h}_2\in\widetilde{\mathcal{S}}_{r,2}(\Omega)}\mathbb{E}[\widetilde{h}_2^2] σ 2 2 = sup h 2 ∈ S r , 2 ( Ω ) E [ h 2 2 ] to be the upper bound on the second moment of functions in S ~ r , 2 ( Ω ) . N o w w e h a v e v e r i f i e d t h a t a n y f u n c t i o n \widetilde{\mathcal{S}}_{r,2}(\Omega). Now we have verified that any function S r , 2 ( Ω ) . N o w w e ha v e v er i f i e d t ha t an y f u n c t i o n \widetilde{h}_2\in\widetilde{\mathcal{S}}_{r,2}(\Omega)$ satisfies all the required conditions. By taking μ \mu μ to be the uniform distribution on the domain Ω \Omega Ω and applying Talagrand’s Concentration inequality given in Lemma A.6, we have:
P x ′ [ sup h ~ 2 ∈ S ~ r , 2 ( Ω ) 1 n ∑ j = 1 n h ~ 2 ( x j ′ ) ≥ 2 sup h ~ 2 ∈ S ~ r , 2 ( Ω ) E y ′ [ 1 n ∑ j = 1 n h ~ 2 ( y j ′ ) ] + 2 t σ 2 2 n + 2 t β 2 n ] ≤ e − t . (96) \mathbb{P}_{x'}\left[\sup_{\widetilde{h}_2\in\widetilde{\mathcal{S}}_{r,2}(\Omega)}\frac{1}{n}\sum_{j=1}^{n}\widetilde{h}_2(x'_j)\geq2\sup_{\widetilde{h}_2\in\widetilde{\mathcal{S}}_{r,2}(\Omega)}\mathbb{E}_{y'}\left[\frac{1}{n}\sum_{j=1}^{n}\widetilde{h}_2(y'_j)\right]+\sqrt{\frac{2t\sigma_2^2}{n}+\frac{2t\beta_2}{n}}\right]\leq e^{-t}. \tag*{(96)} P x ′ [ h 2 ∈ S r , 2 ( Ω ) sup n 1 j = 1 ∑ n h 2 ( x j ′ ) ≥ 2 h 2 ∈ S r , 2 ( Ω ) sup E y ′ [ n 1 j = 1 ∑ n h 2 ( y j ′ ) ] + n 2 t σ 2 2 + n 2 t β 2 ] ≤ e − t . (96)
By applying a union bound to the two inequalities derived in A.55 and A.56, we can derive that with probability at least 1 − 2 e − t 1-2e^{-t} 1 − 2 e − t , the inequality below holds:
1 N ∑ i = 1 N h ~ 1 ( x i ′ ) + 1 n ∑ j = 1 n h ~ 2 ( x j ) ≤ sup h ~ 1 ∈ S ~ r , 1 ( Ω ) 1 N ∑ i = 1 N h ~ 1 ( x i ) + sup h ~ 2 ∈ S ~ r , 2 ( Ω ) 1 n ∑ j = 1 n h ~ 2 ( x j ′ ) ≤ 2 sup h ~ 1 ∈ S ~ r , 1 ( Ω ) E y [ 1 N ∑ i = 1 N h ~ 1 ( y i ) ] + 2 t σ 1 2 N + 2 t β 1 N + 2 sup h ~ 2 ∈ S ~ r , 2 ( Ω ) E y ′ [ 1 n ∑ j = 1 n h ~ 2 ( y j ′ ) ] + 2 t σ 2 2 n + 2 t β 2 n ≤ 16 ϕ ( r ) r + 2 t n ( σ 1 + σ 2 ) + 2 t ( β 1 + β 2 ) n . \begin{aligned}
\frac{1}{N}\sum_{i=1}^{N}\tilde{h}_1(x_i')+\frac{1}{n}\sum_{j=1}^{n}\tilde{h}_2(x_j)
&\leq\sup_{\tilde{h}_1\in\tilde{\mathcal{S}}_{r,1}(\Omega)}\frac{1}{N}\sum_{i=1}^{N}\tilde{h}_1(x_i)+\sup_{\tilde{h}_2\in\tilde{\mathcal{S}}_{r,2}(\Omega)}\frac{1}{n}\sum_{j=1}^{n}\tilde{h}_2(x_j')\\
&\leq2\sup_{\tilde{h}_1\in\tilde{\mathcal{S}}_{r,1}(\Omega)}\mathbb{E}_{y}\left[\frac{1}{N}\sum_{i=1}^{N}\tilde{h}_1(y_i)\right]+\sqrt{\frac{2t\sigma_1^2}{N}+\frac{2t\beta_1}{N}}\\
&\quad+2\sup_{\tilde{h}_2\in\tilde{\mathcal{S}}_{r,2}(\Omega)}\mathbb{E}_{y'}\left[\frac{1}{n}\sum_{j=1}^{n}\tilde{h}_2(y_j')\right]+\sqrt{\frac{2t\sigma_2^2}{n}+\frac{2t\beta_2}{n}}\\
&\leq\frac{16\phi(r)}{r}+\sqrt{\frac{2t}{n}}(\sigma_1+\sigma_2)+\frac{2t(\beta_1+\beta_2)}{n}.
\end{aligned} N 1 i = 1 ∑ N h ~ 1 ( x i ′ ) + n 1 j = 1 ∑ n h ~ 2 ( x j ) ≤ h ~ 1 ∈ S ~ r , 1 ( Ω ) sup N 1 i = 1 ∑ N h ~ 1 ( x i ) + h ~ 2 ∈ S ~ r , 2 ( Ω ) sup n 1 j = 1 ∑ n h ~ 2 ( x j ′ ) ≤ 2 h ~ 1 ∈ S ~ r , 1 ( Ω ) sup E y [ N 1 i = 1 ∑ N h ~ 1 ( y i ) ] + N 2 t σ 1 2 + N 2 t β 1 + 2 h ~ 2 ∈ S ~ r , 2 ( Ω ) sup E y ′ [ n 1 j = 1 ∑ n h ~ 2 ( y j ′ ) ] + n 2 t σ 2 2 + n 2 t β 2 ≤ r 16 ϕ ( r ) + n 2 t ( σ 1 + σ 2 ) + n 2 t ( β 1 + β 2 ) .
By the definition of β 1 \beta_1 β 1 and β 2 \beta_2 β 2 , we have that the term 2 t ( β 1 + β 2 ) n \frac{2t(\beta_1+\beta_2)}{n} n 2 t ( β 1 + β 2 ) can be upper bounded by:
2 t ( β 1 + β 2 ) n = 4 t ( M 1 + M 2 ) n r ≤ 4 ( V max + 3 ) C 2 t n r . \frac{2t(\beta_1+\beta_2)}{n}=\frac{4t(M_1+M_2)}{nr}\leq\frac{4(V_{\max}+3)C^2t}{nr}. n 2 t ( β 1 + β 2 ) = n r 4 t ( M 1 + M 2 ) ≤ n r 4 ( V m a x + 3 ) C 2 t .
Now we will derive some upper bound on the sum σ 1 + σ 2 \sigma_1+\sigma_2 σ 1 + σ 2 . By definition we have that:
( σ 1 + σ 2 ) 2 ≤ 2 ( σ 1 2 + σ 2 2 ) = 2 [ sup h ~ 1 ∈ S ~ r , 1 ( Ω ) E [ h ~ 1 2 ] + sup h ~ 2 ∈ S ~ r , 2 ( Ω ) E [ h ~ 2 2 ] ] = 2 [ sup h ∈ S ( Ω ) E [ h 1 2 ] − E [ h 1 ] 2 ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 + sup h ∈ S ( Ω ) E [ h 2 2 ] − E [ h 2 ] 2 ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 ] ≤ 4 sup h ∈ S ( Ω ) E [ h 1 2 ] + E [ h 2 2 ] ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 . \begin{aligned}
(\sigma_1+\sigma_2)^2
&\leq2(\sigma_1^2+\sigma_2^2)=2\left[\sup_{\tilde{h}_1\in\tilde{\mathcal{S}}_{r,1}(\Omega)}\mathbb{E}[\tilde{h}_1^2]+\sup_{\tilde{h}_2\in\tilde{\mathcal{S}}_{r,2}(\Omega)}\mathbb{E}[\tilde{h}_2^2]\right]\\
&=2\left[\sup_{h\in\mathcal{S}(\Omega)}\frac{\mathbb{E}[h_1^2]-\mathbb{E}[h_1]^2}{|\mathbb{E}[h_1]+\mathbb{E}[h_2]+r|^2}+\sup_{h\in\mathcal{S}(\Omega)}\frac{\mathbb{E}[h_2^2]-\mathbb{E}[h_2]^2}{|\mathbb{E}[h_1]+\mathbb{E}[h_2]+r|^2}\right]\\
&\leq4\sup_{h\in\mathcal{S}(\Omega)}\frac{\mathbb{E}[h_1^2]+\mathbb{E}[h_2^2]}{|\mathbb{E}[h_1]+\mathbb{E}[h_2]+r|^2}.
\end{aligned} ( σ 1 + σ 2 ) 2 ≤ 2 ( σ 1 2 + σ 2 2 ) = 2 [ h ~ 1 ∈ S ~ r , 1 ( Ω ) sup E [ h ~ 1 2 ] + h ~ 2 ∈ S ~ r , 2 ( Ω ) sup E [ h ~ 2 2 ] ] = 2 [ h ∈ S ( Ω ) sup ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 E [ h 1 2 ] − E [ h 1 ] 2 + h ∈ S ( Ω ) sup ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 E [ h 2 2 ] − E [ h 2 ] 2 ] ≤ 4 h ∈ S ( Ω ) sup ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 E [ h 1 2 ] + E [ h 2 2 ] .
Now it suffices to derive an upper bound of E [ h 1 2 ] + E [ h 2 2 ] ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 \frac{\mathbb{E}[h_1^2]+\mathbb{E}[h_2^2]}{|\mathbb{E}[h_1]+\mathbb{E}[h_2]+r|^2} ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 E [ h 1 2 ] + E [ h 2 2 ] for any h ∈ S ( Ω ) h\in\mathcal{S}(\Omega) h ∈ S ( Ω ) . The existence of such an upper bound is guaranteed because of the regularity results of the PDE. We aim to show that there exist some constants α , α ′ > 0 \alpha,\alpha'>0 α , α ′ > 0 , such that for any h ∈ S ( Ω ) h\in\mathcal{S}(\Omega) h ∈ S ( Ω ) , the following inequality holds:
α ( E [ h 1 2 ] + E [ h 2 2 ] ) ≤ ∥ u − u ∗ ∥ H 1 ( Ω ) 2 ≤ α ′ ( E [ h 1 ] + E [ h 2 ] ) . (97) \alpha(\mathbb{E}[h_1^2]+\mathbb{E}[h_2^2])\leq\lVert u-u^*\rVert_{H^1(\Omega)}^2\leq\alpha'(\mathbb{E}[h_1]+\mathbb{E}[h_2]). \tag*{(97)} α ( E [ h 1 2 ] + E [ h 2 2 ]) ≤ ∥ u − u ∗ ∥ H 1 ( Ω ) 2 ≤ α ′ ( E [ h 1 ] + E [ h 2 ]) . (97)
The RHS of the inequality follows from strong convexity of the DRM objective function proved in Theorem A.1 :
E [ h 1 ] + E [ h 2 ] = E ( u ) − E ( u ∗ ) ≥ min { 1 , V min } 4 ∥ u − u ∗ ∥ H 1 ( Ω ) 2 . \mathbb{E}[h_1]+\mathbb{E}[h_2]=\mathcal{E}(u)-\mathcal{E}(u^*)\geq\frac{\min\{1,V_{\min}\}}{4}\lVert u-u^*\rVert_{H^1(\Omega)}^2. E [ h 1 ] + E [ h 2 ] = E ( u ) − E ( u ∗ ) ≥ 4 min { 1 , V m i n } ∥ u − u ∗ ∥ H 1 ( Ω ) 2 .
The LHS of the inequality follows from boundedness condition listed in equation 5.4 and the QM-AM inequality:
E [ h 1 2 ] + E [ h 2 2 ] = ∫ Ω 1 4 ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) 2 d x + ∫ Ω [ 1 2 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] 2 d x ≤ 1 4 ∫ Ω ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) 2 d x + 1 2 ∫ Ω V 2 ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) 2 d x + 2 ∫ Ω f 2 ( u − u ∗ ) 2 d x ≤ 1 4 ∫ Ω ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ 2 ( ∥ ∇ u ∥ + ∥ ∇ u ∗ ∥ ) 2 d x + 1 2 V max 2 ∫ Ω ∣ ∣ u ∣ − ∣ u ∗ ∣ ∣ 2 ( ∣ u ∣ + ∣ u ∗ ∣ ) 2 d x + 2 C 2 ∫ Ω ( u − u ∗ ) 2 d x ≤ C 2 ∫ Ω ∥ ∇ u − ∇ u ∗ ∥ 2 d x + 2 C 2 ( 1 + V max 2 ) ∫ Ω ∣ u − u ∗ ∣ 2 d x ≤ 2 C 2 ( 1 + V max 2 ) ∥ u − u ∗ ∥ H 1 ( Ω ) 2 . \begin{aligned}
\mathbb{E}[h_1^2]+\mathbb{E}[h_2^2]
&= \int_{\Omega} \frac{1}{4}\left(\lVert\nabla u\rVert^2-\lVert\nabla u^*\rVert^2\right)^2\,dx
+\int_{\Omega}\left[\frac{1}{2}V\left(\lvert u\rvert^2-\lvert u^*\rvert^2\right)-f\left(u-u^*\right)\right]^2\,dx\\
&\leq\frac{1}{4}\int_{\Omega}\left(\lVert\nabla u\rVert^2-\lVert\nabla u^*\rVert^2\right)^2\,dx
+\frac{1}{2}\int_{\Omega}V^2\left(\lvert u\rvert^2-\lvert u^*\rvert^2\right)^2\,dx
+2\int_{\Omega}f^2\left(u-u^*\right)^2\,dx\\
&\leq\frac{1}{4}\int_{\Omega}\left\lVert\lVert\nabla u\rVert-\lVert\nabla u^*\rVert\right\rVert^2\left(\lVert\nabla u\rVert+\lVert\nabla u^*\rVert\right)^2\,dx
+\frac{1}{2}V_{\max}^2\int_{\Omega}\left\lvert\lvert u\rvert-\lvert u^*\rvert\right\rvert^2\left(\lvert u\rvert+\lvert u^*\rvert\right)^2\,dx\\
&\qquad+2C^2\int_{\Omega}\left(u-u^*\right)^2\,dx
\leq C^2\int_{\Omega}\lVert\nabla u-\nabla u^*\rVert^2\,dx
+2C^2\left(1+V_{\max}^2\right)\int_{\Omega}\lvert u-u^*\rvert^2\,dx\\
&\leq2C^2\left(1+V_{\max}^2\right)\lVert u-u^*\rVert_{H^1(\Omega)}^2.
\end{aligned} E [ h 1 2 ] + E [ h 2 2 ] = ∫ Ω 4 1 ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) 2 d x + ∫ Ω [ 2 1 V ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) − f ( u − u ∗ ) ] 2 d x ≤ 4 1 ∫ Ω ( ∥ ∇ u ∥ 2 − ∥ ∇ u ∗ ∥ 2 ) 2 d x + 2 1 ∫ Ω V 2 ( ∣ u ∣ 2 − ∣ u ∗ ∣ 2 ) 2 d x + 2 ∫ Ω f 2 ( u − u ∗ ) 2 d x ≤ 4 1 ∫ Ω ∥ ∥ ∇ u ∥ − ∥ ∇ u ∗ ∥ ∥ 2 ( ∥ ∇ u ∥ + ∥ ∇ u ∗ ∥ ) 2 d x + 2 1 V m a x 2 ∫ Ω ∣ ∣ u ∣ − ∣ u ∗ ∣ ∣ 2 ( ∣ u ∣ + ∣ u ∗ ∣ ) 2 d x + 2 C 2 ∫ Ω ( u − u ∗ ) 2 d x ≤ C 2 ∫ Ω ∥ ∇ u − ∇ u ∗ ∥ 2 d x + 2 C 2 ( 1 + V m a x 2 ) ∫ Ω ∣ u − u ∗ ∣ 2 d x ≤ 2 C 2 ( 1 + V m a x 2 ) ∥ u − u ∗ ∥ H 1 ( Ω ) 2 .
By picking α ′ = 4 min { 1 , V min } \alpha'=\frac{4}{\min\{1,V_{\min}\}} α ′ = m i n { 1 , V m i n } 4 and α = 1 2 C 2 ( 1 + V max 2 ) \alpha=\frac{1}{2C^2(1+V_{\max}^2)} α = 2 C 2 ( 1 + V m a x 2 ) 1 , we have finished proving inequality A.57.
Then we can can upper bound the term E [ h 1 2 ] + E [ h 2 2 ] ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 \frac{\mathbb{E}[h_1^2]+\mathbb{E}[h_2^2]}{\lvert\mathbb{E}[h_1]+\mathbb{E}[h_2]+r\rvert^2} ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 E [ h 1 2 ] + E [ h 2 2 ] as:
E [ h 1 2 ] + E [ h 2 2 ] ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 ≤ α ′ α ( E [ h 1 ] + E [ h 2 ] ) 2 r ( E [ h 1 ] + E [ h 2 ] ) ≤ α ′ 2 α r . \frac{\mathbb{E}[h_1^2]+\mathbb{E}[h_2^2]}{\lvert\mathbb{E}[h_1]+\mathbb{E}[h_2]+r\rvert^2}
\leq\frac{\frac{\alpha'}{\alpha}\left(\mathbb{E}[h_1]+\mathbb{E}[h_2]\right)}{2r\left(\mathbb{E}[h_1]+\mathbb{E}[h_2]\right)}
\leq\frac{\alpha'}{2\alpha r}. ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 E [ h 1 2 ] + E [ h 2 2 ] ≤ 2 r ( E [ h 1 ] + E [ h 2 ] ) α α ′ ( E [ h 1 ] + E [ h 2 ] ) ≤ 2 α r α ′ .
Combining the bounds derived above helps us upper bound the term 2 t n ( σ 1 + σ 2 ) \sqrt{\frac{2t}{n}}(\sigma_1+\sigma_2) n 2 t ( σ 1 + σ 2 ) as below:
2 t n ( σ 1 + σ 2 ) ≤ 8 t n sup h ∈ S ( Ω ) E [ h 1 2 ] + E [ h 2 2 ] ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 ≤ 4 α ′ t n α r . \sqrt{\frac{2t}{n}}(\sigma_1+\sigma_2)
\leq\sqrt{\frac{8t}{n}}\sqrt{\sup_{h\in S(\Omega)}\frac{\mathbb{E}[h_1^2]+\mathbb{E}[h_2^2]}{\lvert\mathbb{E}[h_1]+\mathbb{E}[h_2]+r\rvert^2}}
\leq\sqrt{\frac{4\alpha't}{n\alpha r}}. n 2 t ( σ 1 + σ 2 ) ≤ n 8 t h ∈ S ( Ω ) sup ∣ E [ h 1 ] + E [ h 2 ] + r ∣ 2 E [ h 1 2 ] + E [ h 2 2 ] ≤ n α r 4 α ′ t .
Thus, using the two upper bounds on 2 t n ( σ 1 + σ 2 ) \sqrt{\frac{2t}{n}}(\sigma_1+\sigma_2) n 2 t ( σ 1 + σ 2 ) and 2 t ( β 1 + β 2 ) n \frac{2t(\beta_1+\beta_2)}{n} n 2 t ( β 1 + β 2 ) , we have
1 N ∑ i = 1 N h ~ ( x i ′ ) + 1 n ∑ j = 1 n h ~ ( x j ) ≤ 16 ϕ ( r ) r + 2 t n ( σ 1 + σ 2 ) + 2 t ( β 1 + β 2 ) n ≤ 16 ϕ ( r ) r + 4 α ′ t n α r + 4 ( V max + 3 ) C 2 t n r = ψ ( r ) \begin{aligned}
\frac{1}{N}\sum_{i=1}^{N}\widetilde{h}(x_i')+\frac{1}{n}\sum_{j=1}^{n}\widetilde{h}(x_j)
\leq\frac{16\phi(r)}{r}+\sqrt{\frac{2t}{n}}(\sigma_1+\sigma_2)+\frac{2t(\beta_1+\beta_2)}{n} \\
&\leq\frac{16\phi(r)}{r}+\sqrt{\frac{4\alpha't}{n\alpha r}}+\frac{4(V_{\max}+3)C^2t}{nr}=\psi(r)
\end{aligned} N 1 i = 1 ∑ N h ( x i ′ ) + n 1 j = 1 ∑ n h ( x j ) ≤ r 16 ϕ ( r ) + n 2 t ( σ 1 + σ 2 ) + n 2 t ( β 1 + β 2 ) ≤ r 16 ϕ ( r ) + n α r 4 α ′ t + n r 4 ( V m a x + 3 ) C 2 t = ψ ( r )
Let’s pick the critical radius r 0 r_0 r 0 to be:
r 0 = max { 2 14 r ∗ , 24 M t n , 144 α ′ t α n } . (98) r_0=\max\left\{2^{14}r^*,\frac{24Mt}{n},\frac{144\alpha't}{\alpha n}\right\}. \tag*{(98)} r 0 = max { 2 14 r ∗ , n 24 M t , α n 144 α ′ t } . (98)
Note that concavity of the function ϕ \phi ϕ implies that ϕ ( r ) ≤ r \phi(r)\leq r ϕ ( r ) ≤ r for any r ≥ r ∗ r\geq r^* r ≥ r ∗ . Combining this with the first inequality listed in 5.5 5.5 5.5 yields:
16 ϕ ( r 0 ) r 0 ≤ 2 11 ϕ ( r 0 2 14 ) 2 14 r 0 2 14 = 1 8 × ϕ ( r 0 2 14 ) r 0 2 14 ≤ 1 8 . \frac{16\phi(r_0)}{r_0}
\leq\frac{2^{11}\phi\left(\frac{r_0}{2^{14}}\right)}{2^{14}\frac{r_0}{2^{14}}}
=\frac{1}{8}\times\frac{\phi\left(\frac{r_0}{2^{14}}\right)}{\frac{r_0}{2^{14}}}
\leq\frac{1}{8}. r 0 16 ϕ ( r 0 ) ≤ 2 14 2 14 r 0 2 11 ϕ ( 2 14 r 0 ) = 8 1 × 2 14 r 0 ϕ ( 2 14 r 0 ) ≤ 8 1 .
On the other hand, applying equation A.58 yields:
4 α ′ t n α r 0 ≤ 4 α ′ t n α α n 144 α ′ t = 1 6 , \sqrt{\frac{4\alpha't}{n\alpha r_0}} \le\sqrt{\frac{4\alpha't}{n\alpha}\frac{\alpha n}{144\alpha't}} = \frac{1}{6}, n α r 0 4 α ′ t ≤ n α 4 α ′ t 144 α ′ t α n = 6 1 ,
4 ( V max + 3 ) C 2 t n r 0 ≤ 4 ( V max + 3 ) C 2 t n × n 24 ( V max + 3 ) C 2 t = 1 6 . \frac{4(V_{\max}+3)C^{2}t}{nr_0} \le\frac{4(V_{\max}+3)C^{2}t}{n}\times\frac{n}{24(V_{\max}+3)C^{2}t} = \frac{1}{6}. n r 0 4 ( V m a x + 3 ) C 2 t ≤ n 4 ( V m a x + 3 ) C 2 t × 24 ( V m a x + 3 ) C 2 t n = 6 1 .
Summing the three inequalities above implies:
ψ ( r 0 ) = 16 ϕ ( r 0 ) r 0 + 4 α ′ t n α r 0 + 4 ( V max + 3 ) C 2 t n r 0 ≤ 1 8 + 1 6 + 1 6 < 1 2 . \psi(r_0)=\frac{16\phi(r_0)}{r_0}+\sqrt{\frac{4\alpha't}{n\alpha r_0}}+\frac{4(V_{\max}+3)C^{2}t}{nr_0}\le\frac{1}{8}+\frac{1}{6}+\frac{1}{6}<\frac{1}{2}. ψ ( r 0 ) = r 0 16 ϕ ( r 0 ) + n α r 0 4 α ′ t + n r 0 4 ( V m a x + 3 ) C 2 t ≤ 8 1 + 6 1 + 6 1 < 2 1 .
By picking r = r 0 r=r_0 r = r 0 , we can further deduce that for any function u ∈ F ( Ω ) u\in\mathcal{F}(\Omega) u ∈ F ( Ω ) , the following inequality holds with probability 1 − e − t 1-e^{-t} 1 − e − t .
E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) E ( u ) − E ( u ∗ ) + r 0 = 1 n ∑ i = 1 n h ~ ( x i ) ≤ ψ ( r 0 ) < 1 2 . \frac{\mathcal{E}(u)-\mathcal{E}(u^*)-\mathcal{E}_n(u)+\mathcal{E}_n(u^*)}{\mathcal{E}(u)-\mathcal{E}(u^*)+r_0}
=\frac{1}{n}\sum_{i=1}^{n}\widetilde{h}(x_i)\le\psi(r_0)<\frac{1}{2}. E ( u ) − E ( u ∗ ) + r 0 E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) = n 1 i = 1 ∑ n h ( x i ) ≤ ψ ( r 0 ) < 2 1 .
Multiplying the denominator on both sides indicates:
Δ E g e n = E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) ≤ 1 2 [ E ( u ) − E ( u ∗ ) ] + 1 2 r 0 = 1 2 Δ E ( n ) + 1 2 r 0 . \Delta\mathcal{E}_{\mathrm{gen}}=\mathcal{E}(u)-\mathcal{E}(u^*)-\mathcal{E}_n(u)+\mathcal{E}_n(u^*)\le\frac{1}{2}\left[\mathcal{E}(u)-\mathcal{E}(u^*)\right]+\frac{1}{2}r_0=\frac{1}{2}\Delta\mathcal{E}^{(n)}+\frac{1}{2}r_0. Δ E gen = E ( u ) − E ( u ∗ ) − E n ( u ) + E n ( u ∗ ) ≤ 2 1 [ E ( u ) − E ( u ∗ ) ] + 2 1 r 0 = 2 1 Δ E ( n ) + 2 1 r 0 .
Substituting the upper bound above into the decomposition Δ E ( n ) ≤ Δ E g e n + 2 Δ E a p p + 4 t n \Delta\mathcal{E}^{(n)}\le\Delta\mathcal{E}_{\mathrm{gen}}+2\Delta\mathcal{E}_{\mathrm{app}}+\frac{4t}{n} Δ E ( n ) ≤ Δ E gen + 2Δ E app + n 4 t yields that with probability 1 − 3 e − min { 1 , C q } t 1-3e^{-\min\{1,C_q\}t} 1 − 3 e − m i n { 1 , C q } t , we have:
Δ E ( n ) ≤ Δ E g e n + 3 2 Δ E a p p + t 2 n ≤ 1 2 Δ E ( n ) + 1 2 r 0 + 3 2 Δ E a p p + t 2 n . \Delta\mathcal{E}^{(n)}\le\Delta\mathcal{E}_{\mathrm{gen}}+\frac{3}{2}\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{2n}\le\frac{1}{2}\Delta\mathcal{E}^{(n)}+\frac{1}{2}r_0+\frac{3}{2}\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{2n}. Δ E ( n ) ≤ Δ E gen + 2 3 Δ E app + 2 n t ≤ 2 1 Δ E ( n ) + 2 1 r 0 + 2 3 Δ E app + 2 n t .
Simplifying the inequality above yields that with probability 1 − 3 e − min { 1 , C q } t 1-3e^{-\min\{1,C_q\}t} 1 − 3 e − m i n { 1 , C q } t , we have:
Δ E ( n ) ≤ r 0 + 3 Δ E a p p + t n = 3 inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { 2 14 r ∗ , 24 M t n , 36 α ′ t α n } + t n \Delta\mathcal{E}^{(n)}\le r_0+3\Delta\mathcal{E}_{\mathrm{app}}+\frac{t}{n}
=3\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right)+\max\left\{2^{14}r_*,24M\frac{t}{n},\frac{36\alpha't}{\alpha n}\right\}+\frac{t}{n} Δ E ( n ) ≤ r 0 + 3Δ E app + n t = 3 u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { 2 14 r ∗ , 24 M n t , α n 36 α ′ t } + n t
≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \lesssim\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right)+\max\left\{r_*,\frac{t}{n}\right\}. ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Moreover, using strong convexity of the DRM objective function proved in Theorem A.1 implies:
Δ E ( n ) = E ( u ^ M D R M ) − E ( u ∗ ) ≥ ∥ u ^ M D R M − u ∗ ∥ H 1 ( Ω ) 2 . \Delta\mathcal{E}^{(n)}=\mathcal{E}(\widehat{u}_{\mathrm{MDRM}})-\mathcal{E}(u^*)\ge\left\|\widehat{u}_{\mathrm{MDRM}}-u^*\right\|_{H^1(\Omega)}^2. Δ E ( n ) = E ( u MDRM ) − E ( u ∗ ) ≥ ∥ u MDRM − u ∗ ∥ H 1 ( Ω ) 2 .
Combining the two bounds above yields that with probability 1 − 3 e − min { 1 , C q } t 1-3e^{-\min\{1,C_q\}t} 1 − 3 e − m i n { 1 , C q } t , we have:
∥ u ^ M D R M − u ∗ ∥ H 1 ( Ω ) 2 ≲ inf u F ∈ F ( Ω ) ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , t n } . \left\|\widehat{u}_{\mathrm{MDRM}}-u^*\right\|_{H^1(\Omega)}^2\lesssim\inf_{u_{\mathcal{F}}\in\mathcal{F}(\Omega)}\left(\mathcal{E}(u_{\mathcal{F}})-\mathcal{E}(u^*)\right)+\max\left\{r_*,\frac{t}{n}\right\}. ∥ u MDRM − u ∗ ∥ H 1 ( Ω ) 2 ≲ u F ∈ F ( Ω ) inf ( E ( u F ) − E ( u ∗ ) ) + max { r ∗ , n t } .
Intuition Behind the Sub-optimality of the Unmodified Deep Ritz Methods
In this section, we aim to discuss the intuition behind the sub-optimality of the unmodified DRM via using the truncation Fourier basis. To simplify the notation, in this section we consider the following simplest Poisson equation Δ u = f \Delta u=f Δ u = f on the hypercube with zero Dirichlet boundary condition. To illustrate the necessity of the modification we made, we consider the difference between the following two estimators
Estimator 1. We use the truncated Fourier basis estimator to learn the right hand side function f f f and then we invert the PDE exactly to get the estimated u u u .
Estimator 2. We plug in a parametrization of the truncated Fourier basis into the empirical DRM objective
We would like to point out that estimator 1 isn’t build for computational consideration. Instead, we use it to consider the statistical limit of our sampled data. We first show that the estimator 1 can achieve the minimax optimal estimation error.
Error Of Estimator 1. Firstly, we show that if one wants to learn the function u u u in H 0 1 H_0^1 H 0 1 norm, one need to learn the right hand side function f f f in H 0 − 1 H_0^{-1} H 0 − 1 norm. The H 0 − 1 H_0^{-1} H 0 − 1 norm is defined as the dual norm of the H 1 H^1 H 1 norm, i.e. ∥ u ∥ H 0 − 1 = max ∥ v ∥ H 0 1 ≤ 1 ⟨ u , v ⟩ \|u\|_{H_0^{-1}}=\max_{\|v\|_{H_0^1}\leq1}\langle u,v\rangle ∥ u ∥ H 0 − 1 = max ∥ v ∥ H 0 1 ≤ 1 ⟨ u , v ⟩ . Once we assume we have an estimate f ^ \hat{f} f ^ of f f f in H 0 − 1 H_0^{-1} H 0 − 1 , we can have an estimate of u u u via u ^ : = ( Δ ) − 1 f ^ \hat{u}:=(\Delta)^{-1}\hat{f} u ^ := ( Δ ) − 1 f ^ , whose distance to u u u in the H 1 H^1 H 1 norm satisfies:
∥ ∇ u − ∇ u ^ ∥ H 0 1 = max ∥ v ∥ H 0 1 ≤ 1 ⟨ ∇ u − ∇ u ^ , ∇ v ⟩ = max ∥ v ∥ H 0 1 ≤ 1 ⟨ Δ u − Δ u ^ , v ⟩ = max ∥ v ∥ H 0 1 ≤ 1 ⟨ f − f ^ , v ⟩ = ∥ f − f ^ ∥ H 0 − 1 . \begin{aligned}
\|\nabla u-\nabla\hat{u}\|_{H_0^1}
&=\max_{\|v\|_{H_0^1}\leq1}\langle\nabla u-\nabla\hat{u},\nabla v\rangle\\
&=\max_{\|v\|_{H_0^1}\leq1}\langle\Delta u-\Delta\hat{u},v\rangle\\
&=\max_{\|v\|_{H_0^1}\leq1}\langle f-\hat{f},v\rangle
=\|f-\hat{f}\|_{H_0^{-1}}.
\end{aligned} ∥∇ u − ∇ u ^ ∥ H 0 1 = ∥ v ∥ H 0 1 ≤ 1 max ⟨ ∇ u − ∇ u ^ , ∇ v ⟩ = ∥ v ∥ H 0 1 ≤ 1 max ⟨ Δ u − Δ u ^ , v ⟩ = ∥ v ∥ H 0 1 ≤ 1 max ⟨ f − f ^ , v ⟩ = ∥ f − f ^ ∥ H 0 − 1 .
Estimator 1 using the truncated fourier estimator to estimate the right hand side function f f f . Suppose we can access a random sample of observed data as { x i , f ( x i ) } i = 1 n \{x_i,f(x_i)\}_{i=1}^n { x i , f ( x i ) } i = 1 n , then the Fourier coefficient f z : = ⟨ u , ϕ z ⟩ f_z:=\langle u,\phi_z\rangle f z := ⟨ u , ϕ z ⟩ can be estimated as f ^ z : = 1 n ∑ i = 1 n f ( x i ) ϕ z ( x i ) \hat{f}_z:=\frac{1}{n}\sum_{i=1}^n f(x_i)\phi_z(x_i) f ^ z := n 1 ∑ i = 1 n f ( x i ) ϕ z ( x i ) . To bound the estimation error of f ^ : = ∑ ∥ z ∥ ∞ ≤ Z f ^ z ϕ z \hat{f}:=\sum_{\|z\|_\infty\leq Z}\hat{f}_z\phi_z f ^ := ∑ ∥ z ∥ ∞ ≤ Z f ^ z ϕ z in H 0 − 1 H_0^{-1} H 0 − 1 , we first apply the bias-variance decomposition:
E ∥ f ^ − f ∥ H 0 − 1 2 ≤ ∥ E f ^ − f ∥ H 0 − 1 2 + E ∥ f − E f ^ ∥ H 0 − 1 2 \mathbb{E}\|\hat{f}-f\|_{H_0^{-1}}^2\leq\|\mathbb{E}\hat{f}-f\|_{H_0^{-1}}^2+\mathbb{E}\|f-\mathbb{E}\hat{f}\|_{H_0^{-1}}^2 E ∥ f ^ − f ∥ H 0 − 1 2 ≤ ∥ E f ^ − f ∥ H 0 − 1 2 + E ∥ f − E f ^ ∥ H 0 − 1 2
We first bound the bias term ∥ E f ^ − f ∥ H 0 − 1 2 \|\mathbb{E}\hat{f}-f\|_{H_0^{-1}}^2 ∥ E f ^ − f ∥ H 0 − 1 2 . Given E f ^ = ∑ ∥ z ∥ ∞ ≤ Z f z ϕ z \mathbb{E}\hat{f}=\sum_{\|z\|_\infty\leq Z}f_z\phi_z E f ^ = ∑ ∥ z ∥ ∞ ≤ Z f z ϕ z , we have that for a truncation set Z : = { z ∈ N d ∣ ∥ z ∥ ∞ ≤ Z } \mathcal{Z}:=\{z\in\mathbb{N}^d\mid\|z\|_\infty\leq Z\} Z := { z ∈ N d ∣ ∥ z ∥ ∞ ≤ Z } , the bias term can be controlled by:
∥ ∑ ∥ z ∥ ∞ > Z f z ϕ z ∥ H 0 − 1 2 ≤ C ∑ ∥ z ∥ ∞ > Z f z 2 z − 2 ≤ ∥ z ∥ ∞ − 2 ( s − 1 ) ∥ f ∥ H α − 2 2 \left\|\sum_{\|z\|_\infty>Z}f_z\phi_z\right\|_{H_0^{-1}}^2
\leq C\sum_{\|z\|_\infty>Z}f_z^2z^{-2}
\leq\|z\|_\infty^{-2(s-1)}\|f\|_{H^{\alpha-2}}^2 ∥ z ∥ ∞ > Z ∑ f z ϕ z H 0 − 1 2 ≤ C ∥ z ∥ ∞ > Z ∑ f z 2 z − 2 ≤ ∥ z ∥ ∞ − 2 ( s − 1 ) ∥ f ∥ H α − 2 2
Next we estimate the variance of the estimator by decomposing the variance into the following sum:
E ∥ f − f ^ ∥ H 0 − 1 2 ≤ E ∑ ∥ z ∥ ∞ ≤ Z ( f ^ z − f z ) 2 ∥ ϕ z ∥ H 0 − 1 2 ≤ ∑ ∥ z ∥ ∞ ≤ Z ∣ z ∣ − 1 Var ( f ^ z ) . \mathbb{E}\|f-\hat{f}\|_{H_0^{-1}}^2
\leq\mathbb{E}\sum_{\|z\|_\infty\leq Z}(\hat{f}_z-f_z)^2\|\phi_z\|_{H_0^{-1}}^2
\leq\sum_{\|z\|_\infty\leq Z}|z|^{-1}\operatorname{Var}(\hat{f}_z). E ∥ f − f ^ ∥ H 0 − 1 2 ≤ E ∥ z ∥ ∞ ≤ Z ∑ ( f ^ z − f z ) 2 ∥ ϕ z ∥ H 0 − 1 2 ≤ ∥ z ∥ ∞ ≤ Z ∑ ∣ z ∣ − 1 Var ( f ^ z ) .
Finally we achieve a Z − 2 ( s − 1 ) + Z d − 2 n Z^{-2(s-1)}+\frac{Z^{d-2}}{n} Z − 2 ( s − 1 ) + n Z d − 2 upper bound for estimator 1. With optimal selection of Z Z Z , we can achieve the min-max optimal convergence rate n − 2 s − 2 d + 2 s − 4 n^{-\frac{2s-2}{d+2s-4}} n − d + 2 s − 4 2 s − 2 .
Difference Between Estimator 1 and Estimator 2. Next we aim to understand the Deep Ritz Method objective function via plugging in a truncated Fourier series estimator. We consider an estimator of the form u = ∑ z u ^ z ϕ z ( x ) u=\sum_z \hat{u}_z\phi_z(x) u = ∑ z u ^ z ϕ z ( x ) , which lies in the space of truncated fourier series. Then the empirical DRM objective function can be expressed as
1 2 n ∑ i = 1 n ( ∑ z u ^ z ∇ ϕ z ( x i ) ) 2 + ∑ z u ^ z ϕ z ( x i ) f ( x i ) . (99) \frac{1}{2n}\sum_{i=1}^{n}\left(\sum_z\hat{u}_z\nabla\phi_z(x_i)\right)^2+\sum_z\hat{u}_z\phi_z(x_i)f(x_i). \tag*{(99)} 2 n 1 i = 1 ∑ n ( z ∑ u ^ z ∇ ϕ z ( x i ) ) 2 + z ∑ u ^ z ϕ z ( x i ) f ( x i ) . (99)
We observe that (B.1) is a quadratic formula with respect to the Fourier coefficients u : = ( u z ) ∥ z ∥ ∞ ≤ Z \mathbf{u}:=(u_z)_{\lVert z\rVert_\infty\leq Z} u := ( u z ) ∥ z ∥ ∞ ≤ Z . Thus, we can rewrite it as the following matrix form
min 1 2 u T A ^ u + u T f ^ , where A ^ = ( 1 n ∑ i = 1 n ∇ ϕ i ( x i ) ∇ ϕ j ( x i ) ) ∥ i ∥ ∞ ≤ Z , ∥ j ∥ ∞ ≤ Z . (100) \min\frac{1}{2}\mathbf{u}^{\mathsf{T}}\hat{A}\mathbf{u}+\mathbf{u}^{\mathsf{T}}\hat{f},\quad\text{where }\hat{A}=\left(\frac{1}{n}\sum_{i=1}^{n}\nabla\phi_i(x_i)\nabla\phi_j(x_i)\right)_{\lVert i\rVert_\infty\leq Z,\lVert j\rVert_\infty\leq Z}. \tag*{(100)} min 2 1 u T A ^ u + u T f ^ , where A ^ = ( n 1 i = 1 ∑ n ∇ ϕ i ( x i ) ∇ ϕ j ( x i ) ) ∥ i ∥ ∞ ≤ Z , ∥ j ∥ ∞ ≤ Z . (100)
Based on the matrix formulation B.2, we can compare the solution given by the two estimators
u ^ 1 = diag ( ∥ z ∥ 2 ) ∥ z ∥ ∞ ≤ Z − 1 f ^ . (101) \hat{u}_1=\operatorname{diag}\left(\lVert z\rVert^2\right)_{\lVert z\rVert_\infty\leq Z}^{-1}\hat{f}. \tag*{(101)} u ^ 1 = diag ( ∥ z ∥ 2 ) ∥ z ∥ ∞ ≤ Z − 1 f ^ . (101)
u ^ 2 = A ^ − 1 f ^ . (102) \hat{u}_2=\hat{A}^{-1}\hat{f}. \tag*{(102)} u ^ 2 = A ^ − 1 f ^ . (102)
Note that E A ^ = ( ∥ z ∥ 2 ) ∥ z ∥ ∞ ≤ Z \mathbb{E}\hat{A}=\left(\lVert z\rVert^2\right)_{\lVert z\rVert_\infty\leq Z} E A ^ = ( ∥ z ∥ 2 ) ∥ z ∥ ∞ ≤ Z . Thus, we can further introduce another variance from the sampling of A A A . By directly estimating u ^ 1 − u ^ 2 \hat{u}_1-\hat{u}_2 u ^ 1 − u ^ 2 , we will show that this term will be larger than the final convergence rate. Notice that
∥ u ^ 1 − u ^ 2 ∥ H 1 2 = f ^ T ( ( E A ^ ) − 1 − A ^ − 1 ) T diag ( ∥ z ∥ 2 ) ∥ z ∥ ∞ ≤ Z ( ( E A ^ ) − 1 − A ^ − 1 ) f ^ (103) \lVert\hat{u}_1-\hat{u}_2\rVert_{H^1}^2=\hat{f}^{\mathsf{T}}\left((\mathbb{E}\hat{A})^{-1}-\hat{A}^{-1}\right)^{\mathsf{T}}\operatorname{diag}\left(\lVert z\rVert^2\right)_{\lVert z\rVert_\infty\leq Z}\left((\mathbb{E}\hat{A})^{-1}-\hat{A}^{-1}\right)\hat{f} \tag*{(103)} ∥ u ^ 1 − u ^ 2 ∥ H 1 2 = f ^ T ( ( E A ^ ) − 1 − A ^ − 1 ) T diag ( ∥ z ∥ 2 ) ∥ z ∥ ∞ ≤ Z ( ( E A ^ ) − 1 − A ^ − 1 ) f ^ (103)
Next we aim to bound ( E A ^ ) − 1 − A ^ − 1 (\mathbb{E}\hat{A})^{-1}-\hat{A}^{-1} ( E A ^ ) − 1 − A ^ − 1 . We first use the Matrix Bernstein Inequality[73 ] to bound the H 1 H^1 H 1 distance between u ^ 1 \hat{u}_1 u ^ 1 and u ^ 2 \hat{u}_2 u ^ 2 . According to the Matrix Bernstein Inequality, we have that with probability 1 − e − t 1-e^{-t} 1 − e − t , the following inequality holds
∥ ( ( E A ^ ) − A ^ ) ∥ H ≤ Z d n + t n . (104) \left\lVert\left((\mathbb{E}\hat{A})-\hat{A}\right)\right\rVert_H\leq\sqrt{\frac{Z^d}{n}}+\frac{t}{n}. \tag*{(104)} ( ( E A ^ ) − A ^ ) H ≤ n Z d + n t . (104)
where ∥ ⋅ ∥ H \lVert\cdot\rVert_H ∥ ⋅ ∥ H is the matrix operator norm respect to the vector ∥ ⋅ ∥ H \lVert\cdot\rVert_H ∥ ⋅ ∥ H defined as ∥ z ∥ H 2 = z T diag ( ∥ z ∥ 2 ) ∥ z ∥ ∞ ≤ Z − 1 z \lVert z\rVert_H^2=z^{\mathsf{T}}\operatorname{diag}\left(\lVert z\rVert^2\right)_{\lVert z\rVert_\infty\leq Z}^{-1}z ∥ z ∥ H 2 = z T diag ( ∥ z ∥ 2 ) ∥ z ∥ ∞ ≤ Z − 1 z . Note that
( I + ( E A ^ ) − 1 ( A ^ − ( E A ^ ) ) ) ( ( E A ^ ) − 1 − A ^ − 1 ) = ( E A ^ ) − 1 ( A ^ − ( E A ^ ) ) ( E A ^ ) − 1 (105) \left(I+(\mathbb{E}\hat{A})^{-1}\left(\hat{A}-(\mathbb{E}\hat{A})\right)\right)\left((\mathbb{E}\hat{A})^{-1}-\hat{A}^{-1}\right)=(\mathbb{E}\hat{A})^{-1}\left(\hat{A}-(\mathbb{E}\hat{A})\right)(\mathbb{E}\hat{A})^{-1} \tag*{(105)} ( I + ( E A ^ ) − 1 ( A ^ − ( E A ^ ) ) ) ( ( E A ^ ) − 1 − A ^ − 1 ) = ( E A ^ ) − 1 ( A ^ − ( E A ^ ) ) ( E A ^ ) − 1 (105)
When n n n is large enough, we know that 1 2 I ≤ I + ( E A ^ ) − 1 ( A ^ − ( E A ^ ) ) ≤ I \frac{1}{2}I\leq I+(\mathbb{E}\hat{A})^{-1}\left(\hat{A}-(\mathbb{E}\hat{A})\right)\leq I 2 1 I ≤ I + ( E A ^ ) − 1 ( A ^ − ( E A ^ ) ) ≤ I with high probability. Thus the term ∥ u ^ 1 − u ^ 2 ∥ H 1 2 \lVert\hat{u}_1-\hat{u}_2\rVert_{H^1}^2 ∥ u ^ 1 − u ^ 2 ∥ H 1 2 is at the scale of ∥ ( ( E A ^ ) − A ^ ) ∥ H 2 ≈ Z d n \left\lVert\left((\mathbb{E}\hat{A})-\hat{A}\right)\right\rVert_H^2\approx\frac{Z^d}{n} ( ( E A ^ ) − A ^ ) H 2 ≈ n Z d , which is of the same magnitude as what we get from the empirical process approach in our main proof. It is also larger than Z d − 2 / n Z^{d-2}/n Z d − 2 / n , which is the magnitude of the variance term for u ^ 1 \hat{u}_1 u ^ 1 . Therefore, here we conjecture that the our bound for DRM itself is tight and leads to the sub-optimal convergence rate.
In this section, we repeat the standard tools we use to establish the lower bound. The main tool we use is the Fano’s inequality and the Varshamov-Gilber Lemma.
Lemma C.1 (Fano’s methods). Assume that V V V is a uniform random variable over set V \mathcal{V} V , then for any markov chain V → X → V ^ V \to X \to\hat{V} V → X → V ^ , we always have
P ( V ^ ≠ V ) ≥ 1 − I ( V ; X ) + log 2 log ( ∣ V ∣ ) \mathbb{P}(\hat{V} \ne V) \ge1 - \frac{I(V;X) + \log2}{\log(|\mathcal{V}|)} P ( V ^ = V ) ≥ 1 − log ( ∣ V ∣ ) I ( V ; X ) + log 2
Lemma C.2 (Varshamov-Gilbert Lemma, [74 ] Theorem 2.9). Let D ≥ 8 D \ge8 D ≥ 8 . There exists a subset V = { τ ( 0 ) , ⋯ , τ ( 2 D / 8 ) } \mathcal{V} = \{\tau^{(0)}, \cdots, \tau^{(2^{D/8})}\} V = { τ ( 0 ) , ⋯ , τ ( 2 D /8 ) } of D D D -dimensional hypercube H D = { 0 , 1 } D \mathcal{H}^{D} = \{0,1\}^{D} H D = { 0 , 1 } D such that τ ( 0 ) = ( 0 , 0 , ⋯ , 0 ) \tau^{(0)} = (0,0,\cdots,0) τ ( 0 ) = ( 0 , 0 , ⋯ , 0 ) and the ℓ 1 \ell_1 ℓ 1 distance between every two elements is larger than D / 8 D/8 D /8
∑ l = 1 D ∥ τ ( j ) − τ ( k ) ∥ ℓ 1 ≥ D 8 , for all 0 ≤ j , k ≤ 2 D / 8 \sum_{l=1}^{D} \left\|\tau^{(j)}-\tau^{(k)}\right\|_{\ell_1} \ge\frac{D}{8}, \text{ for all } 0 \le j,k \le2^{D/8} l = 1 ∑ D τ ( j ) − τ ( k ) ℓ 1 ≥ 8 D , for all 0 ≤ j , k ≤ 2 D /8