betaFeedback

WorksComputer Science

Residual-based error bound for physics-informed neural networks

Abstract

Neural networks are universal approximators and are studied for their use in solving differential equations. However, a major criticism is the lack of error bounds for obtained solutions. This paper proposes a technique to rigorously evaluate the error bound of Physics-Informed Neural Networks (PINNs) on most linear ordinary differential equations (ODEs), certain nonlinear ODEs, and first-order linear partial differential equations (PDEs). The error bound is based purely on equation structure and residual information and does not depend on assumptions of how well the networks are trained. We propose algorithms that bound the error efficiently. Some proposed algorithms provide tighter bounds than others at the cost of longer run time.

Introduction

Differential equations (DEs) are a useful mathematical tool for describing various phenomena in natural sciences, engineering, and humanity studies. As universal approximators, neural networks are powerful in approximating unknown functions. With back-propagation and modern computing devices, neural networks are convenient to differentiate, making them an ideal choice for solving differential equations.

However, a major criticism of neural network solutions to DEs is the lack of error bound. Traditional numerical methods, such as the finite difference method (FDM) and the finite element method (FEM), compute numerical solutions with known error bounds. Unlike traditional methods, the error bounds of neural network solutions are not well-studied. Therefore, solving DEs with neural networks requires ad hoc customization and empirical hyperparameter finetuning. If the error of any given network can be bounded, we can

Symbols and Notations

DEs in this paper are posed w.r.t. unknown function vv,

Dv=f,\mathcal{D}v = f,

where D\mathcal{D} is a possibly nonlinear differential operator and ff is some forcing function. Unlike the exact solution v(⋅)v(\cdot), a neural network solution u(⋅)u(\cdot) does not strictly satisfy the equation. Instead, it incurs an additional residual term, rr, which the network aims to minimize, to the equation,

Du=f+r.\mathcal{D}u = f + r.

The input to vv, uu, ff, and rr is time tt for ODEs and spatial coordinates (x,y)(x, y) for PDEs. We limit our reasoning to 2-dimensional PDEs in this work. In cases with multiple unknown functions, we use vector notations v\mathbf{v}, u\mathbf{u}, and r\mathbf{r} instead of the scalar notations vv, uu, and rr.

Accepted for the 39th Conference on Uncertainty in Artificial Intelligence (UAI 2023). The loss function of the network solution is defined as the L2L^2 norm of residual rr over the domain of interest,

Loss⁡(u):=1∣I∣∫I∥r∥2 dI=1∣I∣∫I∥Du−f∥2 dI,(1)\operatorname{Loss}(u) := \frac{1}{|I|}\int_I \lVert r\rVert^2 \, \mathrm{d}I = \frac{1}{|I|}\int_I \lVert\mathcal{D}u-f\rVert^2 \, \mathrm{d}I, \tag*{(1)}

where a spatial domain Ω\Omega is substituted for the temporal domain II in the case of a PDE.

Initial and Boundary Conditions

For a neural network to satisfy initial or boundary conditions, we apply a technique called parametrization. As an intuitive example, the parametrization u(t)=(1−e−t)Net⁡(t)+v(0)u(t)=(1-e^{-t})\operatorname{Net}(t)+v(0) guarantees that u(t)u(t) satisfies the initial condition u(0)=v(0)u(0)=v(0) regardless of the network Net⁡(⋅)\operatorname{Net}(\cdot). This does not affect the capability of Net⁡(⋅)\operatorname{Net}(\cdot) to learn any solution.

The parametrization is more complicated for higher-order ODEs and most PDEs and has been extensively studied by Lagaris et al. [ [13], Lagaris et al. [ [14], McFall and Mahan [ [16], Lagari et al. [ [12], and Sukumar and Srivastava [ [20]. In this work, we assume all initial and boundary conditions are exactly satisfied.

Error and Error Bound

The error of a network solution uu is defined as

η:=u−v.(2)\eta:= u-v. \tag*{(2)}

We are interested in bounding the error with a scalar function B\mathcal{B} such that

∥η(t)∥≤B(t)or∥η(x,y)∥≤B(x,y)(3)\lVert\eta(t)\rVert\le\mathcal{B}(t) \quad\text{or} \quad\lVert\eta(x,y)\rVert\le\mathcal{B}(x,y) \tag*{(3)}

where ∥η∥=∥u−v∥\lVert\eta\rVert=\lVert u-v\rVert is the absolute error. If B\mathcal{B} takes on the same value B∈R+B\in\mathbb{R}^{+} over the domain, it can be replaced with a constant BB.

Notice that multiple bounds B\mathcal{B} exist for the same network solution uu. For example, ∣η(t)∣≤B(1)(t)≤B(2)(t)≤⋯≤B|\eta(t)|\le\mathcal{B}^{(1)}(t)\le\mathcal{B}^{(2)}(t)\le\cdots\le B are bounds in decreasing order of tightness. Tighter bounds incur a higher computational cost, and looser bounds (such as constant BB) are faster to compute.

A summary of the applicability, restraints, run-time complexity, and relative tightness of all proposed algorithms is listed in Table 1.

AlgorithmApplicable toRestraintRun-TimeComment
Algorithm 1Linear ODESemi-stableO(L)O(L)Looser than Alg 2
Algorithm 2Linear ODEO(nL)O(nL)Tighter than Alg 1
Algorithm 3Linear ODE SystemO(n3L)O(n^{3}L)Norm and elementwise bounds
Algorithm 4Nonlinear ODENonlinear term is εvk\varepsilon v^{k}O(JnL)O(JnL)Bounded solution for family of DEs
Algorithm 5Linear 1st-Order PDECoeff. c≠0c \ne0 over domainO(mesh)O(\mathrm{mesh})Constant bound; Looser than Alg 6
Algorithm 6Linear 1st-Order PDESolvable characteristicsO(KL)O(KL)Tighter than Alg 5 if computable

Table 1. Overview of Proposed Algorithms. The symbols in run-time analysis are defined and explained in detail in Sections 5 and 6, with the exception of KK, which is the number of steps used in each numerical integration.

Literature Review

Hornik et al. [ [9] showed that neural networks are universal function approximators. Lagaris et al. [ [13] first studied the application of neural networks in solving DEs. The term physics-informed neural networks, or PINNs, was first introduced by Raissi et al. [ [18] to name neural networks that satisfy DEs while fitting observed data points.

Although we train PINNs only to solve DEs without any observed data in this work, the error-bounding algorithms we propose work for any given neural network, regardless of the training process.

Flamant et al. [ [6] and Desai et al. [ [5] showed that one main advantage of neural networks over traditional numerical methods, such as FDM and FEM, is that neural networks can potentially learn the structure of the solution space and give a bundle of solutions u(x;Θ)u(x;\Theta) for different equation setup and initial/boundary conditions parameterized by Θ\Theta. For traditional methods, a new solution must be recomputed for any slight changes in equation setup or initial/boundary conditions.

Some effort has been made to redefine the objective loss function. Yu et al. [ [21] applied the Ritz method to a particular class of variational problems. Mattheakis et al. [ [15] incorporated an additional constraint to force the network to learn solutions with energy conservation. Parwani and Protopapas [ [17] used an adversarial network for sampling in particular areas of the domain where the residual is large.

There are also works that study the failure modes of PINNs and quantify the error of PINN solutions in recent years. Graf et al. [ [7] worked on quantifying the uncertainty of PINNs using the Bayesian framework. Krishnapriyan et al. [ [11] characterized possible failure modes of PINNs by studying the performance of PINNs on simple problems and analyzing their loss landscape. Krishnapriyan et al. [ [11] also concluded that optimization difficulty is the essential cause of failure.

Our work uncovers the mathematical relationship between residual information and the error of PINNs on several classes of ODEs and PDEs. We propose different algorithms for various classes of equations and experimentally validate these algorithms.

Existing Work

Sirignano and Spiliopoulos [ [19] showed that for a class of quasi-linear parabolic PDEs, a neural network with a single hidden layer and sufficiently many hidden units could arbitrarily approximate the exact solutions. Guo and Haghighat [ [8] proposed an energy-based constitutive relation error bound for elasticity problems.

De Ryck and Mishra [ [2] derived an error bound for ReLU networks on parametric hyperbolic conservation laws. De Ryck and Mishra [ [3] showed that there exists some PINN with arbitrarily small residual for Kolmogorov PDEs. De Ryck and Mishra [ [4] derived an error bound for operator learning with PINNs. The works of De Ryck and Mishra mentioned above did not bound the error of every given network. Instead, they mathematically proved the existence of a network with errors below a specified bound, under certain assumptions of network architecture, including width, depth, and activation functions. The question remaining to be answered is how to overcome optimization difficulties and find such a neural network.

Our work differs from the above in that we bound the error of any neural network regardless of finetuning, even networks with randomly initialized weights. Our algorithms only depend on inputs of residual information rr, often used as training loss, and equations structure Dv=f\mathcal{D}v=f. The output is a (possibly constant) function that guarantees to bound the error at any point in domain.

Error Bound for ODE

This section considers both linear and nonlinear ODEs over the temporal domain I=[0,T]I=[0,T]. Initial conditions are imposed on dkdtkv(t=0)\frac{\mathrm{d}^{k}}{\mathrm{d}t^{k}}v(t=0) for k=0,…,(n−1)k=0,\ldots,(n-1), where nn is the highest order of derivative terms in ODE.

Error Bound for Linear ODE

Consider the linear ODE Lv(t)=f(t)\mathcal{L}v(t)=f(t), where L\mathcal{L} is a linear differential operator. Its neural network solution uu satisfies Lu(t)=f(t)+r(t)\mathcal{L}u(t)=f(t)+r(t). Since error η:=u−v\eta:=u-v, there is

Lη(t)=r(t).(4)\mathcal{L}\eta(t)=r(t). \tag*{(4)}

With the assumption in § 1.2 that uu satisfies the initial conditions at t=0t=0, there is

η(0)=0,ddtη(0)=0,d2dt2η(0)=0,…(5)\eta(0)=0,\qquad\frac{\mathrm{d}}{\mathrm{d}t}\eta(0)=0,\qquad\frac{\mathrm{d}^{2}}{\mathrm{d}t^{2}}\eta(0)=0,\qquad\ldots \tag*{(5)}

With initial conditions 5 known, a unique inverse transform L−1\mathcal{L}^{-1} to L\mathcal{L} exists. Applying L−1\mathcal{L}^{-1} to Eq. 4, there is

η(t)=L−1r(t).(6)\eta(t)=\mathcal{L}^{-1}r(t). \tag*{(6)}

Hence, bounding the absolute error ∣η∣|\eta| is equivalent to bounding ∣L−1r∣|\mathcal{L}^{-1}r|. Notice that only a) the equation structure L\mathcal{L} and b) the residual information rr are relevant to estimating the error bound. All other factors, including parameters of the neural network uu, forcing function ff, and initial conditions, do not affect the error bound at all.

Single Linear ODE with Constant Coefficients

Consider the case where L=dndtn+∑j=0n−1ajdjdtj\mathcal{L}=\frac{\mathrm{d}^{n}}{\mathrm{d}t^{n}}+\sum_{j=0}^{n-1}a_{j}\frac{\mathrm{d}^{j}}{\mathrm{d}t^{j}} consists of only constant coefficients a0,a1,…,an∈Ra_{0},a_{1},\ldots,a_{n}\in\mathbb{R}. Its characteristic polynomial (defined below) can be factorized into

λn+an−1λn−1+⋯+a0=∏j=1n(λ−λj),(7)\lambda^{n}+a_{n-1}\lambda^{n-1}+\cdots+a_{0}=\prod_{j=1}^{n}(\lambda-\lambda_{j}), \tag*{(7)}

where λ1,…,λn∈C\lambda_{1},\ldots,\lambda_{n}\in\mathbb{C} are the characteristic roots.

It can be shown that, for a semi-stable system (Re(λj)≤0\mathop{\mathrm{Re}}(\lambda_{j})\leq0 for all λj\lambda_{j}), an error bound can be formulated as

∣η(t)∣≤Bloose(t):=Cλ1:nRmax⁡tZ,(8)|\eta(t)|\leq\mathcal{B}_{\mathrm{loose}}(t):=C_{\lambda_{1:n}}R_{\max}t^{Z}, \tag*{(8)}

where 0≤Z≤n0\leq Z\leq n is the number of λj\lambda_{j} whose real part is 0, Cλ1:n:=1Z!∏j=1;λj≠0n1Re(−λj)C_{\lambda_{1:n}}:=\frac{1}{Z!}\prod_{\substack{j=1;\lambda_{j}\ne0}}^{n}\frac{1}{\mathop{\mathrm{Re}}(-\lambda_{j})} is a constant coefficient, and Rmax⁡:=max⁡t∈I∣r(t)∣R_{\max}:=\max_{t\in I}|r(t)| is the maximum absolute residual. Knowing bound 8 is sufficient to qualitatively estimate the error for applications where only the order of error is concerned. See Alg. 1 for reference.

An issue with Eq. 8 and Alg. 1 is that they assume Re(λj)≤0\mathop{\mathrm{Re}}(\lambda_{j})\leq0 for all characteristic roots λj\lambda_{j}. To address this issue, we propose an alternative error-bounding Alg. 2, which requires more computation but does not require the system to be semi-stable and provides a tighter bound.

Notice that the bounds of η\eta in Eq. 6 can be estimated if the inverse operator L−1\mathcal{L}^{-1} is known. Let Eq. 7 the factorization of characteristic polynomial of L\mathcal{L}. Define operator Iλ\mathcal{I}_{\lambda} as1

Iλψ(t):=eλt∫0te−λτψ(τ) dτ,∀ψ:I→C.(9)\mathcal{I}_{\lambda}\psi(t):=e^{\lambda t}\int_{0}^{t}e^{-\lambda\tau}\psi(\tau)\,\mathrm{d}\tau,\qquad\forall\psi:I\to\mathbb{C}. \tag*{(9)}

We show in supplementary material that L−1=Iλn∘Iλn−1∘⋯∘Iλ1\mathcal{L}^{-1}=\mathcal{I}_{\lambda_{n}}\circ\mathcal{I}_{\lambda_{n-1}}\circ\cdots\circ\mathcal{I}_{\lambda_{1}} and that ∣Iλψ∣≤IRe(λ)∣ψ∣|\mathcal{I}_{\lambda}\psi|\leq\mathcal{I}_{\mathop{\mathrm{Re}}(\lambda)}|\psi| for any λ∈C\lambda\in\mathbb{C} and function ψ\psi. Hence, another error bound can be formulated as

Btight(t):=(IRe(λn)∘⋯∘IRe(λ1))∣r(t)∣.(10)\mathcal{B}_{\mathrm{tight}}(t):=\left(\mathcal{I}_{\mathop{\mathrm{Re}}(\lambda_{n})}\circ\cdots\circ\mathcal{I}_{\mathop{\mathrm{Re}}(\lambda_{1})}\right)|r(t)|. \tag*{(10)}

Algorithm 1 Loose Error Bound Estimation for Linear ODE with Constant Coefficients (Requires Semi-Stability)

Input: Coefficients {aj}j=0n−1\{a_j\}_{j=0}^{n-1} for operator L\mathcal{L}, residual information r(⋅)r(\cdot), domain of interest I=[0,T]I=[0,T], and a sequence of time points {tℓ}ℓ=1L\{t_\ell\}_{\ell=1}^{L} where error bound is to be evaluated

Output: Error bound at given time points {B(tℓ)}ℓ=1L\{B(t_\ell)\}_{\ell=1}^{L}

Require: L\mathcal{L} is semi-stable, and tℓ∈It_\ell\in I for all ℓ\ell

Ensure: ∣η(tℓ)∣≤B(tℓ)|\eta(t_\ell)| \le B(t_\ell) for all ℓ\ell

{λj}j=1n←\{\lambda_j\}_{j=1}^{n} \leftarrow numerical roots of λn+an−1λn−1+⋯=0\lambda^n+a_{n-1}\lambda^{n-1}+\cdots=0

assert λj≤0\lambda_j \le0 for 1≤j≤n1 \le j \le n

Z,C←0,1Z,C \leftarrow0,1

for j←1…nj \leftarrow1\ldots n do

if Re⁡(λj)=0\operatorname{Re}(\lambda_j)=0 then

Z←Z+1Z \leftarrow Z+1

else

C←C/Re⁡(−λj)C \leftarrow C/\operatorname{Re}(-\lambda_j)

end if

end for

Rmax⁡←max⁡τ∈I∣r(τ)∣R_{\max} \leftarrow\max_{\tau\in I}|r(\tau)| ▷ Use linspace with mini-steps

{B(tℓ)}ℓ=1L←{CZ!Rmax⁡tℓZ}ℓ=1L\{B(t_\ell)\}_{\ell=1}^{L} \leftarrow\left\{\frac{C}{Z!}R_{\max}t_\ell^{Z}\right\}_{\ell=1}^{L}

return {B(tℓ)}ℓ=1L\{B(t_\ell)\}_{\ell=1}^{L}

Note: Jenkins and Traub [ [10] solves polynomial roots.

It is also proven in supplementary material that BtightB_{\mathrm{tight}} is tighter than BlooseB_{\mathrm{loose}} when BlooseB_{\mathrm{loose}} is applicable,

∣η(t)∣≤Btight(t)≤Bloose(t)∀t∈I.(11)|\eta(t)| \le B_{\mathrm{tight}}(t) \le B_{\mathrm{loose}}(t) \quad\forall t \in I. \tag*{(11)}

Based on Eq. 10, we propose Alg. 2, which computes BtightB_{\mathrm{tight}} by repeatedly evaluating integrals in 9 using the cumulative trapezoidal rule.

Before moving on to the next section, we discuss the effect of numerical error in Alg. 2 and all pursuant algorithms which involve numerical integration. Empirically, the error introduced by numerical integration in negligible for most cases. To ensure the accuracy of the error bound, we recommend using a slightly modified trapezoidal rule that, instead of directly using the node value, takes the maximum of the node and its two adjacent nodes. This modification leads to a slightly looser bound. Yet, this influence is almost negligible in practice.

Single Linear ODE of the General Form

In general, the coefficients for L\mathcal{L} can be functions of tt. Namely, L=dndtn+∑j=0n−1aj(t)djdtj\mathcal{L}=\frac{\mathrm{d}^{n}}{\mathrm{d}t^{n}}+\sum_{j=0}^{n-1}a_j(t)\frac{\mathrm{d}^{j}}{\mathrm{d}t^{j}}. Similar to Eq. 7, L\mathcal{L} have characteristic roots {λj(t)}j=1n\{\lambda_j(t)\}_{j=1}^{n} as functions of tt,

λn+an−1(t)λn−1+⋯+a0(t)=∏j=1n(λ−λj(t)).\lambda^n+a_{n-1}(t)\lambda^{n-1}+\cdots+a_0(t)=\prod_{j=1}^{n}(\lambda-\lambda_j(t)).

We can replace constant λj\lambda_j with functions λj(t)\lambda_j(t) in Eq. 9 and compute bound BtightB_{\mathrm{tight}} as in Eq. 10. However, the factorization in Eq. 5.1.2 is hard to implement in practice except

Algorithm 2 Tighter Error Bound Estimation for Linear ODE with Constant Coefficients (Stable and Unstable)

Input & Output: Same as Alg. 1

Require: Same as Alg. 1, except L\mathcal{L} can be unstable

Ensure: Same as Alg. 1

{λj}j=1n←\{\lambda_j\}_{j=1}^{n}\leftarrow numerical roots of λn+an−1λn−1+⋯=0\lambda^n+a_{n-1}\lambda^{n-1}+\cdots=0

{tk}k=0K←\{t_k\}_{k=0}^{K}\leftarrow linspace(0,T,(0,T, sufficient steps))

{B(tk)}k=0K←{∣r(tk)∣}k=0K\{B(t_k)\}_{k=0}^{K}\leftarrow\{|r(t_k)|\}_{k=0}^{K}

for j←1…nj\leftarrow1\ldots n do

intgr⁡k=0K←CumTrap⁡({e−λjtkB(tk)}k=0K,{tk}k=0K)\operatorname{intgr}_{k=0}^{K}\leftarrow\operatorname{CumTrap}\left(\{e^{-\lambda_jt_k}B(t_k)\}_{k=0}^{K},\{t_k\}_{k=0}^{K}\right)

{B(tk)}k=0K←{eλjtk⋅intgr⁡k}k=0K\{B(t_k)\}_{k=0}^{K}\leftarrow\{e^{\lambda_jt_k}\cdot\operatorname{intgr}_k\}_{k=0}^{K}

end for

{B(tℓ)}ℓ=1L←Interp⁡({B(tk)}k=0K,{tk}k=0K,{tℓ}ℓ=0L)\{B(t_\ell)\}_{\ell=1}^{L}\leftarrow\operatorname{Interp}\left(\{B(t_k)\}_{k=0}^{K},\{t_k\}_{k=0}^{K},\{t_\ell\}_{\ell=0}^{L}\right)

return {B(tℓ)}ℓ=1L\{B(t_\ell)\}_{\ell=1}^{L}

Note: CumTrap⁡({yk}k=1K,{xk}k=1K)\operatorname{CumTrap}(\{y_k\}_{k=1}^{K},\{x_k\}_{k=1}^{K}) computes cumulative integral ∫0xy(x) dx\int_0^x y(x)\,\mathrm{d}x at discrete points {xk}k=1K\{x_k\}_{k=1}^{K} using trapezoidal rule.

Note: Interp⁡({yk}k=1K,{xk}k=1K,{xℓ}ℓ=1L)\operatorname{Interp}(\{y_k\}_{k=1}^{K},\{x_k\}_{k=1}^{K},\{x_\ell\}_{\ell=1}^{L}) computes interpolant to {yk}k=1K\{y_k\}_{k=1}^{K} with given discrete data points {(xk,yk)}k=1K\{(x_k,y_k)\}_{k=1}^{K} evaluated at {xℓ}ℓ=1L\{x_\ell\}_{\ell=1}^{L}.

for the first-order case where Lv=dvdt+a0(t)v\mathcal{L}v=\frac{\mathrm{d}v}{\mathrm{d}t}+a_0(t)v. Cases of second order and higher are out of the scope of this paper.

System of Linear ODEs with Constant Coefficients

Consider a system of linear ODEs with constant coefficients

ddtv+Av=f(t)(12)\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{v}+A\mathbf{v}=\mathbf{f}(t) \tag*{(12)}

where v\mathbf{v} and f\mathbf{f} are Rn\mathbb{R}^{n} vectors and AA is a n×nn\times n matrix. Denote the Jordan canonical form of AA as,

J=P−1AP=(J1⋱JK)whereJk=(λk1λk⋱⋱1λk).(13)J=P^{-1}AP= \begin{pmatrix} J_1 & & \\ & \ddots& \\ & & J_K \end{pmatrix} \quad\text{where}\quad J_k= \begin{pmatrix} \lambda_k & 1 & \\ & \lambda_k & \ddots\\ & & \ddots& 1 \\ & & & \lambda_k \end{pmatrix}. \tag*{(13)}

Let nkn_k be the size of Jordan block JkJ_k, we construct an operator matrix I=diag⁡(I1,I2,…)\mathcal{I}=\operatorname{diag}(\mathcal{I}_1,\mathcal{I}_2,\ldots), where

Ik=(I−Re⁡(λk)I−Re⁡(λk)2⋯I−Re⁡(λk)nk0I−Re⁡(λk)⋯I−Re⁡(λk)nk−1⋮⋮⋱⋮00⋯I−Re⁡(λk)).(14)\mathcal{I}_k= \begin{pmatrix} \mathcal{I}_{-\operatorname{Re}(\lambda_k)} & \mathcal{I}_{-\operatorname{Re}(\lambda_k)}^2 & \cdots& \mathcal{I}_{-\operatorname{Re}(\lambda_k)}^{n_k}\\ 0 & \mathcal{I}_{-\operatorname{Re}(\lambda_k)} & \cdots& \mathcal{I}_{-\operatorname{Re}(\lambda_k)}^{n_k-1}\\ \vdots& \vdots& \ddots& \vdots\\ 0 & 0 & \cdots& \mathcal{I}_{-\operatorname{Re}(\lambda_k)} \end{pmatrix}. \tag*{(14)}

An elementwise bound (vector) B(t)\mathcal{B}(t) can be formulated as

η∣⋅∣(t)⪯B(t):=P∣⋅∣I(P−1)∣⋅∣∣r∣(t),(15)\boldsymbol{\eta}^{\lvert\cdot\rvert}(t)\preceq\mathcal{B}(t):=P^{\lvert\cdot\rvert}\mathcal{I}\left(P^{-1}\right)^{\lvert\cdot\rvert}\lvert\mathbf{r}\rvert(t), \tag*{(15)}

where superscript ∣⋅∣\lvert\cdot\rvert denotes taking elementwise absolute value and symbol ⪯\preceq denotes elementwise inequality. In the meantime, a norm bound (scalar) B(t)\mathcal{B}(t) also exists

∥η(t)∥≤B(t):=cond⁡(P)∥I[∥r∥11](t)∥(16)\lVert\eta(t) \rVert\le\mathcal{B}(t) := \operatorname{cond}(P)\left\lVert\mathcal{I}\left[\lVert r\rVert_1\mathbf{1}\right](t)\right\rVert \tag*{(16)}

where cond⁡(P)\operatorname{cond}(P) is the conditional number of PP w.r.t. induced matrix norm and 1\mathbf{1} is an n×1n \times1 column vector of 1s. Proof of (49) and (50) can be found in supplementary material. See Alg. 3 for implementation.

Algorithm 3 ODE System Bound (norm and elementwise)

Input: Coefficient matrix A∈Rn×nA \in\mathbb{R}^{n \times n}, residual vector r(t)r(t), and a sequence of points {tℓ}ℓ=1L\{t_\ell\}_{\ell=1}^{L} where error is to be bounded

Output: Norm bound (scalar) {B(tℓ)}ℓ=1L\{\mathcal{B}(t_\ell)\}_{\ell=1}^{L} and componentwise bound (vector) {B(tℓ)}ℓ=1L\{\mathbf{B}(t_\ell)\}_{\ell=1}^{L} at given time points

Ensure: ∥η(tℓ)∥≤B(tℓ)\lVert\eta(t_\ell)\rVert\le\mathcal{B}(t_\ell) and η(tℓ)⪯B(tℓ)\eta(t_\ell) \preceq\mathbf{B}(t_\ell) for all ℓ\ell

J,P←J, P \leftarrow Jordan canonicalization of A=PJP−1A = PJP^{-1}

for each Jordan block JkJ_k of shape nk×nkn_k \times n_k do

Ik←I_k \leftarrow construct operator block using (48)

end for

I←diag⁡(I1,I2,…)\mathcal{I} \leftarrow\operatorname{diag}(\mathcal{I}_1,\mathcal{I}_2,\ldots)

{B(tℓ)}ℓ=1L←{P∣I[(P−1)∣r∣]∣(tℓ)}ℓ=1L\{\mathbf{B}(t_\ell)\}_{\ell=1}^{L} \leftarrow\{P\lvert\mathcal{I}[(P^{-1})\lvert r\rvert]\rvert(t_\ell)\}_{\ell=1}^{L}

{B(tℓ)}ℓ=1L←{cond⁡(P)∥I[∥r∥11](tℓ)∥}ℓ=1L\{\mathcal{B}(t_\ell)\}_{\ell=1}^{L} \leftarrow\{\operatorname{cond}(P)\lVert\mathcal{I}[\lVert r\rVert_1\mathbf{1}](t_\ell)\rVert\}_{\ell=1}^{L}

return {B(tℓ)}ℓ=1L,{B(tℓ)}ℓ=1L\{\mathcal{B}(t_\ell)\}_{\ell=1}^{L}, \{\mathbf{B}(t_\ell)\}_{\ell=1}^{L}

Nonlinear ODE

Nonlinear ODEs are hard to solve in general. In this work, we only deal with nonlinear ODEs with a single nonlinear term of the form εvk(t)\varepsilon v^k(t), where ε∈R\varepsilon\in\mathbb{R} is a small number. Ideally, ∣ε∣≪1\lvert\varepsilon\rvert\ll1. The exact requirement for ε\varepsilon is given in § 5.2. The value of ε\varepsilon can vary within a certain range or be fixed. With the perturbation technique, we obtain a family of solutions v(t;ε)v(t;\varepsilon) parameterized by ε\varepsilon at the cost of solving a (countable) collection of equations. As explained below in § 5.1, we train finitely many networks, each approximately solving an equation in the collection.

Perturbation Theory

Consider the nonlinear ODE with nonlinear term εvk(t)\varepsilon v^k(t),

Lv(t)+εvk(t)=f(t),(17)\mathcal{L}v(t) + \varepsilon v^k(t) = f(t), \tag*{(17)}

where L\mathcal{L} is a linear differential operator discussed in 5.1 and initial conditions are specified for the system at time t=0t = 0. Notice that each ε∈R\varepsilon\in\mathbb{R} corresponds to a solution v(t;ε)v(t;\varepsilon). We expand the solution v(t;ε)v(t;\varepsilon) in terms of ε\varepsilon

v(t;ε)=∑j=0∞εjvj(t)=v0(t)+εv1(t)+…(18)v(t;\varepsilon) = \sum_{j=0}^{\infty}\varepsilon^jv_j(t) = v_0(t) + \varepsilon v_1(t) + \ldots \tag*{(18)}

Only v0(t)v_0(t) is subject to the original initial conditions at t=0t = 0, while other components, v1v_1, v2v_2, …\ldots, have initial conditions of 0 at $t = 0. Substituting Eq. 18 into Eq. 17,

L∑j=0∞εjvj+ε(∑j=0∞εjvj)k=f(19)\mathcal{L}\sum_{j=0}^{\infty}\varepsilon^jv_j+\varepsilon\left(\sum_{j=0}^{\infty}\varepsilon^jv_j\right)^k=f \tag*{(19)}
∑j=0∞εjLvj+∑j=0∞εj+1∑j1+⋯+jk=jj1,…,jk≥0vj1⋯vjk=f(20)\sum_{j=0}^{\infty}\varepsilon^j\mathcal{L}v_j+\sum_{j=0}^{\infty}\varepsilon^{j+1}\sum_{\substack{j_1+\cdots+j_k=j\\j_1,\ldots,j_k\ge0}}v_{j_1}\cdots v_{j_k}=f \tag*{(20)}
Lv0+∑j=1∞εj(Lvj+∑j1+⋯+jk=j−1j1,…,jk≥0vj1⋯vjk)=f.(21)\mathcal{L}v_0+\sum_{j=1}^{\infty}\varepsilon^j\left(\mathcal{L}v_j+\sum_{\substack{j_1+\cdots+j_k=j-1\\j_1,\ldots,j_k\ge0}}v_{j_1}\cdots v_{j_k}\right)=f. \tag*{(21)}

In order for (55) to hold true for all ε\varepsilon, the coefficients for each εj\varepsilon^j must match on both sides of (55). Hence,

Lv0=f(22)\mathcal{L}v_0=f \tag*{(22)}
Lv1+v0k=0(23)\mathcal{L}v_1+v_0^k=0 \tag*{(23)}
Lv2+kv0k−1v1=0(24)\mathcal{L}v_2+kv_0^{k-1}v_1=0 \tag*{(24)}
Lv3+k(k−1)2v0k−2v12+kv0k−1v2=0(25)\mathcal{L}v_3+\frac{k(k-1)}{2}v_0^{k-2}v_1^2+kv_0^{k-1}v_2=0 \tag*{(25)}

For ε=0\varepsilon=0, (52) is reduced to v0(t)v_0(t), which solves the linear problem Lv=f\mathcal{L}v=f.

The above system can be solved in a sequential manner, either analytically or using neural networks,

  1. (56) is linear in v0v_0 and can be solved first.

  2. With v0v_0 known, (57) is linear in v1v_1 and can be solved for v1v_1.

  3. Similarly, with v0v_0 and v1v_1 known, (58) is linear in v2v_2 and can be solved for v2v_2.

  4. The process can be repeated for (59) and beyond. Only a linear ODE is solved each time.

To solve the system with PINNs, we approximate exact solutions {vj(t)}j=1∞\{v_j(t)\}_{j=1}^{\infty} with neural network solutions {uj(t)}j=0J\{u_j(t)\}_{j=0}^{J} trained sequentially on (56), (57), and beyond. In practice, we only consider components up to order JJ to avoid the infinity in expansion 18. Ideally, JJ should be large enough so that higher order residuals in expansion 18 can be neglected.

After obtaining {uj(t)}j=0J\{u_j(t)\}_{j=0}^{J}, we can reconstruct the solution u(t;ε)=∑j=0Jεjuj(t)u(t;\varepsilon)=\sum_{j=0}^{J}\varepsilon^ju_j(t) to the original nonlinear equation 17 for varying ε\varepsilon. See Alg. 4 for details.

Expansion of Bounds

The absolute error ∣η(t;ε)∣=∣u(t;ε)−v(t;ε)∣\lvert\eta(t;\varepsilon)\rvert=\lvert u(t;\varepsilon)-v(t;\varepsilon)\rvert is given by

∣η(t;ε)∣=∣∑j=0Jεj(uj(t)−vj(t))−∑j=J+1∞εjvj(t)∣≤∑j=0J∣ηj(t)∣∣ε∣j+∣∑j=J+1∞εjvj(t)∣(26)\begin{aligned} \lvert\eta(t;\varepsilon)\rvert=\left\lvert\sum_{j=0}^{J}\varepsilon^j\left(u_j(t)-v_j(t)\right)-\sum_{j=J+1}^{\infty}\varepsilon^jv_j(t)\right\rvert\\ &\le\sum_{j=0}^{J} \lvert\eta_j(t) \rvert\lvert\varepsilon\rvert^j + \left\lvert\sum_{j=J+1}^{\infty} \varepsilon^j v_j(t) \right\rvert \tag*{(26)} \end{aligned}

where ηj(t):=uj(t)−vj(t)\eta_j(t) := u_j(t) - v_j(t) is the component error between uj(t)u_j(t) and vj(t)v_j(t). Let BjB_j denote the bound component such that ∣ηj(t)∣≤Bj(t)\lvert\eta_j(t) \rvert\le B_j(t). Assuming JJ is large and ε\varepsilon is small such that higher order terms ∣∑j=J+1∞εjvj(t)∣\left\lvert\sum_{j=J+1}^{\infty} \varepsilon^j v_j(t) \right\rvert are negligible, there is

∣η(t;ε)∣≤B(t;ε):=∑j=0JBj(t)∣ε∣j(27)\lvert\eta(t; \varepsilon) \rvert\le B(t; \varepsilon) := \sum_{j=0}^{J} B_j(t) \lvert\varepsilon\rvert^j \tag*{(27)}

where each bound component BjB_j can be evaluated using the technique in Section 5.1. See Alg. 4 for details.

Algorithm 4 Iterative Method for Solution and Error Bound of Nonlinear ODE 17

Input: Linear operator L\mathcal{L}, nonlinear degree kk, domain I=[0,T]I = [0,T], highest order JJ for expansion, and a sequence {(tℓ,εℓ)}ℓ=1L\{(t_\ell,\varepsilon_\ell)\}_{\ell=1}^{L} where solution u(t;ε)u(t;\varepsilon) and error bound B(t;ε)B(t;\varepsilon) are to be evaluated

Output: Solution {u(tℓ;εℓ)}ℓ=1L\{u(t_\ell;\varepsilon_\ell)\}_{\ell=1}^{L} and error bound {B(tℓ;εℓ)}ℓ=1L\{B(t_\ell;\varepsilon_\ell)\}_{\ell=1}^{L}

Require: tℓ∈It_\ell\in I, and ∣εℓ∣\lvert\varepsilon_\ell\rvert to be small (ideally ∣εℓ∣≪1\lvert\varepsilon_\ell\rvert\ll1)

Ensure: η(tℓ;εℓ)≤B(tℓ;εℓ)\eta(t_\ell;\varepsilon_\ell) \le B(t_\ell;\varepsilon_\ell)

u0,r0←u_0, r_0 \leftarrow net solution, residual of Lu0=f\mathcal{L}u_0 = f

{B0(tℓ)}ℓ=1L←\{B_0(t_\ell)\}_{\ell=1}^{L} \leftarrow bound of ∣L−1r0∣\lvert\mathcal{L}^{-1}r_0 \rvert at {tℓ}ℓ=1L\{t_\ell\}_{\ell=1}^{L}

for j←1…Jj \leftarrow1 \ldots J do

Macro NLj[ϕ]←∑j1+⋯+jk=j−1j1,…,jk≥0ϕj1⋯ϕjk\mathrm{NL}_j[\phi] \leftarrow\sum_{\substack{j_1+\cdots+j_k=j-1 \\ j_1,\ldots,j_k\ge0}} \phi_{j_1}\cdots\phi_{j_k}

uj,rj←u_j, r_j \leftarrow net solution, residual of Luj+NLj[u]=0\mathcal{L}u_j + \mathrm{NL}_j[u] = 0

BNL←B_{\mathrm{NL}} \leftarrow upper bound of ∣NLj[u]−NLj[v]∣\lvert\mathrm{NL}_j[u] - \mathrm{NL}_j[v] \rvert

{Bj(tℓ)}ℓ=1L←\{B_j(t_\ell)\}_{\ell=1}^{L} \leftarrow bound of ∣L−1rj∣+∣L−1BNL∣\lvert\mathcal{L}^{-1}r_j \rvert+ \lvert\mathcal{L}^{-1}B_{\mathrm{NL}} \rvert

end for

{u(tℓ;εℓ)}ℓ=1L←{∑j=0Jεℓjuj(tℓ)}ℓ=1L\{u(t_\ell;\varepsilon_\ell)\}_{\ell=1}^{L} \leftarrow\left\{\sum_{j=0}^{J}\varepsilon_\ell^j u_j(t_\ell)\right\}_{\ell=1}^{L}

{B(tℓ;εℓ)}ℓ=1L←{∑j=0JεℓjBj(tℓ)}ℓ=1L\{B(t_\ell;\varepsilon_\ell)\}_{\ell=1}^{L} \leftarrow\left\{\sum_{j=0}^{J}\varepsilon_\ell^j B_j(t_\ell)\right\}_{\ell=1}^{L}

return {u(tℓ;εℓ)}ℓ=1L\{u(t_\ell;\varepsilon_\ell)\}_{\ell=1}^{L}, {B(tℓ;εℓ)}ℓ=1L\{B(t_\ell;\varepsilon_\ell)\}_{\ell=1}^{L}

Note 1: B0B_0 and B1:JB_{1:J} can be evaluated using Alg. 1 or 2.

Note 2: BNLB_{\mathrm{NL}} can be estimated even though exact solutions v0:j−1v_{0:j-1} are unknown. This is because vi∈[ui−Bi,ui+Bi]v_i \in[u_i - B_i, u_i + B_i] for all ii, and u0:j−1,B0:j−1u_{0:j-1}, B_{0:j-1} are known from previous iterations.

Error Bound for PDE

This section considers first-order linear PDEs defined on a 2-dimensional spatial domain Ω\Omega.2

a(x,y)∂v∂x+b(x,y)∂v∂y+c(x,y)v=f(x,y)(28)a(x,y)\frac{\partial v}{\partial x} + b(x,y)\frac{\partial v}{\partial y} + c(x,y)v = f(x,y) \tag*{(28)}

with Dirichlet boundary constraints defined on Γ⊂∂Ω\Gamma\subset\partial\Omega,

v∣(x,y)∈Γ=g(x,y).(29)v\big\vert_{(x,y)\in\Gamma} = g(x,y). \tag*{(29)}

We partition the domain into infinitely many characteristic curves C\mathcal{C}, each passing through a point (x0,y0)∈Γ(x_0,y_0) \in\Gamma. The resulting curve is a parameterized integral curve

C:{x′(s)=a(x,y)y′(s)=b(x,y)where (⋅)′=dds and {x(0)=x0y(0)=y0.\mathcal{C}:\begin{cases} x'(s)=a(x,y)\\ y'(s)=b(x,y) \end{cases} \quad\text{where }(\cdot)'=\frac{\mathrm{d}}{\mathrm{d}s}\text{ and }\begin{cases} x(0)=x_0\\ y(0)=y_0. \end{cases}

For any (x(s),y(s))(x(s),y(s)) on C\mathcal{C}, functions (v,a,b,c,f)(v,a,b,c,f) can be viewed as univariate functions of ss. By chain rule, there is

a(x,y)∂v∂x+b(x,y)∂v∂y=x′(s)∂v∂x+y′(s)∂v∂y=v′(s).a(x,y)\frac{\partial v}{\partial x}+b(x,y)\frac{\partial v}{\partial y}=x'(s)\frac{\partial v}{\partial x}+y'(s)\frac{\partial v}{\partial y}=v'(s).

Hence, (62) is reformulated as an ODE along curve C\mathcal{C},

v′(s)+c(s)v(s)=f(s)s.t. v(0)=g(x0,y0),(30)v'(s)+c(s)v(s)=f(s)\quad\text{s.t. }v(0)=g(x_0,y_0), \tag*{(30)}

where v(s)v(s), c(s)c(s), and f(s)f(s) are shorthand notations for v(x(s),y(s))v(x(s),y(s)), c(x(s),y(s))c(x(s),y(s)), and f(x(s),y(s))f(x(s),y(s)), respectively.

In particular, if c(x,y)≠0c(x,y)\ne0 for all (x,y)∈Ω(x,y)\in\Omega, both sides of (62) can be divided by c(x,y)c(x,y), resulting in a residual of r(x,y)/c(x,y)r(x,y)/c(x,y) where r(x,y)r(x,y) is the residual of the original problem. By (42), a constant error bound on C\mathcal{C} is ∣η(s)∣≤max⁡s∣r(s)/c(s)∣\lvert\eta(s)\rvert\le\max_s\lvert r(s)/c(s)\rvert. Hence, a (loose) constant error bound BB (see Alg. 5) over the entire domain Ω\Omega is

∣η(x,y)∣≤B:=max⁡(x,y)∈Ω∣r(x,y)c(x,y)∣.(31)\lvert\eta(x,y)\rvert\le B:=\max_{(x,y)\in\Omega}\left\lvert\frac{r(x,y)}{c(x,y)}\right\rvert. \tag*{(31)}

Algorithm 5 Constant Err Bound for Linear 1st-Order PDE

Input: Coefficient c(x,y)c(x,y) in (62), residual information r(x,y)r(x,y) and domain of interest Ω\Omega

Output: A constant error bound B∈R+B\in\mathbb{R}^{+}

Require: c(x,y)≠0c(x,y)\ne0 for all (x,y)∈Ω(x,y)\in\Omega

Ensure: ∣η(x,y)∣≤B\lvert\eta(x,y)\rvert\le B for all (x,y)∈Ω(x,y)\in\Omega

{(xk,yk)}k←\{(x_k,y_k)\}_{k} \leftarrow sufficiently dense mesh grid over Ω\Omega

B←max⁡k∣r(xk,yk)c(xk,yk)∣B\leftarrow\max_k\left\lvert\frac{r(x_k,y_k)}{c(x_k,y_k)}\right\rvert

return BB

Independent of the assumption c(x,y)≠0c(x,y)\ne0, in scenarios where the curve C\mathcal{C} passing through any (x,y)(x,y) can be computed, the error can be computed using Alg. 6.

Relevant Experiments

In this section, we perform experiments on equations with manufactured solutions using the NeuroDiffEq library [1], which provides convenient tools for training PINNs.

Algorithm 6 General Err Bound for Linear 1st-Order PDE

Input: Coefficients a(x,y)a(x,y), b(x,y)b(x,y), c(x,y)c(x,y) in Eq. 28, residual information r(x,y)r(x,y), domain of interest Ω\Omega, Dirichlet boundary Γ⊂∂Ω\Gamma\subset\partial\Omega, and a sequence of points {(xℓ,yℓ)}ℓ=1L\{(x_\ell,y_\ell)\}_{\ell=1}^{L} where error is to be bounded

Output: Error bound {B(xℓ,yℓ)}ℓ=1L\{B(x_\ell,y_\ell)\}_{\ell=1}^{L} at given points

Require: Integral curve of vector field [a(x,y) b(x,y)]T\left[a(x,y)\ b(x,y)\right]^{T} passing through any point (xℓ,yℓ)∈Ω(x_\ell,y_\ell)\in\Omega is solvable

Ensure: ∣η(xℓ,yℓ)∣≤B(xℓ,yℓ)|\eta(x_\ell,y_\ell)| \leq B(x_\ell,y_\ell) for all ℓ\ell

Cg←C_g \leftarrow general solution to

{x′(s)=a(x,y)y′(s)=b(x,y)\begin{cases} x'(s)=a(x,y)\\ y'(s)=b(x,y) \end{cases}

for ℓ←1…L\ell\leftarrow1\ldots L do

    x(s),y(s)←x(s),y(s)\leftarrow instance of CgC_g passing through (xℓ,yℓ)(x_\ell,y_\ell)

    s∗←s^* \leftarrow solution to x(s)=xℓx(s)=x_\ell, y(s)=yℓy(s)=y_\ell

    B(xℓ,yℓ)←ec(xℓ,yℓ)∫0s∗r(x(s),y(s))e−c(x(s),y(s))s dsB(x_\ell,y_\ell)\leftarrow e^{c(x_\ell,y_\ell)}\int_{0}^{s^*}r(x(s),y(s))e^{-c(x(s),y(s))s}\,\mathrm{d}s

end for

return {B(xℓ,yℓ)}ℓ=1L\{B(x_\ell,y_\ell)\}_{\ell=1}^{L}

First, we train networks to solve equations and collect their residual information rr. Then, we apply Alg. 1–6 (where applicable) to derive error bounds using only residual information rr and equation structure, characterized by its differential operator D\mathcal{D}. Lastly, we show that the absolute error strictly falls within the bounds, regardless of how well the networks are trained.

Throughout this section, we always use networks with two hidden layers, each consisting of 32 hidden units. Depending on whether the problem is an ODE or PDE, a network can have a single input tt or two inputs (x,y)(x,y), but always have a single output. The activation function is tanh. Unless otherwise noted, the training domain is I=[0,1]I=[0,1] for ODEs and Ω=[0,1]2\Omega=[0,1]^2 for PDEs. We use a PyTorch Adam optimizer with default hyperparameters to train networks for 1000 epochs.

Notice that we list these configurations only for the reproducibility of visualizations. Our error-bounding algorithm works under any other configurations.

Single Linear ODE with Constant Coefficients

Here, we study three equations v′′+3v′+2v=f(t)v''+3v'+2v=f(t), v′′+v=g(t)v''+v=g(t), and v′′−v′=h(t)v''-v'=h(t), whose characteristic roots are {−1,−2}\{-1,-2\}, {±i}\{\pm i\}, and {0,1}\{0,1\} respectively. By § 4.2, the first two equations can be bounded with either Alg. 1 or Alg. 2, while the last must be bounded with Alg. 2.

They all satisfy initial conditions v(0)=v′(0)=1v(0)=v'(0)=1. We pick f(t)=2t2+8t+7f(t)=2t^2+8t+7, g(t)=t2+t+3g(t)=t^2+t+3, and h(t)=1−2th(t)=1-2t, so that the manufactured solution is v(t)=t2+t+1v(t)=t^2+t+1 for all three equations. Figure 1 shows that both BlooseB_{\mathrm{loose}} (Alg. 1) and BtightB_{\mathrm{tight}} (Alg. 2) strictly bounds the absolute error.

Loose and tight error bounds for three second-order linear ODEs

Figure 1. Loose bound (Alg. 1) and tight bound (Alg. 2) for 3 second-order linear ODE with constant coefficients. Notice that the loose bound cannot be applied to the third equation since it has characteristic roots with positive real part.

Linear ODE System with Constant Coefficients

In this subsection, we train 6 networks to solve a 6-dimensional linear system of ODEs with constant coefficients, namely, ddtv+Av=f\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{v}+A\mathbf{v}=\mathbf{f}. We pick A=PJP−1A=PJP^{-1} where

J=(J1J2J3)J=\begin{pmatrix}J_1&\\&J_2&\\&&J_3\end{pmatrix}

with J1=(41414)J_1=\begin{pmatrix}4&1\\&4&1\\&&4\end{pmatrix}, J2=(313)J_2=\begin{pmatrix}3&1\\&3\end{pmatrix}, J3=2J_3=2, and PP is a random orthogonal matrix.

We pick the initial conditions to be v(0)=P(001011)T\mathbf{v}(0)=P(001011)^T and the forcing function to be f(t)=P(cos⁡t+4sin⁡t+ln⁡(1+t),11+t+4ln⁡(1+t)+(t+1),4t+5,2t+3t2+et,4et,2cos⁡t−sin⁡t)T\mathbf{f}(t)=P(\cos t+4\sin t+\ln(1+t),\frac{1}{1+t}+4\ln(1+t)+(t+1),4t+5,2t+3t^2+e^t,4e^t,2\cos t-\sin t)^T, so that the manufactured exact solution is v(t)=P(sin⁡t,ln⁡(t+1),t+1,t2,et,cos⁡t)T\mathbf{v}(t)=P(\sin t,\ln(t+1),t+1,t^2,e^t,\cos t)^T.

After obtaining the residual information r(t)=ddtu(t)+Au(t)−f(t)\mathbf{r}(t)=\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{u}(t)+A\mathbf{u}(t)-\mathbf{f}(t), we apply Alg. 3 to obtain componentwise bound and norm bound of η=u−v\boldsymbol{\eta}=\mathbf{u}-\mathbf{v}. It is shown in Figure 2 that the bounds hold over the domain.

Componentwise and norm error bounds for a linear ODE system

Figure 2. Componentwise bound (upper) and norm bound (lower) for linear ODE system with constant coefficients

Nonlinear ODE – Duffing Equation

In this subsection, we consider a Duffing oscillator, which is characterized by the following 2nd order nonlinear ODE:

d2vdt2+3dvdt+2v+εv3=cos⁡t,(32)\frac{\mathrm{d}^{2}v}{\mathrm{d}t^{2}}+3\frac{\mathrm{d}v}{\mathrm{d}t}+2v+\varepsilon v^{3}=\cos t, \tag*{(32)}

under initial conditions v(0)=1v(0)=1 and v′(0)=1v'(0)=1, where ε\varepsilon controls the nonlinearity of the equation. Using Alg. 4, we solve the equation on I=[0,2]I=[0,2] for linspaced ε∈(−0.9,0.9)\varepsilon\in(-0.9,0.9) using neural networks and bound the errors. The input JJ to Alg. 4 is chosen to be 6. Namely, we expand the solution and bound components from degree 0 to 6.

The analytical solution to (66) is complicated. Hence, we use the RKF4(5) method to compute numerical solutions that are close enough to exact solutions for visualization purposes. See Figure 3 for network solutions against RKF4(5) solutions and Figure 4 for error bounds against absolute error.

RKF45 and Network Solutions

Figure 3. RKF45 and Network Solutions (max-degree 0∼6) to Duffing equation 32 for ε∈(−0.9,0.9)\varepsilon\in(-0.9,0.9)

True Error vs. error bound

Figure 4. True Error vs. error bound (max-degree 0∼6) of neural network solution to Duffing Equation 32 for ε∈(−0.9,0.9)\varepsilon\in(-0.9,0.9)

Linear PDE System with Nonconstant Coefficients

PDE Error Bound Evaluation Using Alg. 6

We try to solve the following first-order linear PDE,

(−x−y)∂v∂x+(x−y)∂v∂y+v=3x−2y(33)(-x-y)\frac{\partial v}{\partial x}+(x-y)\frac{\partial v}{\partial y}+v=3x-2y \tag*{(33)}

in spatial domain Ω=[−1,1]2\Omega=[-1,1]^{2}. The boundary constraints are v(x,±1)=2x±3v(x,\pm1)=2x\pm3 and v(±1,y)=3y±2v(\pm1,y)=3y\pm2. The manufactured solution is given by v(x,y)=2x+3yv(x,y)=2x+3y. The characteristic curves are integral curves CC:

x′(s)=−x−yy′(s)=x−y\begin{aligned} x'(s)&=-x-y\\ y'(s)&=x-y \end{aligned}

or

x(s)=R0e−scos⁡(s+θ0)y(s)=R0e−ssin⁡(s+θ0)\begin{aligned} x(s)&=R_{0}e^{-s}\cos(s+\theta_{0})\\ y(s)&=R_{0}e^{-s}\sin(s+\theta_{0}) \end{aligned}

where R0=x02+y02R_{0}=\sqrt{x_{0}^{2}+y_{0}^{2}} and θ0=atan2⁡(y0,x0)\theta_{0}=\operatorname{atan2}(y_{0},x_{0}) are constants determined by the starting point (x0,y0)∈Γ=∂Ω(x_{0},y_{0})\in\Gamma=\partial\Omega. See Figure 5 for visualization.

Characteristics curves of Eq. 33 and Eq. 34

Figure 5. Characteristics curves of Eq. 33 (left) and Eq. 34 (right). The red curves, with staring points A to P, are selected for visualization of absolute error and error bound in Fig. 6.

Since the analytical expression of the characteristic curves is known, Alg. 6 can be applied to evaluate the bound on each curve. We choose 16 characteristic curves with starting points A,B,…,PA,B,\ldots,P, equidistantly placed on the boundary (Fig. 5a). We plot the absolute error and the computed error bound along these characteristic curves in Fig. 6. It can be seen that absolute error lies strictly within the bounds.

PDE Error Bound Evaluation Using Alg. 5

Consider the following PDE

(x2+y2+1)∂v∂x+(x2−y2+2)∂v∂y+(3−2x)v=f(34)(x^{2}+y^{2}+1)\frac{\partial v}{\partial x}+(x^{2}-y^{2}+2)\frac{\partial v}{\partial y}+(3-2x)v=f \tag*{(34)}

over domain Ω=[−1,1]2\Omega=[-1,1]^{2} where f(x,y)=6−4xf(x,y)=6-4x. The boundary constraints are v(−1,y)=2v(-1,y)=2 and $v(x,1)=2, and the manufactured solution is v(x,y)=2v(x,y)=2.

Absolute error and error bound on selected characteristic curves

Figure 6. Absolute error and error bound on selected characteristic curves. These characteristic curve start at points AA through PP as shown in Figure 5a. The blue solid curves are absolute error along the characteristic curves and red dotted curves are corresponding bounds.

The characteristic curves C\mathcal{C}:

{x′(s)=x2+y2+1y′(s)=x2−y2+2\begin{cases} x'(s)=x^2+y^2+1 \\ y'(s)=x^2-y^2+2 \end{cases}

are given by a nonlinear ODE, which is hard to solve analytically. (See Figure 5b for visualization) Therefore, Alg. 6 cannot be applied to evaluate the error bound.

However, the coefficient (3−2x)(3-2x) is nonzero over domain Ω\Omega. Hence, we can use Alg. 5 to compute a constant error bound ∣η(x,y)∣≤B(x,y)≡B|\eta(x,y)| \le B(x,y) \equiv B for all (x,y)∈Ω(x,y) \in\Omega. We visualize the bound and the maximum absolute error max⁡(x,y)∈Ω∣η∣\max_{(x,y)\in\Omega}|\eta| after each training epoch in Figure 7. As expected, the bound is loose, which is about an order of magnitude larger than the max absolute error. Yet, it consistently holds true for every epoch, even during the early stages of training, when the network performs poorly.

Constant bound and maximum absolute error over training epochs

Figure 7. Constant bound BB, computed using Alg. 5, and max absolute error over domain at different epochs of training.

Conclusion and Future Work

This paper proposes various error-bounding algorithms for any PINN solution to certain classes of ODEs and PDEs.

These algorithms only require the residual information r(⋅)r(\cdot) and the equation structure Dv=fDv=f as input. There are many real-world applications for which the exact solution v(⋅)v(\cdot) is unknown or hard to compute. However, the residual information r(⋅)r(\cdot) is usually, if not always, readily available. With our proposed algorithms, PINNs can be trained until the error is gauranteed to fall below a specified tolerance threshold. The mathematical relationship between residual and error bound also sheds light on optimizing PINN solutions for future studies.

The error-bounding algorithms proposed in this paper only apply to certain classes of ODEs and PDEs. However, the insights of this paper can be beneficial to future work that extends to more general classes of ODEs and PDEs, especially nonlinear ones. We also plan to apply these algorithms to stochastic differential equations, where the error bound is a probabilistic tail bound.

Residual-Based Error Bound for Physics-Informed Neural Networks (Supplementary Material)

Shuheng Liu; Xiyue Huang; Pavlos Protopapas

Proof of Propositions in Section 5.1.1

In this part, we first discuss the properties of the operator Lλ\mathcal{L}_{\lambda}, which is defined in the main paper. We then use these properties to prove relevant statements regarding Alg. 1 and Alg. 2 in Section 5.1.1 of the main paper.

Properties of Inverse Operator Iλ=Lλ−1\mathcal{I}_{\lambda} = \mathcal{L}_{\lambda}^{-1}

Let Lλ\mathcal{L}_{\lambda} (λ∈C\lambda\in\mathbb{C}) be the differential operator Lλϕ:=dϕdt−λϕ\mathcal{L}_{\lambda}\phi:= \frac{\mathrm{d}\phi}{\mathrm{d}t} - \lambda\phi. The inverse of Lλϕ=ψ\mathcal{L}_{\lambda}\phi= \psi is given by ϕ=Iλψ\phi= \mathcal{I}_{\lambda}\psi if ϕ(0)=0\phi(0) = 0, where

Iλψ(t):=eλt∫0te−λτψ(τ) dτ.(35)\mathcal{I}_{\lambda}\psi(t) := e^{\lambda t}\int_{0}^{t}e^{-\lambda\tau}\psi(\tau)\,\mathrm{d}\tau. \tag*{(35)}

In addition to Iλ=Lλ−1\mathcal{I}_{\lambda} = \mathcal{L}_{\lambda}^{-1}, there are a few properties of operator Iλ\mathcal{I}_{\lambda} that we are interested in

  1. Linearity: Iλ(c1ψ1+c2ψ2)=c1Iψ1+c2Iψ2\mathcal{I}_{\lambda}(c_{1}\psi_{1} + c_{2}\psi_{2}) = c_{1}\mathcal{I}\psi_{1} + c_{2}\mathcal{I}\psi_{2} for all functions ψ1,ψ2\psi_{1}, \psi_{2} and constants c1,c2∈Cc_{1}, c_{2} \in\mathbb{C}

  2. Monotonicity: For λ∈R\lambda\in\mathbb{R}, there is (∀t∈I,ψ1(t)≤ψ2(t))⟹(∀t∈I,Lλψ1(t)≤Lλψ2(t))(\forall t \in I, \psi_{1}(t) \le\psi_{2}(t)) \Longrightarrow(\forall t \in I, \mathcal{L}_{\lambda}\psi_{1}(t) \le\mathcal{L}_{\lambda}\psi_{2}(t)),

  3. Commutativity: Iλ1∘Iλ2=Iλ2∘Iλ1\mathcal{I}_{\lambda_{1}} \circ\mathcal{I}_{\lambda_{2}} = \mathcal{I}_{\lambda_{2}} \circ\mathcal{I}_{\lambda_{1}} for all λ1,λ2∈C\lambda_{1}, \lambda_{2} \in\mathbb{C}. This can be shown because Lλ1∘Lλ2=Lλ2∘Lλ1\mathcal{L}_{\lambda_{1}} \circ\mathcal{L}_{\lambda_{2}} = \mathcal{L}_{\lambda_{2}} \circ\mathcal{L}_{\lambda_{1}}. Therefore, the inverse operators Iλ2∘Iλ1\mathcal{I}_{\lambda_{2}} \circ\mathcal{I}_{\lambda_{1}} Iλ1∘Iλ2\mathcal{I}_{\lambda_{1}} \circ\mathcal{I}_{\lambda_{2}} must also be equal.

  4. Absolute Inequality: ∣Iλψ(t)∣≤IRe⁡(λ)∣ψ(t)∣|\mathcal{I}_{\lambda}\psi(t)| \le\mathcal{I}_{\operatorname{Re}(\lambda)}|\psi(t)|, which we prove in the next subsection.

Proof of Operator Inequality ∣Iλψ∣≤IRe⁡(λ)∣ψ∣|\mathcal{I}_{\lambda}\psi| \le\mathcal{I}_{\operatorname{Re}(\lambda)}|\psi|

Proposition For any λ∈C\lambda\in\mathbb{C} and scalar function ψ:R+→C\psi: \mathbb{R}^{+} \to\mathbb{C}, there is

∣Iλψ(t)∣≤IRe⁡(λ)∣ψ(t)∣.(36)|\mathcal{I}_{\lambda}\psi(t)| \le\mathcal{I}_{\operatorname{Re}(\lambda)}|\psi(t)|. \tag*{(36)}

Proof Let ϕ=Iλψ\phi= \mathcal{I}_{\lambda}\psi. Since L=I−1\mathcal{L} = \mathcal{I}^{-1}, the problem is equivalent to proving ∣ϕ∣≤IRe⁡(λ)∣ψ∣|\phi| \le\mathcal{I}_{\operatorname{Re}(\lambda)}|\psi|, where

ddtϕ−λϕ=ψ.(37)\frac{\mathrm{d}}{\mathrm{d}t}\phi- \lambda\phi= \psi. \tag*{(37)}

To see this, we multiply both sides with an integrating factor e−λte^{-\lambda t} and integrate from 00 to tt,

∫0te−λτ(ddτϕ(τ)−λϕ(τ)) dτ=∫0te−λτψ(τ) dτ(38)\int_{0}^{t}e^{-\lambda\tau}\left(\frac{\mathrm{d}}{\mathrm{d}\tau}\phi(\tau) - \lambda\phi(\tau)\right)\,\mathrm{d}\tau= \int_{0}^{t}e^{-\lambda\tau}\psi(\tau)\,\mathrm{d}\tau \tag*{(38)}
e−λtϕ(t)−ϕ(0)=∫0te−λτψ(τ) dτ(39)e^{-\lambda t}\phi(t) - \phi(0) = \int_{0}^{t}e^{-\lambda\tau}\psi(\tau)\,\mathrm{d}\tau \tag*{(39)}

Since ϕ=Iλψ\phi=\mathcal{I}_{\lambda}\psi, there is ϕ(0)=0\phi(0)=0. Hence we have

ϕ(t)=eλt∫0te−λτψ(τ) dτ(40)\phi(t)=e^{\lambda t}\int_{0}^{t}e^{-\lambda\tau}\psi(\tau)\,\mathrm{d}\tau \tag*{(40)}
∣ϕ(t)∣=∣eλt∫0te−λτψ(τ) dτ∣(41)|\phi(t)|=\left|e^{\lambda t}\int_{0}^{t}e^{-\lambda\tau}\psi(\tau)\,\mathrm{d}\tau\right| \tag*{(41)}
≤(42)\leq \tag*{(42)}

For λ∈C\lambda\in\mathbb{C}, there is ∣e±λt∣=e±Re⁡(λ)t|e^{\pm\lambda t}|=e^{\pm\operatorname{Re}(\lambda)t}, where Re⁡(λ)\operatorname{Re}(\lambda) is the real part of λ\lambda. Hence,

∣ϕ(t)∣=eRe⁡(λ)t∣∫0te−λτψ(τ) dτ∣(43)|\phi(t)|=e^{\operatorname{Re}(\lambda)t}\left|\int_{0}^{t}e^{-\lambda\tau}\psi(\tau)\,\mathrm{d}\tau\right| \tag*{(43)}
≤eRe⁡(λ)t∫0t∣e−λτψ(τ)∣ dτ(44)\leq e^{\operatorname{Re}(\lambda)t}\int_{0}^{t}|e^{-\lambda\tau}\psi(\tau)|\,\mathrm{d}\tau \tag*{(44)}
=eRe⁡(λ)t∫0te−Re⁡(λ)τ∣ψ(τ)∣ dτ=:IRe⁡(λ)∣ψ(t)∣(45)=e^{\operatorname{Re}(\lambda)t}\int_{0}^{t}e^{-\operatorname{Re}(\lambda)\tau}|\psi(\tau)|\,\mathrm{d}\tau=:\mathcal{I}_{\operatorname{Re}(\lambda)}|\psi(t)| \tag*{(45)}

Proof of tight and loose bounds

This section proves inequality 11 in the main paper, namely,

∣η(t)∣≤Btight(t):=(IRe⁡(λ1)∘⋯∘IRe⁡(λn))r(t)(46)|\eta(t)|\leq B_{\mathit{tight}}(t):=(\mathcal{I}_{\operatorname{Re}(\lambda_1)}\circ\cdots\circ\mathcal{I}_{\operatorname{Re}(\lambda_n)})r(t) \tag*{(46)}

and, if Re⁡(λj)≤0\operatorname{Re}(\lambda_j)\leq0 for all λj\lambda_j,

Btight(t)≤Bloose(t):=1Z!∏j=1Re⁡(λj)≠0n1Re⁡(−λj)max⁡τ∈I∣r(τ)∣ tZ,(47)B_{\mathit{tight}}(t)\leq B_{\mathit{loose}}(t):=\frac{1}{Z!}\prod_{\substack{j=1\\\operatorname{Re}(\lambda_j)\ne0}}^{n}\frac{1}{\operatorname{Re}(-\lambda_j)}\max_{\tau\in I}|r(\tau)|\,t^{Z}, \tag*{(47)}

where ZZ is the number λj\lambda_j whose real part is 0.

Proof For any linear differential operator L=dndtn+an−1dn−1dtn−1+⋯+a0\mathcal{L}=\frac{\mathrm{d}^{n}}{\mathrm{d}t^{n}}+a_{n-1}\frac{\mathrm{d}^{n-1}}{\mathrm{d}t^{n-1}}+\cdots+a_0 whose coefficients {aj}j=0n−1\{a_j\}_{j=0}^{n-1} satisfy

λn+an−1λn−1+⋯+a0=∏j=1n(λ−λj),\lambda^n+a_{n-1}\lambda^{n-1}+\cdots+a_0=\prod_{j=1}^{n}(\lambda-\lambda_j),

it can be verified that L=Lλ1∘⋯∘Lλn\mathcal{L}=\mathcal{L}_{\lambda_1}\circ\cdots\circ\mathcal{L}_{\lambda_n}, where Lλjϕ:=dϕdt−λjϕ\mathcal{L}_{\lambda_j}\phi:=\frac{\mathrm{d}\phi}{\mathrm{d}t}-\lambda_j\phi as defined in appendix A.1. Then by the proposition in appendix A.1, the inverse operator is given by

L−1=(Lλ1∘⋯∘Lλn)−1=Lλn−1∘⋯∘Lλ1−1=Iλn∘⋯∘Iλ1(48)\mathcal{L}^{-1}=(\mathcal{L}_{\lambda_1}\circ\cdots\circ\mathcal{L}_{\lambda_n})^{-1}=\mathcal{L}_{\lambda_n}^{-1}\circ\cdots\circ\mathcal{L}_{\lambda_1}^{-1}=\mathcal{I}_{\lambda_n}\circ\cdots\circ\mathcal{I}_{\lambda_1} \tag*{(48)}

Through repeated application of Inequality 2, we can prove (46)

∣η(t)∣=∣L−1r(t)∣(49)|\eta(t)|=|\mathcal{L}^{-1}r(t)| \tag*{(49)}
=∣(Iλn∘⋯∘Iλ1)r(t)∣(50)=|(\mathcal{I}_{\lambda_n}\circ\cdots\circ\mathcal{I}_{\lambda_1})r(t)| \tag*{(50)}
=∣Iλn(Iλn−1∘⋯∘Iλ1)r(t)∣(51)=|\mathcal{I}_{\lambda_n}(\mathcal{I}_{\lambda_{n-1}}\circ\cdots\circ\mathcal{I}_{\lambda_1})r(t)| \tag*{(51)}
≤IRe⁡(λn)∣(Iλn−1∘⋯∘Iλ1)r(t)∣(52)\leq\mathcal{I}_{\operatorname{Re}(\lambda_n)}|(\mathcal{I}_{\lambda_{n-1}}\circ\cdots\circ\mathcal{I}_{\lambda_1})r(t)| \tag*{(52)}
≤(IRe⁡(λn)∘IRe⁡(λn−1))∣(Iλn−2∘⋯∘Iλ1)r(t)∣≤⋯(53)\begin{aligned} \leq(\mathcal{I}_{\operatorname{Re}(\lambda_n)}\circ\mathcal{I}_{\operatorname{Re}(\lambda_{n-1})})|(\mathcal{I}_{\lambda_{n-2}}\circ\cdots\circ\mathcal{I}_{\lambda_1})r(t)| \\ &\leq\cdots \tag*{(53)} \end{aligned}
≤(IRe⁡(λn)∘⋯∘IRe⁡(λ1))∣r(t)∣=:Btight(t).(54)\leq(\mathcal{I}_{\operatorname{Re}(\lambda_n)}\circ\cdots\circ\mathcal{I}_{\operatorname{Re}(\lambda_1)})|r(t)|=:B_{\mathit{tight}}(t). \tag*{(54)}

In order to prove (47), consider the cases Re⁡(λ)<0\operatorname{Re}(\lambda)<0 and Re⁡(λ)=0\operatorname{Re}(\lambda)=0 separately.

  • If Re⁡(λ)<0\operatorname{Re}(\lambda)<0, for any constant c∈R+c\in\mathbb{R}^{+}, there is

IRe⁡(λ)[c]=eRe⁡(λ)t∫0tce−Re⁡(λ)τ dτ=c−Re⁡(λ)(1−eRe⁡(λ)t)≤c−Re⁡(λ)for t≥0(55)\mathcal{I}_{\operatorname{Re}(\lambda)}[c]=e^{\operatorname{Re}(\lambda)t}\int_{0}^{t}ce^{-\operatorname{Re}(\lambda)\tau}\,\mathrm{d}\tau=\frac{c}{-\operatorname{Re}(\lambda)}\left(1-e^{\operatorname{Re}(\lambda)t}\right)\leq\frac{c}{-\operatorname{Re}(\lambda)}\quad\text{for }t\geq0 \tag*{(55)}
  • If Re⁡(λ)=0\operatorname{Re}(\lambda)=0, for any monomial ctmct^{m}, there is

IRe⁡(λ)[ctm]=I0[ctm]=∫0tcτm dτ=cm+1tm+1for t>0(56)\mathcal{I}_{\operatorname{Re}(\lambda)}[ct^{m}]=\mathcal{I}_{0}[ct^{m}]=\int_{0}^{t}c\tau^{m}\,\mathrm{d}\tau=\frac{c}{m+1}t^{m+1}\quad\text{for }t>0 \tag*{(56)}

Let Rmax⁡:=max⁡t∈I∣r(t)∣R_{\max}:=\max_{t\in I}|r(t)| be the max absolute residual. Let Z=∣{λj:Re⁡(λj)=0,1≤j≤n}∣Z=|\{\lambda_j:\operatorname{Re}(\lambda_j)=0,1\leq j\leq n\}|. Assume without loss of generality that Re⁡(λ1),…,Re⁡(λn−Z)<0\operatorname{Re}(\lambda_1),\ldots,\operatorname{Re}(\lambda_{n-Z})<0 and that Re⁡(λn−Z+1)=⋯=Re⁡(λn)=0\operatorname{Re}(\lambda_{n-Z+1})=\cdots=\operatorname{Re}(\lambda_n)=0. By the monotonicity of operator IRe⁡(λ)\mathcal{I}_{\operatorname{Re}(\lambda)}, there is IRe⁡(λ)φ1(t)≤IRe⁡(λ)φ2(t)\mathcal{I}_{\operatorname{Re}(\lambda)}\varphi_1(t)\leq\mathcal{I}_{\operatorname{Re}(\lambda)}\varphi_2(t) if φ1(t)≤φ2(t)\varphi_1(t)\leq\varphi_2(t) for all t∈It\in I. Hence,

Btight(t)=(IRe⁡(λn)∘⋯∘IRe⁡(λ1))∣r(t)∣(57)\mathcal{B}_{\mathrm{tight}}(t)=\left(\mathcal{I}_{\operatorname{Re}(\lambda_n)}\circ\cdots\circ\mathcal{I}_{\operatorname{Re}(\lambda_1)}\right)|r(t)| \tag*{(57)}
≤(IRe⁡(λn)∘⋯∘IRe⁡(λ1))Rmax⁡(58)\leq\left(\mathcal{I}_{\operatorname{Re}(\lambda_n)}\circ\cdots\circ\mathcal{I}_{\operatorname{Re}(\lambda_1)}\right)R_{\max} \tag*{(58)}
≤(IRe⁡(λn)∘⋯∘IRe⁡(λ2))1−Re⁡(λ1)Rmax⁡≤⋯(59)\begin{aligned} \leq\left(\mathcal{I}_{\operatorname{Re}(\lambda_n)}\circ\cdots\circ\mathcal{I}_{\operatorname{Re}(\lambda_2)}\right)\frac{1}{-\operatorname{Re}(\lambda_1)}R_{\max} \\ &\leq\cdots \tag*{(59)} \end{aligned}
≤(IRe⁡(λn)∘⋯∘IRe⁡(λn−Z+1))∏j=1n−Z1−Re⁡(λj)Rmax⁡(60)\leq\left(\mathcal{I}_{\operatorname{Re}(\lambda_n)}\circ\cdots\circ\mathcal{I}_{\operatorname{Re}(\lambda_{n-Z+1})}\right)\prod_{j=1}^{n-Z}\frac{1}{-\operatorname{Re}(\lambda_j)}R_{\max} \tag*{(60)}
=I0Z[∏j=1Re⁡(λj)≠0n1−Re⁡(λj)Rmax⁡](61)=\mathcal{I}_{0}^{Z}\left[\prod_{\substack{j=1\\\operatorname{Re}(\lambda_j)\ne0}}^{n}\frac{1}{-\operatorname{Re}(\lambda_j)}R_{\max}\right] \tag*{(61)}
=1Z!∏j=1Re⁡(λj)≠0n1−Re⁡(λj)Rmax⁡tZ=:Bloose(t)(62)=\frac{1}{Z!}\prod_{\substack{j=1\\\operatorname{Re}(\lambda_j)\ne0}}^{n}\frac{1}{-\operatorname{Re}(\lambda_j)}R_{\max}t^{Z}=:\mathcal{B}_{\mathrm{loose}}(t) \tag*{(62)}

which proves Eq. 13.

Proof of Propositions in Section 5.1.3

In this part, we prove relevant statements regarding Alg. 3 in Section 5.1.1 of the main paper.

Consider the problem 12 in main paper. The error η\boldsymbol{\eta} of the network solution u\mathbf{u} satisfies the equation

ddtη+Aη=r(t)s.t.η(t=0)=0(63)\frac{\mathrm{d}}{\mathrm{d}t}\boldsymbol{\eta}+A\boldsymbol{\eta}=\mathbf{r}(t)\quad\text{s.t.}\quad\boldsymbol{\eta}(t=0)=0 \tag*{(63)}

where r(t)=ddtu(t)+Au(t)−f(t)\mathbf{r}(t)=\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{u}(t)+A\mathbf{u}(t)-\mathbf{f}(t) is the residual vector.

With the Jordan canonical form 13, we multiply both sides of Eq. 29 by P−1P^{-1},

P−1ddtη+P−1Aη=P−1r(t)(64)P^{-1}\frac{\mathrm{d}}{\mathrm{d}t}\boldsymbol{\eta}+P^{-1}A\boldsymbol{\eta}=P^{-1}\mathbf{r}(t) \tag*{(64)}
P−1ddtη+JP−1η=P−1r(t)(65)P^{-1}\frac{\mathrm{d}}{\mathrm{d}t}\boldsymbol{\eta}+JP^{-1}\boldsymbol{\eta}=P^{-1}\mathbf{r}(t) \tag*{(65)}
ddtδ+Jδ=q(t)(66)\frac{\mathrm{d}}{\mathrm{d}t}\boldsymbol{\delta}+J\boldsymbol{\delta}=\mathbf{q}(t) \tag*{(66)}

where δ(t):=P−1η(t)\boldsymbol{\delta}(t):=P^{-1}\boldsymbol{\eta}(t) and q(t)=P−1r(t)\mathbf{q}(t)=P^{-1}\mathbf{r}(t). Recall that JJ is a Jordan canonical form consisting of KK Jordan blocks. Each Jordan block JkJ_k (1≤k≤K1\leq k\leq K) is an nk×nkn_k\times n_k square matrix, with eigenvalue λk\lambda_k on its diagonal and 1 on its super-diagonal, where ∑k=1Knk=n\sum_{k=1}^{K} n_k=n. Expanding the vector notations, there is

ddt(δ1⋮δn1δn1δn1+1⋮δn1+n2δn1+n2+1⋮)+(J1000J2000⋱)(δ1⋮δn1δn1δn1+1⋮δn1+n2δn1+n2+1⋮)=(q1(t)⋮qn1(t)qn1+1(t)⋮qn1+n2(t)qn1+n2+1(t)⋮)(67)\frac{\mathrm{d}}{\mathrm{d}t} \begin{pmatrix} \delta_1\\ \vdots\\ \delta_{n_1}\\ \frac{\delta_{n_1}}{\delta_{n_1+1}}\\ \vdots\\ \frac{\delta_{n_1+n_2}}{\delta_{n_1+n_2+1}}\\ \vdots \end{pmatrix} + \begin{pmatrix} J_1 & 0 & 0\\ 0 & J_2 & 0\\ 0 & 0 & \ddots \end{pmatrix} \begin{pmatrix} \delta_1\\ \vdots\\ \delta_{n_1}\\ \frac{\delta_{n_1}}{\delta_{n_1+1}}\\ \vdots\\ \frac{\delta_{n_1+n_2}}{\delta_{n_1+n_2+1}}\\ \vdots \end{pmatrix} = \begin{pmatrix} q_1(t)\\ \vdots\\ \frac{q_{n_1}(t)}{q_{n_1+1}(t)}\\ \vdots\\ \frac{q_{n_1+n_2}(t)}{q_{n_1+n_2+1}(t)}\\ \vdots \end{pmatrix} \tag*{(67)}

Let Nk=n1+⋯+nkN_k=n_1+\cdots+n_k. For kk-th Jordan block indexed by Nk−1<l≤NkN_{k-1}<l\leq N_k, there is

ddt(δNk−1+1⋮δNk)+(λk1⋱⋱λk1λk)(δNk−1+1⋮δNk)=(qNk−1+1(t)⋮qNk(t)),(68)\frac{\mathrm{d}}{\mathrm{d}t} \begin{pmatrix} \delta_{N_{k-1}+1}\\ \vdots\\ \delta_{N_k} \end{pmatrix} + \begin{pmatrix} \lambda_k & 1 & & \\ & \ddots& \ddots& \\ & & \lambda_k & 1\\ & & & \lambda_k \end{pmatrix} \begin{pmatrix} \delta_{N_{k-1}+1}\\ \vdots\\ \delta_{N_k} \end{pmatrix} = \begin{pmatrix} q_{N_{k-1}+1}(t)\\ \vdots\\ q_{N_k}(t) \end{pmatrix}, \tag*{(68)}

which can be formulated as the following sequence of scalar equations, also known as Jordan chains:

ddtδNk−1+1+λkδNk−1+1=qNk−1+1−δNk−1+2,(69)\frac{\mathrm{d}}{\mathrm{d}t}\delta_{N_k-1+1}+\lambda_k\delta_{N_k-1+1}=q_{N_k-1+1}-\delta_{N_k-1+2}, \tag*{(69)}
ddtδNk−1+2+λkδNk−1+2=qNk−1+2−δNk−1+3,(70)\frac{\mathrm{d}}{\mathrm{d}t}\delta_{N_k-1+2}+\lambda_k\delta_{N_k-1+2}=q_{N_k-1+2}-\delta_{N_k-1+3}, \tag*{(70)}
⋮\vdots
ddtδNk−1+λkδNk−1=qNk−1−δNk,(71)\frac{\mathrm{d}}{\mathrm{d}t}\delta_{N_k-1}+\lambda_k\delta_{N_k-1}=q_{N_k-1}-\delta_{N_k}, \tag*{(71)}
ddtδNk+λkδNk=qNk.(72)\frac{\mathrm{d}}{\mathrm{d}t}\delta_{N_k}+\lambda_k\delta_{N_k}=q_{N_k}. \tag*{(72)}

The last equation (Eq. (72)) of the Jordan chain can be used to bound δNk\delta_{N_k} by applying the inequality 2,

∣δNk∣=∣I−λkqNk∣≤I−Re⁡(λk)∣qNk∣(73)\lvert\delta_{N_k}\rvert=\lvert\mathcal{I}_{-\lambda_k}q_{N_k}\rvert\leq\mathcal{I}_{-\operatorname{Re}(\lambda_k)}\lvert q_{N_k}\rvert \tag*{(73)}

Applying the inequality 2 again to Eq. (71), there is

∣δNk−1∣=∣I−λk(qNk−1+δNk)∣(74)\lvert\delta_{N_k-1}\rvert=\lvert\mathcal{I}_{-\lambda_k}(q_{N_k-1}+\delta_{N_k})\rvert \tag*{(74)}
≤I−Re⁡(λk)∣qNk−1−δNk∣(75)\leq\mathcal{I}_{-\operatorname{Re}(\lambda_k)}\lvert q_{N_k-1}-\delta_{N_k}\rvert \tag*{(75)}
≤I−Re⁡(λk)∣qNk−1∣+I−Re⁡(λk)∣δNk∣(76)\leq\mathcal{I}_{-\operatorname{Re}(\lambda_k)}\lvert q_{N_k-1}\rvert+\mathcal{I}_{-\operatorname{Re}(\lambda_k)}\lvert\delta_{N_k}\rvert \tag*{(76)}
≤I−Re⁡(λk)∣qNk−1∣+I−Re⁡(λk)2∣qNk∣.(77)\leq\mathcal{I}_{-\operatorname{Re}(\lambda_k)}\lvert q_{N_k-1}\rvert+\mathcal{I}^{2}_{-\operatorname{Re}(\lambda_k)}\lvert q_{N_k}\rvert. \tag*{(77)}

The first inequality is a direct application of Eq. 2. The second inequality is based on linearity of the operator I\mathcal{I} and the triangle inequality. The third inequality is obtained by substituting Eq. (73). Here the superscript in I2\mathcal{I}^{2} denotes compositional square I2=I∘I\mathcal{I}^{2}=\mathcal{I}\circ\mathcal{I}.

By induction, for the kk-th Jordan block (Nk−1<l≤NkN_{k-1}<l\leq N_k), there is

∣δl∣≤∑j=0Nk−lI−Re⁡(λk)j+1∣ql+j∣(78)\lvert\delta_l\rvert\leq\sum_{j=0}^{N_k-l}\mathcal{I}^{j+1}_{-\operatorname{Re}(\lambda_k)}\lvert q_{l+j}\rvert \tag*{(78)}

We use this inequality to bound the norm of the error vector, ∥η∥\lVert\eta\rVert, as well as absolute value of each component, ∣(η)l∣\lvert(\eta)_l\rvert.

Componentwise Bound

Using matrix notations, Eq. (78) can be rewritten as

δ∣⋅∣⪯Iq∣⋅∣(79)\delta^{|\cdot|} \preceq\mathcal{I}q^{|\cdot|} \tag*{(79)}

where ⪯\preceq denotes componentwise inequality, the superscript ∣⋅∣|\cdot| denotes componentwise absolute value, and I\mathcal{I} is defined as operator matrix $\mathcal{I}:=

(I1I2⋱)\begin{pmatrix} I_1 \\ I_2 \\ \ddots \end{pmatrix}

where each $I_k=

(I−Re⁡(λk)I−Re⁡(λk)2⋯I−Re⁡(λk)nk0I−Re⁡(λk)⋯I−Re⁡(λk)nk−1⋮⋮⋱⋮00⋯I−Re⁡(λk))\begin{pmatrix} \mathcal{I}_{-\operatorname{Re}(\lambda_k)} & \mathcal{I}^{2}_{-\operatorname{Re}(\lambda_k)} & \cdots& \mathcal{I}^{n_k}_{-\operatorname{Re}(\lambda_k)} \\ 0 & \mathcal{I}_{-\operatorname{Re}(\lambda_k)} & \cdots& \mathcal{I}^{n_k-1}_{-\operatorname{Re}(\lambda_k)} \\ \vdots& \vdots& \ddots& \vdots\\ 0 & 0 & \cdots& \mathcal{I}_{-\operatorname{Re}(\lambda_k)} \end{pmatrix}

is an nk×nkn_k \times n_k upper-triangular block. Notice that (AB)∣⋅∣⪯A∣⋅∣B∣⋅∣(AB)^{|\cdot|} \preceq A^{|\cdot|}B^{|\cdot|} for any compatible matrices AA and BB. Recall δ(t)=P−1η(t)\delta(t)=P^{-1}\eta(t) and q(t)=P−1r(t)q(t)=P^{-1}r(t), there is

η∣⋅∣⪯P∣⋅∣δ∣⋅∣⪯P∣⋅∣I[q∣⋅∣]⪯P∣⋅∣I[(P−1)∣⋅∣r∣⋅∣](80)\eta^{|\cdot|} \preceq P^{|\cdot|}\delta^{|\cdot|} \preceq P^{|\cdot|}\mathcal{I}\left[q^{|\cdot|}\right] \preceq P^{|\cdot|}\mathcal{I}\left[\left(P^{-1}\right)^{|\cdot|}r^{|\cdot|}\right] \tag*{(80)}

Norm Bound

By Eq. (79), we have ∥δ∥≤∥I[∥q∥1]∥\lVert\delta\rVert\leq\lVert\mathcal{I}[\lVert q\rVert\mathbf{1}]\rVert, where 1\mathbf{1} is an n×1n \times1 (constant) column vector whose components are all equal to 1.

With η=Pδ\eta=P\delta and q=P−1rq=P^{-1}r, there is ∥η∥≤∥P∥∥δ∥\lVert\eta\rVert\leq\lVert P\rVert\lVert\delta\rVert and ∥q∥≤∥P−1∥∥r∥\lVert q\rVert\leq\lVert P^{-1}\rVert\lVert r\rVert, where ∥⋅∥\lVert\cdot\rVert denotes the norm of a vector or the induced norm of a matrix. Consequently,

∥η(t)∥≤∥P∥∥δ(t)∥(81)\lVert\eta(t)\rVert\leq\lVert P\rVert\lVert\delta(t)\rVert \tag*{(81)}
≤∥P∥∥I[∥q(t)∥1]∥(82)\leq\lVert P\rVert\left\lVert\mathcal{I}\left[\lVert q(t)\rVert\mathbf{1}\right]\right\rVert \tag*{(82)}
≤∥P∥∥I[∥P−1∥∥r∥1]∥(83)\leq\lVert P\rVert\left\lVert\mathcal{I}\left[\lVert P^{-1}\rVert\lVert r\rVert\mathbf{1}\right]\right\rVert \tag*{(83)}
≤∥P∥∥P−1∥∥I[∥r∥1]∥(84)\leq\lVert P\rVert\lVert P^{-1}\rVert\left\lVert\mathcal{I}\left[\lVert r\rVert\mathbf{1}\right]\right\rVert \tag*{(84)}
=cond⁡(P)∥I[∥r(t)∥1]∥(85)= \operatorname{cond}(P)\left\lVert\mathcal{I}\left[\lVert r(t)\rVert\mathbf{1}\right]\right\rVert \tag*{(85)}

References

  1. [1]Feiyu Chen, David Sondak, Pavlos Protopapas, Marios Mattheakis, Shuheng Liu, Devansh Agarwal, and Marco Di Giovanni. Neurodiffeq: A python package for solving differential equations with neural networks. Journal of Open Source Software, 5(46):1931, 2020.DOI
  2. [2]Tim De Ryck and Siddhartha Mishra. Error analysis for deep neural network approximations of parametric hyperbolic conservation laws. arXiv preprint arXiv:2207.07362, 2022a.arxiv.org/abs/2207.07362
  3. [3]Tim De Ryck and Siddhartha Mishra. Error analysis for physics-informed neural networks (PINNs) approximating Kolmogorov PDEs. Advances in Computational Mathematics, 48(6):1–40, 2022b.arxiv.org/abs/2106.14473
  4. [4]Tim De Ryck and Siddhartha Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. arXiv preprint arXiv:2205.11393, 2022c.arxiv.org/abs/2205.11393
  5. [5]Shaan Desai, Marios Mattheakis, Hayden Joy, Pavlos Protopapas, and Stephen Roberts. One-shot transfer learning of physics-informed neural networks. 2021.arxiv.org/abs/2110.11286
  6. [6]Cedric Flamant, Pavlos Protopapas, and David Sondak. Solving differential equations using neural network solution bundles. arXiv preprint arXiv:2006.14372, 2020.arxiv.org/abs/2006.14372
  7. [7]Olga Graf, Pablo Flores, Pavlos Protopapas, and Karim Pichara. Uncertainty quantification in neural differential equations. arXiv preprint arXiv:2111.04207, 2021.arxiv.org/abs/2111.04207
  8. [8]Mengwu Guo and Ehsan Haghighat. Energy-based error bound of physics-informed neural network solutions in elasticity. Journal of Engineering Mechanics, 148(8):04022038, 2022.DOI
  9. [9]Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.DOI
  10. [10]Michael A Jenkins and Joseph F Traub. A three-stage variable-shift iteration for polynomial zeros and its relation to generalized rayleigh iteration. Numerische Mathematik, 14(3):252–263, 1970.
  11. [11]Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34, 2021.arxiv.org/abs/2109.01050
  12. [12]Pola Lydia Lagari, Lefteri H Tsoukalas, Salar Safarkhani, and Isaac E Lagaris. Systematic construction of neural forms for solving partial differential equations inside rectangular domains, subject to initial, boundary and interface conditions. International Journal on Artificial Intelligence Tools, 29(05):2050009, 2020.DOI
  13. [13]Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987–1000, 1998.DOI
  14. [14]Isaac E Lagaris, Aristidis C Likas, and Dimitris G Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11(5):1041–1049, 2000.DOI
  15. [15]Marios Mattheakis, David Sondak, Akshunna S Dogra, and Pavlos Protopapas. Hamiltonian neural networks for solving differential equations. arXiv preprint arXiv:2001.11107, 2020.DOI
  16. [16]Kevin Stanley McFall and James Robert Mahan. Artificial neural network method for solution of boundary value problems with exact satisfaction of arbitrary boundary conditions. IEEE Transactions on Neural Networks, 20 (8):1221–1233, 2009.DOI
  17. [17]Kshitij Parwani and Pavlos Protopapas. Adversarial sampling for solving differential equations with neural networks. arXiv preprint arXiv:2111.12024, 2021.arxiv.org/abs/2111.12024
  18. [18]Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.DOI
  19. [19]Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339–1364, 2018.DOI
  20. [20]N Sukumar and Ankit Srivastava. Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks. arXiv preprint arXiv:2104.08426, 2021.DOI
  21. [21]Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. arXiv preprint arXiv:1710.00211, 2017.arxiv.org/abs/1710.00211

Paper details

vlnc:W_7mywgmqv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents