Neural networks are universal approximators and are studied for their use in solving differential equations. However, a major criticism is the lack of error bounds for obtained solutions. This paper proposes a technique to rigorously evaluate the error bound of Physics-Informed Neural Networks (PINNs) on most linear ordinary differential equations (ODEs), certain nonlinear ODEs, and first-order linear partial differential equations (PDEs). The error bound is based purely on equation structure and residual information and does not depend on assumptions of how well the networks are trained. We propose algorithms that bound the error efficiently. Some proposed algorithms provide tighter bounds than others at the cost of longer run time.
Introduction
Differential equations (DEs) are a useful mathematical tool for describing various phenomena in natural sciences, engineering, and humanity studies. As universal approximators, neural networks are powerful in approximating unknown functions. With back-propagation and modern computing devices, neural networks are convenient to differentiate, making them an ideal choice for solving differential equations.
However, a major criticism of neural network solutions to DEs is the lack of error bound. Traditional numerical methods, such as the finite difference method (FDM) and the finite element method (FEM), compute numerical solutions with known error bounds. Unlike traditional methods, the error bounds of neural network solutions are not well-studied. Therefore, solving DEs with neural networks requires ad hoc customization and empirical hyperparameter finetuning. If the error of any given network can be bounded, we can
Symbols and Notations
DEs in this paper are posed w.r.t. unknown function v,
Dv=f,
where D is a possibly nonlinear differential operator and f is some forcing function. Unlike the exact solution v(⋅), a neural network solution u(⋅) does not strictly satisfy the equation. Instead, it incurs an additional residual term, r, which the network aims to minimize, to the equation,
Du=f+r.
The input to v, u, f, and r is time t for ODEs and spatial coordinates (x,y) for PDEs. We limit our reasoning to 2-dimensional PDEs in this work. In cases with multiple unknown functions, we use vector notations v, u, and r instead of the scalar notations v, u, and r.
Accepted for the 39th Conference on Uncertainty in Artificial Intelligence (UAI 2023). The loss function of the network solution is defined as the L2 norm of residual r over the domain of interest,
Loss(u):=∣I∣1∫I∥r∥2dI=∣I∣1∫I∥Du−f∥2dI,(1)
where a spatial domain Ω is substituted for the temporal domain I in the case of a PDE.
Initial and Boundary Conditions
For a neural network to satisfy initial or boundary conditions, we apply a technique called parametrization. As an intuitive example, the parametrization u(t)=(1−e−t)Net(t)+v(0) guarantees that u(t) satisfies the initial condition u(0)=v(0) regardless of the network Net(⋅). This does not affect the capability of Net(⋅) to learn any solution.
The parametrization is more complicated for higher-order ODEs and most PDEs and has been extensively studied by Lagaris et al. [ [13], Lagaris et al. [ [14], McFall and Mahan [ [16], Lagari et al. [ [12], and Sukumar and Srivastava [ [20]. In this work, we assume all initial and boundary conditions are exactly satisfied.
Error and Error Bound
The error of a network solution u is defined as
η:=u−v.(2)
We are interested in bounding the error with a scalar function B such that
∥η(t)∥≤B(t)or∥η(x,y)∥≤B(x,y)(3)
where ∥η∥=∥u−v∥ is the absolute error. If B takes on the same value B∈R+ over the domain, it can be replaced with a constant B.
Notice that multiple bounds B exist for the same network solution u. For example, ∣η(t)∣≤B(1)(t)≤B(2)(t)≤⋯≤B are bounds in decreasing order of tightness. Tighter bounds incur a higher computational cost, and looser bounds (such as constant B) are faster to compute.
A summary of the applicability, restraints, run-time complexity, and relative tightness of all proposed algorithms is listed in Table 1.
Algorithm
Applicable to
Restraint
Run-Time
Comment
Algorithm 1
Linear ODE
Semi-stable
O(L)
Looser than Alg 2
Algorithm 2
Linear ODE
O(nL)
Tighter than Alg 1
Algorithm 3
Linear ODE System
O(n3L)
Norm and elementwise bounds
Algorithm 4
Nonlinear ODE
Nonlinear term is εvk
O(JnL)
Bounded solution for family of DEs
Algorithm 5
Linear 1st-Order PDE
Coeff. c=0 over domain
O(mesh)
Constant bound; Looser than Alg 6
Algorithm 6
Linear 1st-Order PDE
Solvable characteristics
O(KL)
Tighter than Alg 5 if computable
Table 1. Overview of Proposed Algorithms. The symbols in run-time analysis are defined and explained in detail in Sections 5 and 6, with the exception of K, which is the number of steps used in each numerical integration.
Literature Review
Hornik et al. [ [9] showed that neural networks are universal function approximators. Lagaris et al. [ [13] first studied the application of neural networks in solving DEs. The term physics-informed neural networks, or PINNs, was first introduced by Raissi et al. [ [18] to name neural networks that satisfy DEs while fitting observed data points.
Although we train PINNs only to solve DEs without any observed data in this work, the error-bounding algorithms we propose work for any given neural network, regardless of the training process.
Flamant et al. [ [6] and Desai et al. [ [5] showed that one main advantage of neural networks over traditional numerical methods, such as FDM and FEM, is that neural networks can potentially learn the structure of the solution space and give a bundle of solutions u(x;Θ) for different equation setup and initial/boundary conditions parameterized by Θ. For traditional methods, a new solution must be recomputed for any slight changes in equation setup or initial/boundary conditions.
Some effort has been made to redefine the objective loss function. Yu et al. [ [21] applied the Ritz method to a particular class of variational problems. Mattheakis et al. [ [15] incorporated an additional constraint to force the network to learn solutions with energy conservation. Parwani and Protopapas [ [17] used an adversarial network for sampling in particular areas of the domain where the residual is large.
There are also works that study the failure modes of PINNs and quantify the error of PINN solutions in recent years. Graf et al. [ [7] worked on quantifying the uncertainty of PINNs using the Bayesian framework. Krishnapriyan et al. [ [11] characterized possible failure modes of PINNs by studying the performance of PINNs on simple problems and analyzing their loss landscape. Krishnapriyan et al. [ [11] also concluded that optimization difficulty is the essential cause of failure.
Our work uncovers the mathematical relationship between residual information and the error of PINNs on several classes of ODEs and PDEs. We propose different algorithms for various classes of equations and experimentally validate these algorithms.
Existing Work
Sirignano and Spiliopoulos [ [19] showed that for a class of quasi-linear parabolic PDEs, a neural network with a single hidden layer and sufficiently many hidden units could arbitrarily approximate the exact solutions. Guo and Haghighat [ [8] proposed an energy-based constitutive relation error bound for elasticity problems.
De Ryck and Mishra [ [2] derived an error bound for ReLU networks on parametric hyperbolic conservation laws. De Ryck and Mishra [ [3] showed that there exists some PINN with arbitrarily small residual for Kolmogorov PDEs. De Ryck and Mishra [ [4] derived an error bound for operator learning with PINNs. The works of De Ryck and Mishra mentioned above did not bound the error of every given network. Instead, they mathematically proved the existence of a network with errors below a specified bound, under certain assumptions of network architecture, including width, depth, and activation functions. The question remaining to be answered is how to overcome optimization difficulties and find such a neural network.
Our work differs from the above in that we bound the error of any neural network regardless of finetuning, even networks with randomly initialized weights. Our algorithms only depend on inputs of residual information r, often used as training loss, and equations structure Dv=f. The output is a (possibly constant) function that guarantees to bound the error at any point in domain.
Error Bound for ODE
This section considers both linear and nonlinear ODEs over the temporal domain I=[0,T]. Initial conditions are imposed on dtkdkv(t=0) for k=0,…,(n−1), where n is the highest order of derivative terms in ODE.
Error Bound for Linear ODE
Consider the linear ODE Lv(t)=f(t), where L is a linear differential operator. Its neural network solution u satisfies Lu(t)=f(t)+r(t). Since error η:=u−v, there is
Lη(t)=r(t).(4)
With the assumption in § 1.2 that u satisfies the initial conditions at t=0, there is
η(0)=0,dtdη(0)=0,dt2d2η(0)=0,…(5)
With initial conditions 5 known, a unique inverse transform L−1 to L exists. Applying L−1 to Eq. 4, there is
η(t)=L−1r(t).(6)
Hence, bounding the absolute error ∣η∣ is equivalent to bounding ∣L−1r∣. Notice that only a) the equation structure L and b) the residual information r are relevant to estimating the error bound. All other factors, including parameters of the neural network u, forcing function f, and initial conditions, do not affect the error bound at all.
Single Linear ODE with Constant Coefficients
Consider the case where L=dtndn+∑j=0n−1ajdtjdj consists of only constant coefficients a0,a1,…,an∈R. Its characteristic polynomial (defined below) can be factorized into
λn+an−1λn−1+⋯+a0=j=1∏n(λ−λj),(7)
where λ1,…,λn∈C are the characteristic roots.
It can be shown that, for a semi-stable system (Re(λj)≤0 for all λj), an error bound can be formulated as
∣η(t)∣≤Bloose(t):=Cλ1:nRmaxtZ,(8)
where 0≤Z≤n is the number of λj whose real part is 0, Cλ1:n:=Z!1∏j=1;λj=0nRe(−λj)1 is a constant coefficient, and Rmax:=maxt∈I∣r(t)∣ is the maximum absolute residual. Knowing bound 8 is sufficient to qualitatively estimate the error for applications where only the order of error is concerned. See Alg. 1 for reference.
An issue with Eq. 8 and Alg. 1 is that they assume Re(λj)≤0 for all characteristic roots λj. To address this issue, we propose an alternative error-bounding Alg. 2, which requires more computation but does not require the system to be semi-stable and provides a tighter bound.
Notice that the bounds of η in Eq. 6 can be estimated if the inverse operator L−1 is known. Let Eq. 7 the factorization of characteristic polynomial of L. Define operator Iλ as1
Iλψ(t):=eλt∫0te−λτψ(τ)dτ,∀ψ:I→C.(9)
We show in supplementary material that L−1=Iλn∘Iλn−1∘⋯∘Iλ1 and that ∣Iλψ∣≤IRe(λ)∣ψ∣ for any λ∈C and function ψ. Hence, another error bound can be formulated as
Btight(t):=(IRe(λn)∘⋯∘IRe(λ1))∣r(t)∣.(10)
Algorithm 1 Loose Error Bound Estimation for Linear ODE with Constant Coefficients (Requires Semi-Stability)
Input: Coefficients {aj}j=0n−1 for operator L, residual information r(⋅), domain of interest I=[0,T], and a sequence of time points {tℓ}ℓ=1L where error bound is to be evaluated
Output: Error bound at given time points {B(tℓ)}ℓ=1L
Require:L is semi-stable, and tℓ∈I for all ℓ
Ensure:∣η(tℓ)∣≤B(tℓ) for all ℓ
{λj}j=1n← numerical roots of λn+an−1λn−1+⋯=0
assertλj≤0 for 1≤j≤n
Z,C←0,1
forj←1…ndo
ifRe(λj)=0then
Z←Z+1
else
C←C/Re(−λj)
end if
end for
Rmax←maxτ∈I∣r(τ)∣ ▷ Use linspace with mini-steps
{B(tℓ)}ℓ=1L←{Z!CRmaxtℓZ}ℓ=1L
return{B(tℓ)}ℓ=1L
Note: Jenkins and Traub [ [10] solves polynomial roots.
It is also proven in supplementary material that Btight is tighter than Bloose when Bloose is applicable,
∣η(t)∣≤Btight(t)≤Bloose(t)∀t∈I.(11)
Based on Eq. 10, we propose Alg. 2, which computes Btight by repeatedly evaluating integrals in 9 using the cumulative trapezoidal rule.
Before moving on to the next section, we discuss the effect of numerical error in Alg. 2 and all pursuant algorithms which involve numerical integration. Empirically, the error introduced by numerical integration in negligible for most cases. To ensure the accuracy of the error bound, we recommend using a slightly modified trapezoidal rule that, instead of directly using the node value, takes the maximum of the node and its two adjacent nodes. This modification leads to a slightly looser bound. Yet, this influence is almost negligible in practice.
Single Linear ODE of the General Form
In general, the coefficients for L can be functions of t. Namely, L=dtndn+∑j=0n−1aj(t)dtjdj. Similar to Eq. 7, L have characteristic roots {λj(t)}j=1n as functions of t,
λn+an−1(t)λn−1+⋯+a0(t)=j=1∏n(λ−λj(t)).
We can replace constant λj with functions λj(t) in Eq. 9 and compute bound Btight as in Eq. 10. However, the factorization in Eq. 5.1.2 is hard to implement in practice except
Algorithm 2 Tighter Error Bound Estimation for Linear ODE with Constant Coefficients (Stable and Unstable)
Note:CumTrap({yk}k=1K,{xk}k=1K) computes cumulative integral ∫0xy(x)dx at discrete points {xk}k=1K using trapezoidal rule.
Note:Interp({yk}k=1K,{xk}k=1K,{xℓ}ℓ=1L) computes interpolant to {yk}k=1K with given discrete data points {(xk,yk)}k=1K evaluated at {xℓ}ℓ=1L.
for the first-order case where Lv=dtdv+a0(t)v. Cases of second order and higher are out of the scope of this paper.
System of Linear ODEs with Constant Coefficients
Consider a system of linear ODEs with constant coefficients
dtdv+Av=f(t)(12)
where v and f are Rn vectors and A is a n×n matrix. Denote the Jordan canonical form of A as,
An elementwise bound (vector) B(t) can be formulated as
η∣⋅∣(t)⪯B(t):=P∣⋅∣I(P−1)∣⋅∣∣r∣(t),(15)
where superscript ∣⋅∣ denotes taking elementwise absolute value and symbol ⪯ denotes elementwise inequality. In the meantime, a norm bound (scalar) B(t) also exists
∥η(t)∥≤B(t):=cond(P)∥I[∥r∥11](t)∥(16)
where cond(P) is the conditional number of P w.r.t. induced matrix norm and 1 is an n×1 column vector of 1s. Proof of (49) and (50) can be found in supplementary material. See Alg. 3 for implementation.
Algorithm 3 ODE System Bound (norm and elementwise)
Input: Coefficient matrix A∈Rn×n, residual vector r(t), and a sequence of points {tℓ}ℓ=1L where error is to be bounded
Output: Norm bound (scalar) {B(tℓ)}ℓ=1L and componentwise bound (vector) {B(tℓ)}ℓ=1L at given time points
Ensure:∥η(tℓ)∥≤B(tℓ) and η(tℓ)⪯B(tℓ) for all ℓ
Nonlinear ODEs are hard to solve in general. In this work, we only deal with nonlinear ODEs with a single nonlinear term of the form εvk(t), where ε∈R is a small number. Ideally, ∣ε∣≪1. The exact requirement for ε is given in § 5.2. The value of ε can vary within a certain range or be fixed. With the perturbation technique, we obtain a family of solutions v(t;ε) parameterized by ε at the cost of solving a (countable) collection of equations. As explained below in § 5.1, we train finitely many networks, each approximately solving an equation in the collection.
Perturbation Theory
Consider the nonlinear ODE with nonlinear term εvk(t),
Lv(t)+εvk(t)=f(t),(17)
where L is a linear differential operator discussed in 5.1 and initial conditions are specified for the system at time t=0. Notice that each ε∈R corresponds to a solution v(t;ε). We expand the solution v(t;ε) in terms of ε
v(t;ε)=j=0∑∞εjvj(t)=v0(t)+εv1(t)+…(18)
Only v0(t) is subject to the original initial conditions at t=0, while other components, v1, v2, …, have initial conditions of 0 at $t = 0. Substituting Eq. 18 into Eq. 17,
With v0 known, (57) is linear in v1 and can be solved for v1.
Similarly, with v0 and v1 known, (58) is linear in v2 and can be solved for v2.
The process can be repeated for (59) and beyond. Only a linear ODE is solved each time.
To solve the system with PINNs, we approximate exact solutions {vj(t)}j=1∞ with neural network solutions {uj(t)}j=0J trained sequentially on (56), (57), and beyond. In practice, we only consider components up to order J to avoid the infinity in expansion 18. Ideally, J should be large enough so that higher order residuals in expansion 18 can be neglected.
After obtaining {uj(t)}j=0J, we can reconstruct the solution u(t;ε)=∑j=0Jεjuj(t) to the original nonlinear equation 17 for varying ε. See Alg. 4 for details.
Expansion of Bounds
The absolute error ∣η(t;ε)∣=∣u(t;ε)−v(t;ε)∣ is given by
where ηj(t):=uj(t)−vj(t) is the component error between uj(t) and vj(t). Let Bj denote the bound component such that ∣ηj(t)∣≤Bj(t). Assuming J is large and ε is small such that higher order terms ∑j=J+1∞εjvj(t) are negligible, there is
∣η(t;ε)∣≤B(t;ε):=j=0∑JBj(t)∣ε∣j(27)
where each bound component Bj can be evaluated using the technique in Section 5.1. See Alg. 4 for details.
Algorithm 4 Iterative Method for Solution and Error Bound of Nonlinear ODE 17
Input: Linear operator L, nonlinear degree k, domain I=[0,T], highest order J for expansion, and a sequence {(tℓ,εℓ)}ℓ=1L where solution u(t;ε) and error bound B(t;ε) are to be evaluated
Output: Solution {u(tℓ;εℓ)}ℓ=1L and error bound {B(tℓ;εℓ)}ℓ=1L
Require:tℓ∈I, and ∣εℓ∣ to be small (ideally ∣εℓ∣≪1)
Note 1:B0 and B1:J can be evaluated using Alg. 1 or 2.
Note 2:BNL can be estimated even though exact solutions v0:j−1 are unknown. This is because vi∈[ui−Bi,ui+Bi] for all i, and u0:j−1,B0:j−1 are known from previous iterations.
Error Bound for PDE
This section considers first-order linear PDEs defined on a 2-dimensional spatial domain Ω.2
a(x,y)∂x∂v+b(x,y)∂y∂v+c(x,y)v=f(x,y)(28)
with Dirichlet boundary constraints defined on Γ⊂∂Ω,
v(x,y)∈Γ=g(x,y).(29)
We partition the domain into infinitely many characteristic curves C, each passing through a point (x0,y0)∈Γ. The resulting curve is a parameterized integral curve
C:{x′(s)=a(x,y)y′(s)=b(x,y)where (⋅)′=dsd and {x(0)=x0y(0)=y0.
For any (x(s),y(s)) on C, functions (v,a,b,c,f) can be viewed as univariate functions of s. By chain rule, there is
Hence, (62) is reformulated as an ODE along curve C,
v′(s)+c(s)v(s)=f(s)s.t. v(0)=g(x0,y0),(30)
where v(s), c(s), and f(s) are shorthand notations for v(x(s),y(s)), c(x(s),y(s)), and f(x(s),y(s)), respectively.
In particular, if c(x,y)=0 for all (x,y)∈Ω, both sides of (62) can be divided by c(x,y), resulting in a residual of r(x,y)/c(x,y) where r(x,y) is the residual of the original problem. By (42), a constant error bound on C is ∣η(s)∣≤maxs∣r(s)/c(s)∣. Hence, a (loose) constant error bound B (see Alg. 5) over the entire domain Ω is
∣η(x,y)∣≤B:=(x,y)∈Ωmaxc(x,y)r(x,y).(31)
Algorithm 5 Constant Err Bound for Linear 1st-Order PDE
Input: Coefficient c(x,y) in (62), residual information r(x,y) and domain of interest Ω
Output: A constant error bound B∈R+
Require:c(x,y)=0 for all (x,y)∈Ω
Ensure:∣η(x,y)∣≤B for all (x,y)∈Ω
{(xk,yk)}k← sufficiently dense mesh grid over Ω
B←maxkc(xk,yk)r(xk,yk)
return B
Independent of the assumption c(x,y)=0, in scenarios where the curve C passing through any (x,y) can be computed, the error can be computed using Alg. 6.
Relevant Experiments
In this section, we perform experiments on equations with manufactured solutions using the NeuroDiffEq library [1], which provides convenient tools for training PINNs.
Algorithm 6 General Err Bound for Linear 1st-Order PDE
Input: Coefficients a(x,y), b(x,y), c(x,y) in Eq. 28, residual information r(x,y), domain of interest Ω, Dirichlet boundary Γ⊂∂Ω, and a sequence of points {(xℓ,yℓ)}ℓ=1L where error is to be bounded
Output: Error bound {B(xℓ,yℓ)}ℓ=1L at given points
Require: Integral curve of vector field [a(x,y)b(x,y)]T passing through any point (xℓ,yℓ)∈Ω is solvable
Ensure:∣η(xℓ,yℓ)∣≤B(xℓ,yℓ) for all ℓ
Cg← general solution to
{x′(s)=a(x,y)y′(s)=b(x,y)
forℓ←1…Ldo
x(s),y(s)← instance of Cg passing through (xℓ,yℓ)
s∗← solution to x(s)=xℓ, y(s)=yℓ
First, we train networks to solve equations and collect their residual information r. Then, we apply Alg. 1–6 (where applicable) to derive error bounds using only residual information r and equation structure, characterized by its differential operator D. Lastly, we show that the absolute error strictly falls within the bounds, regardless of how well the networks are trained.
Throughout this section, we always use networks with two hidden layers, each consisting of 32 hidden units. Depending on whether the problem is an ODE or PDE, a network can have a single input t or two inputs (x,y), but always have a single output. The activation function is tanh. Unless otherwise noted, the training domain is I=[0,1] for ODEs and Ω=[0,1]2 for PDEs. We use a PyTorch Adam optimizer with default hyperparameters to train networks for 1000 epochs.
Notice that we list these configurations only for the reproducibility of visualizations. Our error-bounding algorithm works under any other configurations.
Single Linear ODE with Constant Coefficients
Here, we study three equations v′′+3v′+2v=f(t), v′′+v=g(t), and v′′−v′=h(t), whose characteristic roots are {−1,−2}, {±i}, and {0,1} respectively. By § 4.2, the first two equations can be bounded with either Alg. 1 or Alg. 2, while the last must be bounded with Alg. 2.
They all satisfy initial conditions v(0)=v′(0)=1. We pick f(t)=2t2+8t+7, g(t)=t2+t+3, and h(t)=1−2t, so that the manufactured solution is v(t)=t2+t+1 for all three equations. Figure 1 shows that both Bloose (Alg. 1) and Btight (Alg. 2) strictly bounds the absolute error.
Figure 1. Loose bound (Alg. 1) and tight bound (Alg. 2) for 3 second-order linear ODE with constant coefficients. Notice that the loose bound cannot be applied to the third equation since it has characteristic roots with positive real part.
Linear ODE System with Constant Coefficients
In this subsection, we train 6 networks to solve a 6-dimensional linear system of ODEs with constant coefficients, namely, dtdv+Av=f. We pick A=PJP−1 where
J=J1J2J3
with J1=41414, J2=(313), J3=2, and P is a random orthogonal matrix.
We pick the initial conditions to be v(0)=P(001011)T and the forcing function to be f(t)=P(cost+4sint+ln(1+t),1+t1+4ln(1+t)+(t+1),4t+5,2t+3t2+et,4et,2cost−sint)T, so that the manufactured exact solution is v(t)=P(sint,ln(t+1),t+1,t2,et,cost)T.
After obtaining the residual information r(t)=dtdu(t)+Au(t)−f(t), we apply Alg. 3 to obtain componentwise bound and norm bound of η=u−v. It is shown in Figure 2 that the bounds hold over the domain.
Figure 2. Componentwise bound (upper) and norm bound (lower) for linear ODE system with constant coefficients
Nonlinear ODE – Duffing Equation
In this subsection, we consider a Duffing oscillator, which is characterized by the following 2nd order nonlinear ODE:
dt2d2v+3dtdv+2v+εv3=cost,(32)
under initial conditions v(0)=1 and v′(0)=1, where ε controls the nonlinearity of the equation. Using Alg. 4, we solve the equation on I=[0,2] for linspaced ε∈(−0.9,0.9) using neural networks and bound the errors. The input J to Alg. 4 is chosen to be 6. Namely, we expand the solution and bound components from degree 0 to 6.
The analytical solution to (66) is complicated. Hence, we use the RKF4(5) method to compute numerical solutions that are close enough to exact solutions for visualization purposes. See Figure 3 for network solutions against RKF4(5) solutions and Figure 4 for error bounds against absolute error.
Figure 3. RKF45 and Network Solutions (max-degree 0∼6) to Duffing equation 32 for ε∈(−0.9,0.9)
Figure 4. True Error vs. error bound (max-degree 0∼6) of neural network solution to Duffing Equation 32 for ε∈(−0.9,0.9)
Linear PDE System with Nonconstant Coefficients
PDE Error Bound Evaluation Using Alg. 6
We try to solve the following first-order linear PDE,
(−x−y)∂x∂v+(x−y)∂y∂v+v=3x−2y(33)
in spatial domain Ω=[−1,1]2. The boundary constraints are v(x,±1)=2x±3 and v(±1,y)=3y±2. The manufactured solution is given by v(x,y)=2x+3y. The characteristic curves are integral curves C:
x′(s)y′(s)=−x−y=x−y
or
x(s)y(s)=R0e−scos(s+θ0)=R0e−ssin(s+θ0)
where R0=x02+y02 and θ0=atan2(y0,x0) are constants determined by the starting point (x0,y0)∈Γ=∂Ω. See Figure 5 for visualization.
Figure 5. Characteristics curves of Eq. 33 (left) and Eq. 34 (right). The red curves, with staring points A to P, are selected for visualization of absolute error and error bound in Fig. 6.
Since the analytical expression of the characteristic curves is known, Alg. 6 can be applied to evaluate the bound on each curve. We choose 16 characteristic curves with starting points A,B,…,P, equidistantly placed on the boundary (Fig. 5a). We plot the absolute error and the computed error bound along these characteristic curves in Fig. 6. It can be seen that absolute error lies strictly within the bounds.
PDE Error Bound Evaluation Using Alg. 5
Consider the following PDE
(x2+y2+1)∂x∂v+(x2−y2+2)∂y∂v+(3−2x)v=f(34)
over domain Ω=[−1,1]2 where f(x,y)=6−4x. The boundary constraints are v(−1,y)=2 and $v(x,1)=2, and the manufactured solution is v(x,y)=2.
Figure 6. Absolute error and error bound on selected characteristic curves. These characteristic curve start at points A through P as shown in Figure 5a. The blue solid curves are absolute error along the characteristic curves and red dotted curves are corresponding bounds.
The characteristic curves C:
{x′(s)=x2+y2+1y′(s)=x2−y2+2
are given by a nonlinear ODE, which is hard to solve analytically. (See Figure 5b for visualization) Therefore, Alg. 6 cannot be applied to evaluate the error bound.
However, the coefficient (3−2x) is nonzero over domain Ω. Hence, we can use Alg. 5 to compute a constant error bound ∣η(x,y)∣≤B(x,y)≡B for all (x,y)∈Ω. We visualize the bound and the maximum absolute error max(x,y)∈Ω∣η∣ after each training epoch in Figure 7. As expected, the bound is loose, which is about an order of magnitude larger than the max absolute error. Yet, it consistently holds true for every epoch, even during the early stages of training, when the network performs poorly.
Figure 7. Constant bound B, computed using Alg. 5, and max absolute error over domain at different epochs of training.
Conclusion and Future Work
This paper proposes various error-bounding algorithms for any PINN solution to certain classes of ODEs and PDEs.
These algorithms only require the residual information r(⋅) and the equation structure Dv=f as input. There are many real-world applications for which the exact solution v(⋅) is unknown or hard to compute. However, the residual information r(⋅) is usually, if not always, readily available. With our proposed algorithms, PINNs can be trained until the error is gauranteed to fall below a specified tolerance threshold. The mathematical relationship between residual and error bound also sheds light on optimizing PINN solutions for future studies.
The error-bounding algorithms proposed in this paper only apply to certain classes of ODEs and PDEs. However, the insights of this paper can be beneficial to future work that extends to more general classes of ODEs and PDEs, especially nonlinear ones. We also plan to apply these algorithms to stochastic differential equations, where the error bound is a probabilistic tail bound.
Residual-Based Error Bound for Physics-Informed Neural Networks (Supplementary Material)
Shuheng Liu; Xiyue Huang; Pavlos Protopapas
Proof of Propositions in Section 5.1.1
In this part, we first discuss the properties of the operator Lλ, which is defined in the main paper. We then use these properties to prove relevant statements regarding Alg. 1 and Alg. 2 in Section 5.1.1 of the main paper.
Properties of Inverse Operator Iλ=Lλ−1
Let Lλ (λ∈C) be the differential operator Lλϕ:=dtdϕ−λϕ. The inverse of Lλϕ=ψ is given by ϕ=Iλψ if ϕ(0)=0, where
Iλψ(t):=eλt∫0te−λτψ(τ)dτ.(35)
In addition to Iλ=Lλ−1, there are a few properties of operator Iλ that we are interested in
Linearity:Iλ(c1ψ1+c2ψ2)=c1Iψ1+c2Iψ2 for all functions ψ1,ψ2 and constants c1,c2∈C
Monotonicity: For λ∈R, there is (∀t∈I,ψ1(t)≤ψ2(t))⟹(∀t∈I,Lλψ1(t)≤Lλψ2(t)),
Commutativity:Iλ1∘Iλ2=Iλ2∘Iλ1 for all λ1,λ2∈C. This can be shown because Lλ1∘Lλ2=Lλ2∘Lλ1. Therefore, the inverse operators Iλ2∘Iλ1Iλ1∘Iλ2 must also be equal.
Absolute Inequality:∣Iλψ(t)∣≤IRe(λ)∣ψ(t)∣, which we prove in the next subsection.
Proof of Operator Inequality ∣Iλψ∣≤IRe(λ)∣ψ∣
Proposition For any λ∈C and scalar function ψ:R+→C, there is
∣Iλψ(t)∣≤IRe(λ)∣ψ(t)∣.(36)
Proof Let ϕ=Iλψ. Since L=I−1, the problem is equivalent to proving ∣ϕ∣≤IRe(λ)∣ψ∣, where
dtdϕ−λϕ=ψ.(37)
To see this, we multiply both sides with an integrating factor e−λt and integrate from 0 to t,
∫0te−λτ(dτdϕ(τ)−λϕ(τ))dτ=∫0te−λτψ(τ)dτ(38)
e−λtϕ(t)−ϕ(0)=∫0te−λτψ(τ)dτ(39)
Since ϕ=Iλψ, there is ϕ(0)=0. Hence we have
ϕ(t)=eλt∫0te−λτψ(τ)dτ(40)
∣ϕ(t)∣=eλt∫0te−λτψ(τ)dτ(41)
≤(42)
For λ∈C, there is ∣e±λt∣=e±Re(λ)t, where Re(λ) is the real part of λ. Hence,
∣ϕ(t)∣=eRe(λ)t∫0te−λτψ(τ)dτ(43)
≤eRe(λ)t∫0t∣e−λτψ(τ)∣dτ(44)
=eRe(λ)t∫0te−Re(λ)τ∣ψ(τ)∣dτ=:IRe(λ)∣ψ(t)∣(45)
Proof of tight and loose bounds
This section proves inequality 11 in the main paper, namely,
Proof For any linear differential operator L=dtndn+an−1dtn−1dn−1+⋯+a0 whose coefficients {aj}j=0n−1 satisfy
λn+an−1λn−1+⋯+a0=j=1∏n(λ−λj),
it can be verified that L=Lλ1∘⋯∘Lλn, where Lλjϕ:=dtdϕ−λjϕ as defined in appendix A.1. Then by the proposition in appendix A.1, the inverse operator is given by
Let Rmax:=maxt∈I∣r(t)∣ be the max absolute residual. Let Z=∣{λj:Re(λj)=0,1≤j≤n}∣. Assume without loss of generality that Re(λ1),…,Re(λn−Z)<0 and that Re(λn−Z+1)=⋯=Re(λn)=0. By the monotonicity of operator IRe(λ), there is IRe(λ)φ1(t)≤IRe(λ)φ2(t) if φ1(t)≤φ2(t) for all t∈I. Hence,
In this part, we prove relevant statements regarding Alg. 3 in Section 5.1.1 of the main paper.
Consider the problem 12 in main paper. The error η of the network solution u satisfies the equation
dtdη+Aη=r(t)s.t.η(t=0)=0(63)
where r(t)=dtdu(t)+Au(t)−f(t) is the residual vector.
With the Jordan canonical form 13, we multiply both sides of Eq. 29 by P−1,
P−1dtdη+P−1Aη=P−1r(t)(64)
P−1dtdη+JP−1η=P−1r(t)(65)
dtdδ+Jδ=q(t)(66)
where δ(t):=P−1η(t) and q(t)=P−1r(t). Recall that J is a Jordan canonical form consisting of K Jordan blocks. Each Jordan block Jk (1≤k≤K) is an nk×nk square matrix, with eigenvalue λk on its diagonal and 1 on its super-diagonal, where ∑k=1Knk=n. Expanding the vector notations, there is
The last equation (Eq. (72)) of the Jordan chain can be used to bound δNk by applying the inequality 2,
∣δNk∣=∣I−λkqNk∣≤I−Re(λk)∣qNk∣(73)
Applying the inequality 2 again to Eq. (71), there is
∣δNk−1∣=∣I−λk(qNk−1+δNk)∣(74)
≤I−Re(λk)∣qNk−1−δNk∣(75)
≤I−Re(λk)∣qNk−1∣+I−Re(λk)∣δNk∣(76)
≤I−Re(λk)∣qNk−1∣+I−Re(λk)2∣qNk∣.(77)
The first inequality is a direct application of Eq. 2. The second inequality is based on linearity of the operator I and the triangle inequality. The third inequality is obtained by substituting Eq. (73). Here the superscript in I2 denotes compositional square I2=I∘I.
By induction, for the k-th Jordan block (Nk−1<l≤Nk), there is
∣δl∣≤j=0∑Nk−lI−Re(λk)j+1∣ql+j∣(78)
We use this inequality to bound the norm of the error vector, ∥η∥, as well as absolute value of each component, ∣(η)l∣.
Componentwise Bound
Using matrix notations, Eq. (78) can be rewritten as
δ∣⋅∣⪯Iq∣⋅∣(79)
where ⪯ denotes componentwise inequality, the superscript ∣⋅∣ denotes componentwise absolute value, and I is defined as operator matrix $\mathcal{I}:=
is an nk×nk upper-triangular block. Notice that (AB)∣⋅∣⪯A∣⋅∣B∣⋅∣ for any compatible matrices A and B. Recall δ(t)=P−1η(t) and q(t)=P−1r(t), there is
η∣⋅∣⪯P∣⋅∣δ∣⋅∣⪯P∣⋅∣I[q∣⋅∣]⪯P∣⋅∣I[(P−1)∣⋅∣r∣⋅∣](80)
Norm Bound
By Eq. (79), we have ∥δ∥≤∥I[∥q∥1]∥, where 1 is an n×1 (constant) column vector whose components are all equal to 1.
With η=Pδ and q=P−1r, there is ∥η∥≤∥P∥∥δ∥ and ∥q∥≤∥P−1∥∥r∥, where ∥⋅∥ denotes the norm of a vector or the induced norm of a matrix. Consequently,
∥η(t)∥≤∥P∥∥δ(t)∥(81)
≤∥P∥∥I[∥q(t)∥1]∥(82)
≤∥P∥I[∥P−1∥∥r∥1](83)
≤∥P∥∥P−1∥∥I[∥r∥1]∥(84)
=cond(P)∥I[∥r(t)∥1]∥(85)
References
[1]Feiyu Chen, David Sondak, Pavlos Protopapas, Marios Mattheakis, Shuheng Liu, Devansh Agarwal, and Marco Di Giovanni. Neurodiffeq: A python package for solving differential equations with neural networks. Journal of Open Source Software, 5(46):1931, 2020.DOI
[2]Tim De Ryck and Siddhartha Mishra. Error analysis for deep neural network approximations of parametric hyperbolic conservation laws. arXiv preprint arXiv:2207.07362, 2022a.arxiv.org/abs/2207.07362
[3]Tim De Ryck and Siddhartha Mishra. Error analysis for physics-informed neural networks (PINNs) approximating Kolmogorov PDEs. Advances in Computational Mathematics, 48(6):1–40, 2022b.arxiv.org/abs/2106.14473
[4]Tim De Ryck and Siddhartha Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. arXiv preprint arXiv:2205.11393, 2022c.arxiv.org/abs/2205.11393
[5]Shaan Desai, Marios Mattheakis, Hayden Joy, Pavlos Protopapas, and Stephen Roberts. One-shot transfer learning of physics-informed neural networks. 2021.arxiv.org/abs/2110.11286
[6]Cedric Flamant, Pavlos Protopapas, and David Sondak. Solving differential equations using neural network solution bundles. arXiv preprint arXiv:2006.14372, 2020.arxiv.org/abs/2006.14372
[7]Olga Graf, Pablo Flores, Pavlos Protopapas, and Karim Pichara. Uncertainty quantification in neural differential equations. arXiv preprint arXiv:2111.04207, 2021.arxiv.org/abs/2111.04207
[8]Mengwu Guo and Ehsan Haghighat. Energy-based error bound of physics-informed neural network solutions in elasticity. Journal of Engineering Mechanics, 148(8):04022038, 2022.DOI
[9]Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.DOI
[10]Michael A Jenkins and Joseph F Traub. A three-stage variable-shift iteration for polynomial zeros and its relation to generalized rayleigh iteration. Numerische Mathematik, 14(3):252–263, 1970.
[11]Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34, 2021.arxiv.org/abs/2109.01050
[12]Pola Lydia Lagari, Lefteri H Tsoukalas, Salar Safarkhani, and Isaac E Lagaris. Systematic construction of neural forms for solving partial differential equations inside rectangular domains, subject to initial, boundary and interface conditions. International Journal on Artificial Intelligence Tools, 29(05):2050009, 2020.DOI
[13]Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987–1000, 1998.DOI
[14]Isaac E Lagaris, Aristidis C Likas, and Dimitris G Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11(5):1041–1049, 2000.DOI
[15]Marios Mattheakis, David Sondak, Akshunna S Dogra, and Pavlos Protopapas. Hamiltonian neural networks for solving differential equations. arXiv preprint arXiv:2001.11107, 2020.DOI
[16]Kevin Stanley McFall and James Robert Mahan. Artificial neural network method for solution of boundary value problems with exact satisfaction of arbitrary boundary conditions. IEEE Transactions on Neural Networks, 20 (8):1221–1233, 2009.DOI
[17]Kshitij Parwani and Pavlos Protopapas. Adversarial sampling for solving differential equations with neural networks. arXiv preprint arXiv:2111.12024, 2021.arxiv.org/abs/2111.12024
[18]Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.DOI
[19]Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339–1364, 2018.DOI
[20]N Sukumar and Ankit Srivastava. Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks. arXiv preprint arXiv:2104.08426, 2021.DOI
[21]Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. arXiv preprint arXiv:1710.00211, 2017.arxiv.org/abs/1710.00211