betaFeedback

WorksComputer Science

Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems

Abstract

Recent years have witnessed a growth in mathematics for deep learning–which seeks a deeper understanding of the concepts of deep learning with mathematics and explores how to make it more robust–and deep learning for mathematics, where deep learning algorithms are used to solve problems in mathematics. The latter has popularised the field of scientific machine learning where deep learning is applied to problems in scientific computing. Specifically, more and more neural network architectures have been developed to solve specific classes of partial differential equations (PDEs). Such methods exploit properties that are inherent to PDEs and thus solve the PDEs better than standard feed-forward neural networks, recurrent neural networks, or convolutional neural networks. This has had a great impact in the area of mathematical modeling where parametric PDEs are widely used to model most natural and physical processes arising in science and engineering. In this work, we review such methods as well as their extensions for parametric studies and for solving the related inverse problems. We equally proceed to show their relevance in some industrial applications.

Motivation

Arguably, partial differential equations (PDEs) provide the most widely used models for a large variety of problems in natural sciences, engineering and industry. The ever-increasing complexity of these models such as digital twins for high-tech industrial applications requires the most efficient solvers.

These models follow the full life cycle of products from classical simulation and optimisation during the development phase to process monitoring and control during production. Such models typically rely on accurate calibration of critical parameters, which might be scalar parameters or even distributed spatial-time varying parameter functions. The calibration process itself requires multiple runs of the model and increases the need for efficient solvers even more.

Not surprisingly, data-driven concepts and in particular neural network approaches have been studied extensively over the last few years as they have the potential to overcome three major obstacles in PDE simulations: First of all, no mathematical-physical model is ever complete, however, even the finest detail or tricky non-linearity is contained in a sufficient data set; secondly, the parameters to be determined most often are not arbitrary but follow an unknown, application specific distribution, which can be recovered and exploited from training data; thirdly, as already mentioned, the complexity of PDE-based modelling has reached a level such that conventional and most widely used methods such as finite elements or finite difference do require computational times beyond reasonable limits, while neural network concepts after training are most efficient.

This is matched by our own experience over the last few years, where an increasing number of our industrial and engineering collaboration partners started to experiment with deep learning (DL) concepts based on neural networks for PDE problems. However, they as much as we were overwhelmed by the somewhat confusing variety of different DL concepts for these tasks. These span from general approaches for large classes of problems to very specifically tailored approaches for individual equations. The present paper is based on our endeavour to provide an unbiased overview of the most common general approaches and to provide limited, but scientifically based guidance on how to select appropriate methods. We are well aware of the many extension of the mentioned methods and the exploding body of literature dealing with such problems, and we readily apologise to those scientists, whose research we neglected to include. To be more precise, we are considering second-order partial differential equations defined in a bounded domain Ω⊂Rd\Omega\subset\mathbb{R}^{d}, which depend on a parameter function λ\lambda:

N(u,∇u,Δu,ut;λ)=0(1)\mathcal{N}(u,\nabla u,\Delta u,u_t;\lambda)=0 \tag*{(1)}
λ:Ω→R(2)\lambda:\Omega\to\mathbb{R} \tag*{(2)}

In this general notation, N\mathcal{N} encodes the differential equation as well as boundary conditions. We always assume, that the parameter-to-state operator FF, which maps a given parameter λ\lambda to the solution of the PDE, is well-posed. I.e. we assume a function space setting such that the solution uu of the PDE is unique and depends continuously on the parameter λ\lambda. We will consider three related classes of problems:

Level 1: Forward problem: solving a single PDE: given λ\lambda, compute u=F(λ)u=F(\lambda)

Level 2: Parametric studies: given many parameters, λ∈{λ1,…,λN}\lambda\in\{\lambda_1,\ldots,\lambda_N\}, compute corresponding uu’s

Level 3: Parameter identification (inverse problem): given a measured uδu^\delta or its values PuδPu^\delta under a measurement operator PP, determine a corresponding λ\lambda, e.g. solve F(λ)∼uδF(\lambda)\sim u^\delta.

Our main target is the third class of problems, i.e. inverse problems stated as parameter identification problems for PDEs. These problems are typically non-linear and ill-posed, even if the differential equation is linear and the forward operator is well-posed. Nevertheless, we start with the first problem and review the most common DL approaches for solving the forward problem. We then discuss their potential for parametric studies and parameter identification problems. As an underlying motivation for using DL concepts in this setting, we assume that an evaluation of the forward operator FF by classical methods, e.g. finite difference schemes or finite elements, is computationally expensive and not suitable for large-scale parametric studies.

We have two remarks for clarifying the scope of the present survey. First of all, parameter identification problems are inverse problems and can be attacked by well-established general regularisation schemes for operator equations. For a recent overview of such data-driven concepts for inverse problems and their regularisation properties, see e.g. [5]. However, these concepts, e.g. unrolled iteration schemes, typically involve an evaluation of the forward operator or its adjoint. This will not lead to efficient schemes in our framework, where the evaluation of FF is assumed to be too costly for large-scale parametric studies. We will remark shortly on that in our section on the state of the art. Hence, in the present paper, we only consider DL concepts, where the forward operator itself, i.e. the parameter-to-state operator, or its inverse are replaced by a neural network. Secondly, there exists a growing number of highly optimised DL concepts for very specific PDE problems e.g. for coupled physics systems, molecular dynamics, or complex fluid dynamic problems. These approaches do have a limited potential for transfer to other problems, and we will not address these concepts. We will rather focus on general classes of DL concepts that apply to a larger variety of PDE-based problems. However, we will highlight some of those successful but specialised approaches in the section on the state of the art. All the codes and dataset used for the numerical experiment will equally be made available online on GitHub (see also Section A.2)

Classical methods for solving PDEs numerically

Let us first comment on the main competitors for data-driven concepts, namely the classical and well-established concepts of finite difference (FDM), finite element (FEM), boundary element Method (BEM), finite volume (FVM) or particle methods.

The so-called finite difference method (FDM) is based on the replacement of the occurring derivatives by difference quotients. In this process, the function values of the solution are approximated at individual discrete grid points. The result is a so-called grid function. We refer, e.g., to [145] for a more detailed description. The great advantage of the FDM is its straightforward realisation. However, its extension to more complex problems is limited and rather unrealistic smoothness assumptions are necessary for its analysis.

The finite element method (FEM) overcomes the mentioned disadvantages of FDM methods to a large extent. In contrast to the FDM, it is based on the weak formulation of the PDE under consideration and searches for the discrete solution over a finite-dimensional subspace of the underlying function space, see [15] for more details. Especially in connection with adaptive and domain decomposition methods, it shows its full potential. An interesting variant is discontinuous Galerkin (dG) methods, in which the smoothness assumptions for the discrete solution, which stems from the weak formulation of the PDE, are only enforced by penalty terms.

The boundary element method (BEM) follows a different idea. In this concept, the PDE is transformed into a boundary integral equation, which, however, is only possible in special cases. This reduces the dimension of the problem. The boundary integral equation is then discretised by means of FEM, for example. However, the resulting system of equations is dense and its numerical solution is often tricky and numerically very elaborate. Nevertheless, the BEM offers great advantages for exterior space problems in which an unbounded area is considered. We refer for instance to [59] for more information.

The Finite Volume Method (FVM) brings together ideas from the FEM and the FDM and combines them with arbitrary control volumes, see e.g. [100]. The FVM is particularly well suited for the discretisation of hyperbolic equations and convection-dominated problems. However, its theoretical analysis is not yet that advanced.

Another approach are particle methods or discrete element methods (DEM), see e.g. [19], which are based on a completely different approach. Here, individual particles and their interaction with each other are considered. By using different approaches to describe the interaction, different materials and processes can be simulated. They thus achieve very good accuracy in many applications, e.g. in the simulation of sand, which is difficult to simulate with the other methods.

The mentioned concepts are well-established for solving PDE-based forward problems and all of them have been adapted to inverse problems, in particular in the context tasks of optimal control or numerical parameter identification. They lead to accurate results in this context. However, two main difficulties have to be mentioned: Firstly and as said already in the previous section, solving these inverse problems requires solving the forward problem multiple times, which leads to unacceptable overall computing times at least for large problems. Furthermore, the optimisation algorithms for solving the above-mentioned parameter identification tasks require the first and, if possible, also the second derivative with respect to the unknown parameters. At first, it is often unclear whether the mentioned derivatives exist at all. In addition, their calculation is algorithmically very complex and time-consuming, especially for time-dependent problems. We refer, for example, to [151] for a detailed discussion.

In summary, the classical methods are well-researched and reliable and should be used whenever possible within the given restrictions of data and computer power. Hence, we once again stress, that the present paper refers to potential applications where solving the forward problem multiple times is simply too costly.

Deep learning for PDE solvers and scope of this survey

The main purpose of this section is to list the DL methods considered in this present survey. In the subsequent sections, we will then analyse their potential for parametric studies and inverse problems.

As already mentioned, mesh-based methods for solving PDEs such as finite element methods, finite difference methods, and finite volume methods, are the dominant techniques for obtaining numerical solutions of PDEs. When implementing these methods, the computational domain of interest should be first discretised using a set of mesh points and the solution is then approximated at the chosen grid points. The solutions are usually obtained from a finite linear space by solving a linear or non-linear system of equations. The classical methods are very stable and efficient for low-dimensional problems and regular geometries. They have been well understood in terms of existence, stability and error estimate, and we can usually achieve the desired accuracy by increasing the resolution of the discretisation. In addition to the more general remarks above, the classical methods have several additional drawbacks and limitations. Firstly, the curse of dimensionality has limited the use of mesh-based methods, since the number of discretisation points will increase exponentially with the dimension, and there is no straightforward way to discretise irregular domains in high-dimensional spaces. Secondly, traditional methods are designed to model one specific instance of the PDE system, not the parameter-to-state operator. Thus, for any given new instance of the PDE system, the problem has to be solved again. Thirdly, the numerical solution is only computed at the mesh points and evaluation of the solution at any other point requires interpolation or some other reconstruction method. Fourthly, classical methods require the knowledge of the underlying analytic system of differential equations, but sometimes the exact physics is unknown. Lastly, solving inverse problems by classical problems is often prohibitively expensive and requires complex formulations, repeated computation of forward problems, new algorithms and elaborate computer coding.

Deep learning methods have shown great power in addressing the above difficulties. Since there are many different deep learning methods for PDEs, and each method has its particular properties, we will discuss them one by one. We should add, that the theoretical investigation of deep learning concepts for PDEs, beyond more or less direct applications of the universal approximation theorem, is still scarce. This is in contrast to the classical PDE theory, where there exists an exhaustive literature on the analytic properties of different types of PDEs. The theoretical characterisation of PDEs, as well as the type of theoretical background needed for proving solvability and uniqueness, differentiates between linear and non-linear elliptic, parabolic or hyperbolic equations in terms of the given boundary conditions. Analytic tools are different for these different classes, hence it is somewhat surprising that DL concepts for PDEs hardly ever use this classical classification but rather use a classification based on whether or not the PDE (no matter which type of PDE) is known, which type of data is given for training and which type of network architecture as well as which loss function for training is used.

We follow this data-driven classification scheme. Our first level of characterisation is based on the input and output structure of the network. A first class of networks takes xx or (t,x)(t,x) as input and outputs a one- or low-dimensional scalar value or vector uθ(⋅)u_\theta(\cdot). During training the weights Θ\Theta of the network are optimised such that uθ(⋅)=u^(⋅)u_\theta(\cdot)=\hat{u}(\cdot) is an approximation to the solution uu of the PDE at the specified point xx or (t,x)(t,x), i.e. a regression problem for the function uθ≈uu_\theta\approx u is solved. These networks aim at a function approximation. Otherwise, one aims at an operator approximation for the parameter-to-state map and uses networks which take a parameter function, either in discretised form or as a vector of precomputed feature values, as input and outputs the full solution function uu. The solution uu again is delivered either in discrete form or as a related feature vector, that can be turned into a proper function representation in a post-processing step. The general idea of these two characterisations is summarised in the following Figures 1-2:

  • Function evaluation, neural network uθ(t,x)≈u(t,x)u_\theta(t,x)\approx u(t,x) based on the PINN concept, [137]

Image not extracted

Figure 1. A network with a low input dimension for (t,x)(t,x), which outputs a scalar or low-dimensional u^(t,x)=uθ(t,x)\hat{u}(t,x)=u_\theta(t,x). Hence, the neural network is trained to be a function approximation of the solution u^(t,x)≈u(t,x)\hat{u}(t,x)\approx u(t,x). During training, automatic differentiation by backpropagation is used to compute all necessary derivatives for checking, whether the PDE is satisfied. In addition, the loss function typically sums up over several values of (t,x)(t,x) and their corresponding function values. After training, the Deep Neural Network is then used and u^\hat{u} can be evaluated at arbitrary points (t,x)(t,x).

  • Operator evaluation, neural network ΦΘ(λ)\Phi_{\Theta}(\lambda) following the PCANN concept [12]

Typical examples of DL concepts aiming for a function approximation are Deep Ritz method [165, 107], Physics-Informed Neural Network (PINN) [137], Deep Galerkin Method (DGM) [144], Weak Adversarial Networks [167], and Deep Splitting Approximation methods [9]. In all these concepts, information of the underlying physics must be used when training the neural network, i.e. the PDE must be known analytically. These concepts are usually mesh-independent and accurate, while they require knowledge of the governing laws of the PDEs, and the optimisation problem needs to be solved for every new instance. This is similar to classical methods.

Examples for the second type, i.e. methods aiming at an operator approximation, include Model reduction neural network (PCANN) [12], Deep Operator Network (DeepONet) [113] and its extensions [156, 48], Fourier neural operator (FNO) [102] and its physics-informed extension—Physics Informed Neural Operator (PINO) [105], Graph Kernel Network [103, 104], Bayesian deep convolutional encoder-decoder networks [170], Wavelet Neural Operators [150], Multiwavelet based operator (MWT) [58] and many more. These operator methods usually learn the neural network with some paired input-output (parameter-solution) observations with or without the knowledge of the physical system (using the physics information could sometimes alleviate the need for much data as in the case of PINO and PI-DeepONet) as highlighted in Table 1. Thus, the neural network only needs to be trained once, and a new parameter identification task can be directly solved by a forward pass of the network. Some of these deep learning methods still need to discretise the domain of interest, and the solutions are sought in a finite linear space [170]. A recently published paper [29] aims at comparing some of these operator approximation concepts for solving various PDEs. The primary criterion for comparison in this well-written paper is the cost-to-accuracy curve. Accordingly, the authors compare results obtained with different network sizes. However, their comparison includes the basic methods PCANN, DeepONet and FNO which in our tests did not yield the best results. Also, the numerical test stays in a range - when compared with FEM methods - of rather large approximation errors. This matches our experience, it is hard to get to high precision for forward solvers with DL concepts. Anyway, our focus is on inverse problems, which are not covered in the mentioned paper.

NetworksPDE usageData usageNeural Network approximationParameterSolutionReference
Deep Ritzyesnofunctionnono[165, 107]
PINNyesnofunctionnono[137]
WANyesnofunctionnono[167, 6]
PCANNnomanyoperatorfree gridsfree grids[12]
FNOnomanyoperatorfree gridsfree grids[102]
UFNOnomanyoperatorfree gridsfree grids[159]
MWTnomanyoperatorfree gridsfree grids[58]
DeepONetnomanyoperatorfixedno[113]
PINOyesif neededoperatorfree gridsfree grids[105]
PI-DeepONetyesif neededoperatorfixedno[156]

Table 1. The list of DL concepts considered in this survey is based on a pre-selection mirroring our own interest in inverse problems. Hence, concepts for operator approximation are preferred. In their original papers, most of these concepts were designed for Level 1 problems only. Hence, we need to specify in later sections how to extend them for Level 2 or Level 3 problems. Also, we will later refine the features characterising the different methods in terms of limitations, e.g. which PDE equations are most suitable and which are not.

In terms of the involved computational load, the computational cost of classical methods for solving linear PDEs is mainly determined by the need to solve a large linear system. For deep learning methods, it is the training process (optimisation) of the neural network. As a general rule of thumb, for low dimensional problems with regular geometries, classical methods are in general more powerful and efficient than deep learning methods. Ignoring rounding errors, the main error of the classical methods arises from the step size of the discretisation. There exists a trade-off on the resolution: coarse grids are fast but less accurate; fine grids are more accurate but slower. On the other hand, for DL approaches one usually has to carefully choose the neural network, optimisation method and hyperparameters in consideration of the underlying differential equation. Additionally, the error of deep learning methods is usually difficult to estimate and consists of several factors. The accuracy can be characterised by dividing the total error into three main types: approximation, optimisation, and generalisation errors. More specifically, the approximation error measures the smallest difference between the neural networks and the underlying function or operator, which we aim to approximate. This error is influenced by the size and architecture of the neural networks. The optimisation error arises from the non-convexity property of the loss function and the limited number of iterations used for its minimisation. Stochastic Gradient Descent is an efficient algorithm for escaping local minima and reducing the optimisation error, nevertheless, in real-life applications, we never achieve a global minimum. The generalisation error refers to how the trained networks perform on unseen data and is mainly affected by the amount of training data, the modelling information of the PDEs, and the architecture of the neural networks.

The selection of DL-based PDE solvers which are further investigated in this survey is based on our endeavour to cover a most complete range of different concepts. Hence we included the basic concepts of learned DL solvers as well as some recent improvements of these.

A more detailed description of these approaches, their algorithmic implementation and their analytical properties will be given in the subsequent sections. There we will also highlight, how these methods, which were typically designed as PDE-forward solvers can be extended to parameter identification problems.

Some remarks on concepts not covered in this survey

In this section, we want to highlight those directions of research, which are not covered in this survey.

First of all, there exist several specialised and powerful DL concepts for complex but specific PDEs. Research on neural networks for such specialised cases of PDEs as well as related parametric studies and inverse problems is exploding and it is impossible to give an exhaustive list of references. Hence, we only want to highlight areas of research, which have reached a certain level of maturity in terms of experimental success as well as mathematical rigor. Again, this selection is based on personal preferences and we apologise to those, who quite rightly might want to see their own work included.

A particularly successful area for DL applications are coupled-physics systems, which lead to some of the most complex inverse problems. Typically, those inverse problems cannot be solved by standard concepts, neither in the classical analytical regularisation setting nor in data-driven frameworks. Hence, such problems, e.g. opto-acoustic tomography, require specialised concepts, which typically integrate domain-specific expert knowledge or at least partially draw their motivation from analytic reconstruction formulae, see e.g. [26, 148, 66, 4]. While we are at it, we also mention successes of DL methods in other sub-fields of tomography such as electrical impedance tomography [64, 69], diffuse optical tomography [126] and computed tomography [99, 98, 98]. Another class of papers do take numerical schemes for PDEs as a starting point for developing network architectures, see e.g. [60, 140]. In these papers, the different layers of a neural network are regarded as time stepping in a discretised scheme. In particular, this allows to specifically mimic and improve numerical schemes for parabolic equations, which e.g. are the basis for many imaging problems based on diffusion processes. Simulation of turbulent flows also is an area, where DL concepts have shown good success. We only list some papers, which in our opinion together with the cited literature therein allow an overview of this topic [153, 121, 34]. One should also mention the spectacular results on simulating molecular dynamics [83, 14] and many other fields of applications, where DL simulations lead to groundbreaking novel insights.

We also do not cover the topic of DL for stochastic PDEs and recent results, which we regard to be more on the side of PDE theory and simulation. Hence, we do not cover recent developments as e.g. described in [36, 17, 32].

Secondly, we want to remark on general regularisation schemes for inverse problems using a data-driven approach, which can be applied to but are not particularly tailored for PDE-based parameter identification problems. Typically these general data-driven regularisation schemes for parameter identification problems do need an evaluation of FF or any form of its adjoints. For reference to this type of schemes we refer to [5] or to [1] for a recent paper, which embeds the learning of the network together with the parameter identification into a regularisation scheme. We would like to remark that unrolled iteration schemes such as LISTA or unrolled primal-dual, are somewhat hybrid methods and could be included in our survey. However, LISTA is predominately successful for linear forward operators AA and it still needs the adjoint A∗A^{*} for initialising the input of the network. One could extend such DL schemes, which are derived from classical regularisation theory, in such a way that the evaluation FF or its adjoint would be done by training an embedded network for every operator evaluation. However, this leads to a rather complex training task and this extension is not part of the scope of the original papers for e.g. primal-dual, NETT or DeepPrior or similar [2, 101, 32].

Characterisation of DL concepts for PDEs

Having specified the restricted scope of the present survey, we will now list the concepts under consideration in the following sections. Before starting the introduction of the individual methods we include some general remarks on the characterisation of deep learning methods for PDEs. As already mentioned, the characterisation of DL concepts for solving PDEs differs considerably from the analytic characterisation of PDEs. They are mainly characterised in terms of the information needed for applying them. The two main classes of information are data and physical models, where ’physical models’ refers to the mathematical formulation of the differential equations and their boundary conditions.

Data

Data-driven methods rely on the availability of sufficiently large data sets of good quality. The data provided can be sampled values of single or multiple solutions, where sampling can be done either on a predefined grid or with arbitrary sampling points. Some of the most advanced theoretical investigations do start from assuming a sampling in function space, which opens the path for incorporating functional analytical machinery in their analysis, see e.g. [12, 136, 1].

However, the volume of available data is usually rather scarce,e.g. useful experimental data is generally limited or even intractable for many practical scenarios and high-fidelity numerical simulations are often computationally expensive. Sometimes it is also a challenge to extract interpretable information and knowledge from the data deluge. Moreover, purely data-driven methods may fit observations very well, but predictions may be physically inconsistent or implausible. Theoretically, finite data can not fully and correctly determine a mapping which maps an infinite set to an infinite set. However, we can still see the success of data-driven methods, which is mainly due to two reasons: first of all the search for a solution or parameter typically follows a certain prior distribution, which we might be able to approximate well with few samples; and secondly most problems are continuous, which in connection with suitable regularisation schemes leads to good generalisation properties even for limited sets of training data, [137]

In addition to sampled data, there exists a wealth of domain-specific expert knowledge for most applications. This information may come in the form of observational, physical or mathematical understanding of the system under consideration. Integrating data and this prior knowledge can yield more interpretable machine learning methods and can improve the accuracy and generalisation performances without large amounts of data.

In our context information from physics may come in different forms: most commonly, the system of partial differential equations as well as boundary conditions are specified. Otherwise, some energy functionals or a weak formulation may be given. Also, certain approaches [142] incorporate conservation laws in the network architecture. Whenever such information is used, the concept is typically called ’physics informed’.

Different from data-driven methods, the physics information may uniquely determine the systems, however, only within the limitations of the chosen model. For example, given the PDE formula and some initial and boundary equations, the solution is uniquely determined if the system is well-posed. In other words, mathematical formulae may inherently contain all information of the physical systems, while data are just some observations and reflections of the physical systems and thus can not fully reveal the whole information of the systems.

The most common approach is to use this physics information in the loss function during the training of the network. Hence, the type of physics information determines the loss function.

Network training and application

Training the resulting networks typically leads to additional problems. This may require extensive hyperparameter search, data preprocessing or subtle parameter settings for controlling the convergence of the training scheme. We have listed the respective training concepts for each method in the Appendix A.

As pointed out in [84], specific to physics-informed learning, there are currently three pathways that can be followed separately or in tandem to accelerate training and enhance the generalisation of ML models by embedding physics in them. The first one is observational biases from observational data, which is the foundation of data-driven methods in machine learning. The second is inductive biases by designing special NN architectures that can automatically and exactly satisfy some properties and conditions. The most common example are convolutional NNs which can be designed to respect properties, such as symmetry, rotations and reflections. Other examples are Graph neural networks [103] which are adapted to graph-structured data. Another concept is based on Quadratic residual networks(QREs) [16], which increases the potential of the neural networks to model non-linearities by adding a quadratic residual term to the weighted sum of inputs before applying activation functions. They were shown to have good performance in learning high-frequency patterns. The invertible neural network [3] makes the inverse of the neural network well-defined. It has applications in inverse problems and generative models. Specific architectures can also be designed to satisfy the initial/boundary conditions. The last pathway is learning biases, where the physics constraints are imposed in a soft manner by appropriately penalising the loss function of conventional NN approximations. Representative examples include the Deep Ritz method and PINN.

We also want to comment on the arguably most important tools for physics-informed machine learning, namely automatic differentiation [8]. In principle, this general concept us to compute exactly the derivatives of the network output with respect to the input variables. Hence, we can incorporate weak or strong formulations of the underlying PDE conveniently. However, compared to conventional numerical gradients such as finite difference, the automatic differentiation method is usually slower and requires more memory.

Moreover, after training, the methods will again differ substantially in their scope of applicability. Methods might allow the sampling of a solution of a PDE at arbitrary points. In this case, the method is called meshfree, which is one of the most desirable properties of most DL concepts for PDEs. Another important property, in particular for Level 2 and Level 3 problems, is the computational load needed for applying the network to a different parameter of the PDEs. This is most efficient for most operator learning schemes but might require re-training or the computation of specific feature vectors.

In conclusion, different deep learning methods for PDEs are mainly different in terms of their neural network architecture and the particular choice of the loss function for training.

General DL concepts for PDE-based inverse problems

So far, we have sketched two main approaches for solving PDEs (forward problems), namely function or operator approximation schemes. In the following sections, we will introduce these methods and their baseline. implementation in more detail. We will also include, whenever appropriate, the related concepts for the PDE-based inverse problems.

The extension of function approximations uΘ(t,x)u_{\Theta}(t,x) for solving inverse problems is not straightforward and differs from method to method. In contrast to that, the application of learned operator approximation schemes ΦΘ(λ)\Phi_{\Theta}(\lambda) to inverse problems is always based on one of the two following concepts:

  • The direct (or backward) method, where the network is trained with reversed input-output. In other words, the solution of the PDE is the input of a neural network which outputs the parameter function. This method is similar to the forward problem, but we learn the backward operator.

  • Tikhonov Regularisation, for this case, the forward operator ΦΘ(λ)\Phi_{\Theta}(\lambda) is trained as usual. For an inverse problem with given noisy data uδu^{\delta} we then approximate a suitable parameter λ\lambda by minimising a Tikhonov functional with respect to λ\lambda

∥ΦΘ(λ)−uδ∥2+αR(λ).\left\|\Phi_{\Theta}(\lambda)-u^{\delta}\right\|^{2}+\alpha R(\lambda).

The choice of the penalty term RR adds to the flexibility of the method and can encode further properties of the solution such as smoothness, sparsity or piece-wise constant structures. minimisation is done iteratively via gradient descent, where the differentiation with respect to λ\lambda can be implemented using backpropagation with respect to the input but with fixed weights Θ\Theta. We exemplify the procedure for the model reduction concept PCANN in Algorithm 6. A similar idea can be applied to the other solution operators mentioned in § 5.2 and § 5.4.

Theoretical Results of Deep Learning for PDEs

One reason for the success of finite element methods is that the functions in Sobolev space can be approximated by piece-wise polynomials with a certain convergence rate depending on the mesh size. However, classical methods require degrees of freedom of the order of ε−d\varepsilon^{-d} for a desired accuracy ε\varepsilon in dd dimensional problems, which limits their applications to high-dimensional problems. The task of deep learning for PDEs is to employ neural networks to approximate the solution or parameter-to-state operator with a limited amount of data or physical information. A fundamental question is therefore whether the neural network can effectively approximate the solution/parametric operator of a PDE system, and if so, how to estimate the complexity of a neural network in terms of the dimensionality growth and accuracy requirements. In this section, we will briefly review some approximation theoretical aspects of deep learning for PDEs.

Approximation Results for Solution Learning in PDEs

Neural networks are universal approximators [28, 71] in the sense that continuous functions on compact domains can be uniformly approximated by neural networks with arbitrary accuracy, provided the number of neurons is sufficiently large. A lower bound on the size of the neural network for achieving a desired accuracy was not given in these papers. With some assumptions on the activation functions and the space of functions to be considered, more refined results on the relationship between the size and the approximation accuracy of neural networks have been reported in e.g., [7, 13, 118, 143, 112, 31, 50, 56, 57]. The universal approximation theorem has also been generalised to convolutional neural networks in [169] and complex-valued neural networks in [152]. These results concern general function approximations and do not refer to PDE solutions.

Neural Networks Approximation for High Dimensional PDEs

We now shortly discuss the theoretical foundations for approximating solutions of PDEs by neural networks. Recent years have witnessed great empirical success of various deep learning methods [165, 65, 73, 167] in particular for solving high-dimensional PDEs. To explain these successes, several mathematical results have been proven which show that neural networks have the ability to approximate solutions of some PDEs without the curse of dimensionality (CoD). In particular, in [52] it is shown that the solutions of linear Black-Scholes PDEs can be approximated by neural networks with the size increasing at most polynomially with respect both the reciprocal of the prescribed approximation accuracy and the PDE dimension dd. A number of articles have appeared that significantly extend the results in [52] to more classes of PDEs [10, 46, 72, 75, 80, 139]. In particular, in [75] it is proven that neural networks have the expressiveness to overcome the CoD for semilinear heat PDEs.

On the other hand, there are also some articles, e.g. [54, 133, 162, 51], which derive some lower bounds on the complexity of neural networks with ReLU activation function to achieve a certain accuracy and show that there are natural classes of functions for which deep neural networks with ReLU activation cannot escape the CoD.

Barron Spaces

Typical classical approximation results for PDE solutions use a setting in Sobolev or Besov spaces. These spaces can be defined by their approximation properties using piece-wise polynomials or wavelets. In the same sense, the Barron space [22, 37] is the analogue when we consider approximation by two-layer neural networks. Roughly speaking, the Barron space consists of the functions that can be approximated by two-layer neural networks, and the approximation error is governed by the norm of the Barron space. A nice property of Sobolev/Besov type spaces is that solutions to partial differential equations lie in these spaces. This is the core of regularity theory for PDEs. When introducing Barron spaces in PDEs, a natural corollary and fundamental question is whether the solutions of the dimensional partial differential equations (PDEs) we are interested in belong to a Barron space. We will briefly review the definition of the Barron space and its approximation theorems, as well as an example of its use in escaping CoD in high-dimensional PDEs.

Consider functions g⊂Rd:X→Rg \subset\mathbb{R}^{d}: X \to\mathbb{R} which allow the below representation:

g(x;Θ)=∫Ωaσ(wTx+b)ρ(da,dw,db),x∈Rd,(3)g(x;\Theta)=\int_{\Omega}a\sigma(w^{T}x+b)\rho(da,dw,db),\quad x\in\mathbb{R}^{d}, \tag*{(3)}

where ρ\rho is a probability distribution on Ω=R×Rd×R\Omega=\mathbb{R}\times\mathbb{R}^{d}\times\mathbb{R} and σ:R→R\sigma:\mathbb{R}\to\mathbb{R} is some activation function. This representation can be thought of as a continuum realisation of two-layer neural networks, which are given as

gn(x,Θ)=1n∑k=1nakσ(wkTx+bk),x∈Rd.(4)g_n(x,\Theta)=\frac{1}{n}\sum_{k=1}^{n}a_k\sigma(w_k^{T}x+b_k),\quad x\in\mathbb{R}^{d}. \tag*{(4)}

We present the following definition from [22] for Barron functions defined in a domain Ω⊂Rd\Omega\subset\mathbb{R}^{d}. This definition is an adaptation of the definition given in [37], with some crucial modifications, for the purpose of analysing PDEs.

Definition 3.1. For a domain Ω⊂Rd\Omega\subset\mathbb{R}^{d} and a fixed radius R∈[0,+∞]R\in[0,+\infty], the corresponding Barron space with index pp is defined as

BRp(Ω)={f:∥f∥BRp(Ω)<∞},(5)\mathcal{B}_{R}^{p}(\Omega)=\{f:\|f\|_{\mathcal{B}_{R}^{p}(\Omega)}<\infty\}, \tag*{(5)}

where

∥f∥BRp(Ω):=inf⁡ρ∈PR{(∫∣a∣pρ(da,dw,db))1/p:f=∫aσ(wTx+b)ρ(da,dw,db)}(6)\|f\|_{\mathcal{B}_{R}^{p}(\Omega)}:=\inf_{\rho\in\mathcal{P}_{R}}\left\{\left(\int|a|^{p}\rho(da,dw,db)\right)^{1/p}:f=\int a\sigma(w^{T}x+b)\rho(da,dw,db)\right\} \tag*{(6)}

and PR:={ρ:ρ is supported on R×{x∈Rd:∥x∥≤R}×R}\mathcal{P}_{R}:=\left\{\rho:\rho\text{ is supported on }\mathbb{R}\times\{x\in\mathbb{R}^{d}:\|x\|\leq R\}\times\mathbb{R}\right\}.

The most essential feature that distinguishes Barron function spaces from Sobolev or Besov spaces is that elements of the former can be approximated by a two-layer neural network with an approximation rate that is independent of the dimension, as shown in the next theorem [22]:

References

  1. [1]C. Aarset, M. Holler, and T. T. N. Nguyen. Learning-informed parameter identification in nonlinear time-dependent PDEs. arXiv preprint arXiv:2202.10915, 2022.DOI
  2. [2]J. Adler and O. Öktem. Learned primal-dual reconstruction. IEEE transactions on medical imaging, 37(6):1322–1332, 2018.
  3. [3]L. Ardizzone, J. Kruse, S. Wirkert, D. Rahner, E. W. Pellegrini, R. S. Klessen, L. Maier-Hein, C. Rother, and U. Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv:1808.04730, 2018.arxiv.org/abs/1808.04730
  4. [4]S. Arridge and A. Hauptmann. Networks for nonlinear diffusion problems in imaging. Journal of Mathematical Imaging and Vision, 62(3):471–487, 2020.
  5. [5]S. Arridge, P. Maass, O. Öktem, and C.-B. Schönlieb. Solving inverse problems using data-driven models. Acta Numerica, 28:1–174, 2019.
  6. [6]G. Bao, Y. Ye, Y. Zang, and H. Zhou. Numerical solution of inverse problems by weak adversarial networks. Inverse Problems, 36(11):115003, 2020.DOI
  7. [7]A. R. Barron. Approximation and estimation bounds for artificial neural networks. Machine learning, 14:115–133, 1994.
  8. [8]A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research, 18:1–43, 2018.
  9. [9]C. Beck, S. Becker, P. Cheridito, A. Jentzen, and A. Neufeld. Deep splitting method for parabolic PDEs. SIAM Journal on Scientific Computing, 43(5):A3135–A3154, 2021.DOI
  10. [10]P. Beneventano, P. Cheridito, A. Jentzen, and P. von Wurstemberger. High-dimensional approximation spaces for artificial neural networks and applications to partial differential equations. arXiv preprint arXiv:2012.04326, 2020.arxiv.org/abs/2012.04326
  11. [11]P. Benner, W. Schilders, S. Grivet-Talocia, A. Quarteroni, G. Rozza, and L. Miguel Silveira. Model Order Reduction: Volume 2: Snapshot-Based Methods and Algorithms. De Gruyter, 2020.
  12. [12]K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model reduction and neural networks for parametric PDEs. arXiv preprint arXiv:2005.03180, 2020.arxiv.org/abs/2005.03180
  13. [13]H. Bolcskei, P. Grohs, G. Kutyniok, and P. Petersen. Optimal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science, 1(1):8–45, 2019.
  14. [14]L. Böselt, M. Thürlemann, and S. Riniker. Machine learning in QM/MM molecular dynamics simulations of condensed-phase systems. Journal of Chemical Theory and Computation, 17(5):2641–2658, 2021.DOI
  15. [15]S. C. Brenner and L. R. Scott. The Mathematical Theory of Finite Element Methods, volume 15 of Texts in Applied Mathematics. Springer, 2008.
  16. [16]J. Bu and A. Karpatne. Quadratic residual networks: A new class of neural networks for solving forward and inverse problems in physics involving PDEs. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 675–683. SIAM, 2021.arxiv.org/abs/2101.08366
  17. [17]M. Burger and A. Neubauer. Analysis of Tikhonov regularization for function approximation by neural networks. Neural Networks, 16(1):79–90, 2003.
  18. [18]G. Chen, X. Liu, Y. Li, Q. Meng, and L. Chen. Laplace neural operator for complex geometries. arXiv preprint arXiv:2302.08166, 2023.
  19. [19]J.-S. Chen, M. Hillman, and S.-W. Chi. Meshfree methods: Progress made after 20 years. Journal of Engineering Mechanics, 143(4):04017001, 2017.DOI
  20. [20]T. Chen and H. Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911–917, 1995.
  21. [21]Y. Chen, L. Lu, G. E. Karniadakis, and L. Dal Negro. Physics-informed neural networks for inverse problems in nano-optics and metamaterials. Optics express, 28(8):11618–11633, 2020.DOI
  22. [22]Z. Chen, J. Lu, and Y. Lu. On the representation of solutions to elliptic PDEs in Barron spaces. Advances in neural information processing systems, 34:6454–6465, 2021.arxiv.org/abs/2106.07539
  23. [23]Z. Chen, J. Lu, Y. Lu, and S. Zhou. A regularity theory for static schrödinger equations on ℝ^d in spectral Barron spaces. SIAM Journal on Mathematical Analysis, 55(1):557–570, 2023.
  24. [24]F. Chinesta, A. Huerta, G. Rozza, and K. Willcox. Model order reduction. Encyclopedia of computational mechanics, 2016.
  25. [25]L. Cicci, S. Fresca, and A. Manzoni. Deep-HyROMnet: A deep learning-based operator approximation for hyper-reduction of nonlinear parametrized PDEs. arXiv preprint arXiv:2202.02658, 2022.arxiv.org/abs/2202.02658
  26. [26]D. Colombo, E. Turkoglu, W. Li, and D. Rovetta. Coupled physics-deep learning inversion. Computers & Geosciences, 157:104917, 2021.
  27. [27]S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physics-informed neural networks: where we are and what’s next. Journal of Scientific Computing, 92(3):88, 2022.arxiv.org/abs/2201.05624
  28. [28]G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.
  29. [29]M. De Hoop, D. Z. Huang, E. Qian, and A. M. Stuart. The cost-accuracy trade-off in operator learning with neural networks. arXiv preprint arXiv:2203.13181, 2022.arxiv.org/abs/2203.13181
  30. [30]B. Deng, Y. Shin, L. Lu, Z. Zhang, and G. E. Karniadakis. Convergence rate of DeepONets for learning operators arising from advection-diffusion equations. arXiv preprint arXiv:2102.10621, 2021.arxiv.org/abs/2102.10621
  31. [31]R. DeVore, B. Hanin, and G. Petrova. Neural network approximation. Acta Numerica, 30:327–444, 2021.
  32. [32]S. Dittmer, T. Kluth, P. Maass, and D. O. Baguer. Regularization by architecture: A deep prior approach for inverse problems. Journal of Mathematical Imaging and Vision, 62(3):456–470, 2019.DOI
  33. [33]P. Dondl, J. Müller, and M. Zeinhofer. Uniform convergence guarantees for the Deep Ritz method for nonlinear problems. arXiv preprint arXiv:2111.05637, 2021.arxiv.org/abs/2111.05637
  34. [34]C. Drygala, B. Winhart, F. di Mare, and H. Gottschalk. Generative modeling of turbulence. Physics of Fluids, 34(3):035114, 2022.
  35. [35]C. Duan, Y. Jiao, Y. Lai, X. Lu, and Z. Yang. Convergence rate analysis for Deep Ritz. arXiv preprint arXiv:2103.13330, 2021.DOI
  36. [36]W. E, J. Han, and A. Jentzen. Algorithms for solving high dimensional PDEs: From nonlinear Monte Carlo to machine learning. Nonlinearity, 35(1):278, 2021.DOI
  37. [37]W. E, C. Ma, and L. Wu. The Barron space and the flow-induced function spaces for neural network models. Constructive Approximation, 55(1):369–406, 2022.
  38. [38]N. Elsner. Grundlagen der technischen Thermodynamik. Akademie-Verlag, 1988.
  39. [39]N. R. Franco, S. Fresca, A. Manzoni, and P. Zunino. Approximation bounds for convolutional neural networks in operator learning. Neural Networks, 161:129–141, 2023.
  40. [40]S. Fresca and A. Manzoni. Real-time simulation of parameter-dependent fluid flows through deep learning-based reduced order models. Fluids, 6(7):259, 2021.DOI
  41. [41]S. Fresca and A. Manzoni. POD-DL-ROM: enhancing deep learning-based reduced order models for nonlinear parametrized PDEs by proper orthogonal decomposition. Computer Methods in Applied Mechanics and Engineering, 388:114181, 2022.DOI
  42. [42]S. Fresca, A. Manzoni, L. Dedè, and A. Quarteroni. Deep learning-based reduced order models in cardiac electrophysiology. PloS one, 15(10):e0239416, 2020.
  43. [43]S. Fresca, L. Dede, and A. Manzoni. A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs. Journal of Scientific Computing, 87(2):1–36, 2021.
  44. [44]S. Fresca, G. Gobat, P. Fedeli, A. Frangi, and A. Manzoni. Deep learning-based reduced order models for the real-time simulation of the nonlinear dynamics of microstructures. International Journal for Numerical Methods in Engineering, 123(20):4749–4777, 2022.
  45. [45]S. Garg and S. Chakraborty. Variational bayes deep operator network: A data-driven bayesian solver for parametric differential equations. arXiv preprint arXiv:2206.05655, 2022.arxiv.org/abs/2206.05655
  46. [46]L. Gonon, P. Grohs, A. Jentzen, D. Kofler, and D. Šiška. Uniform error estimates for artificial neural network approximations for heat equations. IMA Journal of Numerical Analysis, 42(3):1991–2054, 2022.DOI
  47. [47]P. Gopalani, S. Karmakar, and A. Mukherjee. Capacity bounds for the DeepONet method of solving differential equations. arXiv preprint arXiv:2205.11359, 2022.
  48. [48]S. Goswami, M. Yin, Y. Yu, and G. Karniadakis. A physics-informed variational DeepONet for predicting the crack path in brittle materials. arXiv preprint arXiv:2108.06905, 2021.DOI
  49. [49]S. Goswami, A. Bora, Y. Yu, and G. E. Karniadakis. Physics-informed deep neural operator networks. arXiv preprint arXiv:2207.05748, 2022.arxiv.org/abs/2207.05748
  50. [50]R. Gribonval, G. Kutyniok, M. Nielsen, and F. Voigtlaender. Approximation spaces of deep neural networks. Constructive approximation, 55(1):259–367, 2022.
  51. [51]P. Grohs and F. Voigtlaender. Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces. arXiv preprint arXiv:2104.02746, 2021.arxiv.org/abs/2104.02746
  52. [52]P. Grohs, F. Hornung, A. Jentzen, and P. von Wurstemberger. A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of black-scholes partial differential equations. arXiv preprint arXiv:1809.02362, 2018.DOI
  53. [53]P. Grohs, A. Jentzen, and D. Salimova. Deep neural network approximations for solutions of PDEs based on Monte Carlo algorithms. Partial Differential Equations and Applications, 3(4):45, 2022.DOI
  54. [54]P. Grohs, S. Ibragimov, A. Jentzen, and S. Koppesteiner. Lower bounds for artificial neural network approximations: A proof that shallow neural networks fail to overcome the curse of dimensionality. Journal of Complexity, page 101746, 2023.DOI
  55. [55]T. G. Grossmann, U. J. Komorowska, J. Latz, and C.-B. Schönlieb. Can physics-informed neural networks beat the finite element method? arXiv preprint arXiv:2302.04107, 2023.arxiv.org/abs/2302.04107
  56. [56]I. Gü hring, G. Kutyniok, and P. Petersen. Error bounds for approximations with deep ReLU neural networks in W^{s,p} norms. Analysis and Applications, 18(05):803–859, 2020.arxiv.org/abs/1902.07896
  57. [57]I. Gühring, M. Raslan, and G. Kutyniok. Expressivity of deep neural networks. arXiv preprint arXiv:2007.04759, 2020.arxiv.org/abs/2007.04759
  58. [58]G. Gupta, X. Xiao, and P. Bogdan. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems, 34:24048–24062, 2021.arxiv.org/abs/2109.13459
  59. [59]J. Gwinner and E. P. Stephan. Advanced boundary element methods. Springer, 2018.
  60. [60]E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1):014004, 2017.
  61. [61]P. S. Hadorn. Shift-DeepONet: Extending deep operator networks for discontinuous output functions. In Master thesis, ETH Computational Science and Engineering. ETH Zurich, Seminar for Applied Mathematics, 2022.DOI
  62. [62]N. Halko, P.-G. Martinsson, Y. Shkolnisky, and M. Tygert. An algorithm for the principal component analysis of large data sets. SIAM Journal on Scientific computing, 33(5):2580–2594, 2011.
  63. [63]N. Halko, P.-G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217–288, 2011.
  64. [64]S. J. Hamilton and A. Hauptmann. Deep D-bar: Real-time electrical impedance tomography imaging with deep neural networks. IEEE transactions on medical imaging, 37(10):2367–2377, 2018.DOI
  65. [65]J. Han, A. Jentzen, and W. E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
  66. [66]A. Hauptmann and B. T. Cox. Deep learning in photoacoustic tomography: current approaches and future directions. Journal of Biomedical Optics, 25(11):112903, 2020.
  67. [67]D. Hendrycks and K. Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 3, 2016.
  68. [68]D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
  69. [69]W. Herzberg, D. B. Rowe, A. Hauptmann, and S. J. Hamilton. Graph convolutional networks for model-based learning in nonlinear inverse problems. IEEE transactions on computational imaging, 7:1341–1353, 2021.
  70. [70]J. S. Hesthaven and S. Ubbiali. Non-intrusive reduced order modeling of nonlinear problems using neural networks. Journal of Computational Physics, 363:55–78, 2018.
  71. [71]K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  72. [72]F. Hornung, A. Jentzen, and D. Salimova. Space-time deep neural network approximations for high-dimensional partial differential equations. arXiv preprint arXiv:2006.02199, 2020.arxiv.org/abs/2006.02199
  73. [73]C. Huré, H. Pham, and X. Warin. Deep backward schemes for high-dimensional nonlinear PDEs. Mathematics of Computation, 89(324):1547–1579, 2020.
  74. [74]M. Hutzenthaler, A. Jentzen, T. Kruse, et al. On multilevel picard numerical approximations for high-dimensional nonlinear parabolic partial differential equations and high-dimensional nonlinear backward stochastic differential equations. Journal of Scientific Computing, 79(3):1534–1571, 2019.DOI
  75. [75]M. Hutzenthaler, A. Jentzen, T. Kruse, and T. A. Nguyen. A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations. SN partial differential equations and applications, 1:1–34, 2020.
  76. [76]A. Jacot, F. Gabriel, and C. Hongler. Neural Tangent Kernel: Convergence and generalization in neural networks. In NIPS’18, page 8580–8589. Curran Associates Inc., 2018.arxiv.org/abs/1806.07572
  77. [77]A. D. Jagtap and G. E. Karniadakis. Extended physics-informed neural networks (XPINNs): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communications in Computational Physics, 28(5):2002–2041, 2020.
  78. [78]A. D. Jagtap, E. Kharazmi, and G. E. Karniadakis. Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365:113028, 2020.
  79. [79]A. Jefferies, J. Kuhnert, L. Aschenbrenner, and U. Giffhorn. Finite pointset method for the simulation of a vehicle travelling through a body of water. In Meshfree methods for partial differential equations VII, pages 205–221. Springer, 2015.
  80. [80]A. Jentzen, D. Salimova, and T. Welti. A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients. arXiv preprint arXiv:1809.07321, 2018.DOI
  81. [81]Y. Jiao, Y. Lai, Y. Lo, Y. Wang, and Y. Yang. Error analysis of Deep Ritz methods for elliptic equations. arXiv preprint arXiv:2107.14478, 2021.arxiv.org/abs/2107.14478
  82. [82]P. Jin, S. Meng, and L. Lu. Mionet: Learning multiple-input operators via tensor product. arXiv preprint arXiv:2202.06137, 2022.arxiv.org/abs/2202.06137
  83. [83]J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
  84. [84]G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, 2021.
  85. [85]D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.arxiv.org/abs/1412.6980
  86. [86]G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
  87. [87]D. Kolymbas. Barodesy: a new hypoplastic approach. International Journal for Numerical and Analytical Methods in Geomechanics, 36(9):1220–1240, 2012.
  88. [88]D. Kolymbas. Barodesy: a new constitutive frame for soils. Géotechnique Letters, 2(2):17–23, 2012.
  89. [89]N. Kovachki, S. Lanthaler, and S. Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art–No, 2021.
  90. [90]N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481, 2021.DOI
  91. [91]N. Kovachki, B. Liu, X. Sun, H. Zhou, K. Bhattacharya, M. Ortiz, and A. Stuart. Multiscale modeling of materials: Computing, data science, uncertainty and goal-oriented optimization. Mechanics of Materials, 165:104156, 2022.
  92. [92]J. Kuhnert. Meshfree simulations in car design: closing the gaps of classical simulation tools. German Success Stories in Industrial Mathematics, 35:130, 2021.
  93. [93]J. Kuhnert, I. Michel, and R. Mack. Fluid structure interaction (FSI) in the meshfree finite pointset method (FPM): Theory and applications. In International workshop on meshfree methods for partial differential equations, pages 73–92. Springer, 2017.
  94. [94]G. Kutyniok, P. Petersen, M. Raslan, and R. Schneider. A theoretical analysis of deep neural networks and parametric PDEs. Constructive Approximation, 55(1):73–125, 2022.
  95. [95]S. Lanthaler. Operator learning with PCA-Net: upper and lower complexity bounds. arXiv preprint arXiv:2303.16317, 2023.arxiv.org/abs/2303.16317
  96. [96]S. Lanthaler, S. Mishra, and G. E. Karniadakis. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1):tnac001, 2022.
  97. [97]S. Lanthaler, Z. Li, and A. M. Stuart. The nonlocal neural operator: Universal approximation. arXiv preprint arXiv:2304.13221, 2023.
  98. [98]J. Leuschner, M. Schmidt, D. O. Baguier, and P. Maass. LoDoPaB-CT, a benchmark dataset for low-dose computed tomography reconstruction. Scientific Data, 8(1):109, 2021.
  99. [99]J. Leuschner, M. Schmidt, P. S. Ganguly, V. Andriias­hen, S. B. Coban, A. Denker, D. Bauer, A. Had­jifaradji, K. J. Batenburg, P. Maass, et al. Quantitative comparison of deep learning-based image reconstruction methods for low-dose and sparse-angle ct applications. Journal of Imaging, 7(3):44, 2021.
  100. [100]R. J. LeVeque. Finite Volume Methods for Hyperbolic Problems. Cambridge University Press, 2002.
  101. [101]H. Li, J. Schwab, S. Antholzer, and M. Haltmeier. NETT: Solving inverse problems with deep neural networks. Inverse Problems, 36(6):065005, 2020.
  102. [102]Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.arxiv.org/abs/2010.08895
  103. [103]Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485, 2020.arxiv.org/abs/2003.03485
  104. [104]Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, A. Stuart, K. Bhattacharya, and A. Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755–6766, 2020.
  105. [105]Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021.arxiv.org/abs/2111.03794
  106. [106]Z. Li, D. Z. Huang, B. Liu, and A. Anandkumar. Fourier neural operator with learned deformations for PDEs on general geometries. arXiv preprint arXiv:2207.05209, 2022.DOI
  107. [107]Y. Liao and P. Ming. Deep Nitsche method: Deep Ritz method with essential boundary conditions. arXiv preprint arXiv:1912.01309, 2019.DOI
  108. [108]G. Lin, C. Moya, and Z. Zhang. Accelerated replica exchange stochastic gradient langevin diffusion enhanced bayesian DeepONet for solving noisy parametric PDEs. arXiv preprint arXiv:2111.02484, 2021.arxiv.org/abs/2111.02484
  109. [109]B. Liu, N. Kovachki, Z. Li, K. Azizzadenesheli, A. Anandkumar, A. M. Stuart, and K. Bhattacharya. A learning-based multiscale method and its application to inelastic impact problems. Journal of the Mechanics and Physics of Solids, 158:104668, 2022.
  110. [110]D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1-3):503–528, 1989.
  111. [111]L. Liu and W. Cai. Multiscale DeepONet for nonlinear operators in oscillatory function spaces for building seismic wave responses. arXiv preprint arXiv:2111.04860, 2021.arxiv.org/abs/2111.04860
  112. [112]J. Lu, Z. Shen, H. Yang, and S. Zhang. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis, 53(5):5465–5506, 2021.
  113. [113]L. Lu, P. Jin, and G. E. Karniadakis. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.DOI
  114. [114]L. Lu, X. Meng, Z. Mao, and G. E. Karniadakis. DeepXDE: A deep learning library for solving differential equations. SIAM Review, 63(1):208–228, 2021.
  115. [115]L. Lu, R. Pestourie, W. Yao, Z. Wang, F. Verdugo, and S. G. Johnson. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1105–B1132, 2021.
  116. [116]L. Lu, X. Meng, S. Cai, Z. Mao, S. Goswami, Z. Zhang, and G. E. Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
  117. [117]G. Łukaszewicz and P. Kalita. Navier–stokes equations. Advances in Mechanics and Mathematics, 34, 2016.
  118. [118]H. N. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural computation, 8(1):164–177, 1996.
  119. [119]I. Michel, I. Bathaeian, J. Kuhnert, D. Kolymbas, C.-H. Chen, I. Polymerou, C. Vrettos, and A. Becker. Meshfree generalized finite difference methods in soil mechanics—part ii: numerical results. GEM-International Journal on Geomathematics, 8(2):191–217, 2017.
  120. [120]I. Michel, T. Seifarth, J. Kuhnert, and P. Suchde. A meshfree generalized finite difference method for solution mining processes. Computational Particle Mechanics, 8(3):561–574, 2021.
  121. [121]P. M. Milani, J. Ling, and J. K. Eaton. Generalization of machine-learned turbulent heat flux models applied to film cooling flows. Journal of Turbomachinery, 142(1), 2020.
  122. [122]S. Mishra and R. Molinaro. Estimates on the generalization error of physics informed neural networks (PINNs) for approximating PDEs. arXiv preprint arXiv:2006.16144, 2020.arxiv.org/abs/2006.16144
  123. [123]S. Mishra and R. Molinaro. Estimates on the generalization error of physics informed neural networks (PINNs) for approximating a class of inverse problems for PDEs. arXiv preprint arXiv:2007.01138, 2020.arxiv.org/abs/2007.01138
  124. [124]C. Moya and G. Lin. Fed-DeepONet: Stochastic gradient-based federated training of deep operator networks. Algorithms, 15(9):325, 2022.
  125. [125]C. Moya, S. Zhang, M. Yue, and G. Lin. DeepONet-Grid-UQ: A trustworthy deep operator framework for predicting the power grid’s post-fault trajectories. arXiv preprint arXiv:2202.07176, 2022.arxiv.org/abs/2202.07176
  126. [126]M. Mozumder, A. Hauptmann, I. Nissilä, S. R. Arridge, and T. Tarvainen. A model-based iterative learning approach for diffuse optical tomography. IEEE Transactions on Medical Imaging, 41(5):1289–1299, 2021.
  127. [127]J. Müller and M. Zeinhofer. Deep Ritz revisited. arXiv preprint arXiv:1912.03937, 2019.arxiv.org/abs/1912.03937
  128. [128]D. Nganyu Tanyu, I. Michel, A. Rademacher, P. Maass, and J. Kuhnert. Parameter identification by deep learning of a material model for granular media. In preparation, 2022.
  129. [129]I. Ostermann, J. Kuhnert, D. Kolymbas, C.-H. Chen, I. Polymerou, V. Šmilauer, C. Vrettos, and D. Chen. Meshfree generalized finite difference methods in soil mechanics—part i: theory. GEM-International Journal on Geomathematics, 4(2):167–184, 2013.
  130. [130]K. Panetta. Gartner top 10 strategic technology trends for 2019. https://www.gartner.com/smarterwithgartner/gartner-top-10-strategic-technology-trends-for-2018, 2018. Accessed: May 17, 2023.
  131. [131]K. Panetta. Gartner top 10 strategic technology trends for 2020. https://www.gartner.com/smarterwithgartner/gartner-top-10-strategic-technology-trends-for-2019, 2019. Accessed: May 17, 2023.
  132. [132]G. Pang, L. Lu, and G. E. Karniadakis. fPINNs: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing, 41(4):A2603–A2626, 2019.
  133. [133]P. Petersen and F. Voigtlaender. Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Networks, 108:296–330, 2018.
  134. [134]A. Pinkus. Approximation theory of the MLP model in neural networks. Acta numerica, 8:143–195, 1999.
  135. [135]M. Prasthofer, T. De Ryck, and S. Mishra. Variable-input deep operator networks. arXiv preprint arXiv:2205.11404, 2022.arxiv.org/abs/2205.11404
  136. [136]M. Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research, 19(1):932–955, 2018.
  137. [137]M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.
  138. [138]B. Raonić, R. Molinaro, T. Rohner, S. Mishra, and E. de Bezenac. Convolutional neural operators. arXiv preprint arXiv:2302.01178, 2023.
  139. [139]C. Reisinger and Y. Zhang. Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems. Analysis and Applications, 18 (06):951–999, 2020.
  140. [140]L. Ruthotto and E. Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, 62(3):352–364, 2020.
  141. [141]W. Schilders. Introduction to model order reduction. Model order reduction: theory, research aspects and applications, pages 3–32, 2008.
  142. [142]W. Schilders. MSODE: Modelling, simulation and optimization in a data-rich environment. In Mathematics: Key Enabling Technology for Scientific Machine Learning, pages 24–30, 2021.
  143. [143]U. Shaham, A. Cloninger, and R. R. Coifman. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 44(3):537–557, 2018.
  144. [144]J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339–1364, 2018.
  145. [145]J. C. Strikwerda. Finite Difference Schemes and Partial Differential Equations. Society for Industrial and Applied Mathematics, second edition, 2004.
  146. [146]Y. Sun, C. Moya, G. Lin, and M. Yue. DeepGraphONet: A deep graph operator network to learn and zero-shot transfer the dynamic response of networked systems. arXiv preprint arXiv:2209.10622, 2022.arxiv.org/abs/2209.10622
  147. [147]L. Tan and L. Chen. Enhanced DeepONet for modeling partial differential operators considering multiple input functions. arXiv preprint arXiv:2202.08942, 2022.arxiv.org/abs/2202.08942
  148. [148]N. Thuerey, P. Holl, M. Mueller, P. Schnell, F. Trost, and K. Um. Physics-based deep learning. arXiv preprint arXiv:2109.05237, 2021.arxiv.org/abs/2109.05237
  149. [149]TorchPhysics. TORCHPHYSICS: A deep learning library to solve differential equations. https://github.com/boschresearch/torchphysics, 2022. Accessed: May 17, 2023.
  150. [150]T. Tripura and S. Chakraborty. Wavelet neural operator: a neural operator for parametric partial differential equations. arXiv preprint arXiv:2205.02191, 2022.arxiv.org/abs/2205.02191
  151. [151]F. Tröltzsch. Optimal Control of Partial Differential Equations, volume 112 of Graduate Studies in Mathematics. American Mathematical Society, Providence, 2010.
  152. [152]F. Voigtlaender. The universal approximation theorem for complex-valued neural networks. Applied and Computational Harmonic Analysis, 64:33–61, 2023.
  153. [153]R. Wang, K. Kashinath, M. Mustafa, A. Albert, and R. Yu. Towards physics-informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1457–1466, 2020.
  154. [154]S. Wang, X. Yu, and P. Perdikaris. When and why PINNs fail to train: A Neural Tangent Kernel perspective. arXiv preprint arXiv:2007.14527, 2020.arxiv.org/abs/2007.14527
  155. [155]S. Wang, H. Wang, and P. Perdikaris. On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 384:113938, 2021.
  156. [156]S. Wang, H. Wang, and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. arXiv preprint arXiv:2103.10974, 2021.arxiv.org/abs/2103.10974
  157. [157]S. Wang, H. Wang, and P. Perdikaris. Improved architectures and training algorithms for deep operator networks. Journal of Scientific Computing, 92(2):35, 2022.
  158. [158]G. Wen, C. Hay, and S. M. Benson. CCSNet: A deep learning modeling suite for CO₂ storage. Advances in Water Resources, 155:104009, 2021.
  159. [159]G. Wen, Z. Li, K. Azizzadenesheli, A. Anandakumar, and S. M. Benson. U-FNO—An enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180, 2022.
  160. [160]S. Wold, K. Esbensen, and P. Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37–52, 1987.
  161. [161]H. Xie, C. Zhai, L. Liu, and H. Yong. A weighted first-order formulation for solving anisotropic diffusion equations with deep neural networks. arXiv preprint arXiv:2205.06658, 2022.arxiv.org/abs/2205.06658
  162. [162]D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103–114, 2017.
  163. [163]S. Yeonjong, J. Darbon, and G. E. Karniadakis. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs. Communications in Computational Physics, 28(5):2042–2074, 2020.
  164. [164]H. You, Y. Yu, M. D’Elia, T. Gao, and S. Silling. Nonlocal kernel network (NKN): A stable and resolution-independent deep neural network. arXiv preprint arXiv:2201.02217, 2022.DOI
  165. [165]B. Yu and W. E. The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems. arXiv preprint arXiv:1710.00211, 2017.arxiv.org/abs/1710.00211
  166. [166]J. Yu, L. Lu, X. Meng, and G. E. Karniadakis. Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems. Computer Methods in Applied Mechanics and Engineering, 393:114823, 2022.
  167. [167]Y. Zang, G. Bao, X. Ye, and H. Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020.
  168. [168]D. Zhang, L. Lu, L. Guo, and G. E. Karniadakis. Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397:108850, 2019.
  169. [169]D.-X. Zhou. Universality of deep convolutional neural networks. Applied and computational harmonic analysis, 48(2):787–794, 2020.
  170. [170]Y. Zhu and N. Zabaras. Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415–447, 2018.

Paper details

vlnc:W_4xa5dbxz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents