betaFeedback

WorksPhysics and Astronomy

Examining the robustness of Physics-Informed Neural Networks to noise for Inverse Problems

Abstract

Approximating solutions to partial differential equations (PDEs) is fundamental for the modeling of dynamical systems in science and engineering. Physics-informed neural networks (PINNs) are a recent machine learning-based approach, for which many properties and limitations remain unknown. PINNs are widely accepted as less computationally efficient and accurate than traditional methods for solving PDEs, such as the finite element method. However, PINNs are commonly claimed to show promise in solving inverse problems and handling noisy or incomplete data. We compare the performance of PINNs in solving inverse problems with that of a traditional approach using the finite element method combined with a numerical optimizer. The models are tested on viscosity identification in 1D Burgers' equation and in 2D/3D Taylor-Green Vortex, in all cases with additive Gaussian noise applied to training and validation data. We find that while PINNs may require less human effort and specialized knowledge, they are outperformed by the traditional approach. For example, for 2D Taylor-Green Vortex with σσ=1 noise, the baseline has a mean prediction RMSE of 0.0013 compared to 0.01 for the best PINN variation. However, PINNs scale better than the baseline with the computational complexity of the problem. We identify failures during training to be addressed if the PINN performance on noisy inverse problems is to become more competitive.

Introduction

Partial differential equations (PDEs) are essential tools for modeling physical systems which are extensively used within engineering [1]. Solving a PDE is called the forward problem. The most widely used methods are backed by a deep theoretical understanding of their limitations and their capabilities on various problems [2]. In many real-life cases, the PDE may not be fully defined. For inverse problems [3], the goal is to identify something missing in the physical model such as a parameter or an initial condition.

Physics-informed neural networks (PINN) leverage the flexibility of neural networks while combining it with physics knowledge to solve both the forward and inverse problems of PDEs within a unified framework [4]. Note that PINNs were introduced to solve inverse problems with static parameters [4]. There has been some limited work to extend PINNs to function-valued inverse problems [5, 6], but we consider this outside of our current scope.

Theoretical understanding of the capabilities and limitations of PINNs remains limited compared to standard methods [7]. Despite extensive literature on the subject [8, 9, 10, 11, 12], PINNs are widely recognized to be inferior to traditional methods for solving the forward problem of PDEs. As an example, Chuang et al. [12] report that PINN takes 32 hours to achieve the same accuracy as a traditional solver running for less than 20 seconds. However, PINNs seem to be less vulnerable to the curse of dimensionality [13]. PINNs may also solve inverse problems many traditional solvers cannot, such as those without known boundary conditions [14, 15].

Despite lower accuracy in the forward problem, PINNs may solve the inverse problem satisfactorily [11, 8, 9], and they have been applied with reasonable success in various fields [16, 17, 18, 19]. CDFbench tests various physics machine learning models on predicting velocity fields of fluid mechanics systems [20] while PINNacle benchmarks variations of PINN on different types of physics problems, including some inverse problems [21]. However, as noted by Karnakov et al. [22], to the best of our knowledge there exists no benchmark to test how PINNs compare with traditional solvers on inverse problems, in particular inverse Navier-Stokes problems with noise.

Numerous efforts have been made to enhance PINN performance [23], such as by balancing the gradients of each training loss, domain decomposition, Fourier space embedding and learning earlier time steps of the physical system first [24, 25, 26, 27]. For the inverse problem, improvement has been demonstrated in parameter identification by using different optimizer for parameter identification and network parameters [28]. Using an ensemble of PINNs has also been suggested to achieve more accurate parameter identification [29]. ODIL is a proposed drop-in replacement for PINNs which directly optimizes discrete formulations of PDEs, removing the neural network [22].

Contributions and outline

We explore the claim that PINNs are effective for inverse problems. As our baseline, we use the general purpose method of combining the finite element method (FEM) with an optimizer. The most efficient solver-based approaches involve significant tailoring to the individual problem. Our purpose is to compare PINNs to a similarly easily accessible but reasonable choice of standard methods, without extensive tuning. We perform tests on parameter identification with varying levels of noise and data availability on a series of increasingly difficult fluid problems leading to the following contributions:

  • We show that while PINNs require less manual labor and specialized knowledge, a simple baseline with a FEM solver and SLSQP optimizer largely outperforms PINNs on noisy inverse problems.

  • We find that the relative performance of PINNs appears to improve with the computational complexity of the problem, consistent with existing literature [13].

  • We identify two particular problems PINNs have on noisy inverse problems:

    • The PINNs were consistently biased in their parameter estimation, eventually locking the model into a poor solution of the PDE.

    • With adaptive weights [24], needed for harder problems, the physics loss grew very large for noisy training data. In these cases, the physics loss does not have the desired effect of correcting the noise.

  • We find that identifying the best model during training is not trivial. New early stopping strategies are needed for noisy inverse problems.

The remainder of the paper is outlined as follows: In Section 2, we provide an overview of PINNs, the finite element method, and the mathematical formulation of the test problems. In Section 3, the shared setup of the experiments is described. Section 4 presents the experimental details and their results. The code can be found on GitHub1. § 5 discusses the results of the experiments and in § 6 we conclude.

Background

Physics-Informed Neural Networks

In a standard neural network, an input vector x\mathbf{x} is passed through layers a1(x;θ1),…,an(x^n−1;θn)a_1(\mathbf{x};\boldsymbol{\theta}_1), \ldots, a_n(\hat{\mathbf{x}}_{n-1};\boldsymbol{\theta}_n) of weighted sums and non-linear activation functions. aja_j, x^j−1\hat{\mathbf{x}}_{j-1}, θj\boldsymbol{\theta}_j, j=1…nj=1\ldots n are respectively the output, the input and the parameters (biases and weights) of layer jj. We may consider the full neural network as a parameterized single function [30]:

a(x;θ)=an(an−1(⋯a1(x;θ1))).(1)a(\mathbf{x};\boldsymbol{\theta}) = a_n\left(a_{n-1}\left(\cdots a_1(\mathbf{x};\boldsymbol{\theta}_1)\right)\right). \tag*{(1)}

a(x;θ)a(\mathbf{x};\boldsymbol{\theta}) can be both a scalar function and a vector function a(x;θ)a(\mathbf{x};\boldsymbol{\theta}). To evaluate the parameters, we define a loss function Ldata\mathcal{L}_{\mathrm{data}} which measures the discrepancy between the predicted output y^i\hat{\mathbf{y}}_i and the true value yi\mathbf{y}_i for observations available in the training set, totaling ndatan_{\mathrm{data}}. For regression type problems, a common choice is the Mean Squared Error (MSE):

Ldata=1ndata∑i=1ndata∥y^i−yi∥22.(2)\mathcal{L}_{\mathrm{data}} = \frac{1}{n_{\mathrm{data}}}\sum_{i=1}^{n_{\mathrm{data}}}\left\|\hat{\mathbf{y}}_i-\mathbf{y}_i\right\|_2^2. \tag*{(2)}

In physics-informed neural networks (PINNs) [4], in addition to the above loss function, we include a second loss function:

Lphys=1nphys∑i=1nphys∥Rθ(xi)∥22.(3)\mathcal{L}_{\mathrm{phys}} = \frac{1}{n_{\mathrm{phys}}}\sum_{i=1}^{n_{\mathrm{phys}}}\left\|\mathcal{R}_{\theta}(\mathbf{x}_i)\right\|_2^2. \tag*{(3)}

Rθ(x)\mathcal{R}_{\theta}(\mathbf{x}) is a function which calculates the residual of the differential equation we want to impose on the solution. Leveraging the automatic differentiation capabilities available in popular machine learning libraries, we can easily calculate the derivates of a(x;θ)a(\mathbf{x};\boldsymbol{\theta}) to check adherence with the differential equation. Missing parameters in the PDE can be optimized along with the parameters of the neural network. The two losses in Equation (2) and Equation (3) are summed to get the total loss minimized by the optimizer: Ltotal=wLdata+Lphys\mathcal{L}_{\mathrm{total}} = w\mathcal{L}_{\mathrm{data}}+\mathcal{L}_{\mathrm{phys}}, where ww is used to balance the weights.

In our experiments, we used a popular algorithm which adjusts ww to ensure that the magnitudes from the two losses are similar [24]. In early stopping [30], we exclude a part of the training set, called the validation set, and use it to approximate when the model has achieved maximum performance.

When training PINNs, the optimizer Adam [31] is often applied first to the loss function since it is considered effective for widely exploring search optimization landscapes. The quasi-Newtonian optimization method L-BFGS (Limited-Memory Broyden-Fletcher-Goldfarb-Shanno) is applied at the end to converge more thoroughly to the closest minimum [32]. To our knowledge, since neural network optimization landscapes are highly nonconvex, we cannot expect monotonous learning for L-BFGS with line search [33].

Finite Element Method

The Finite Element Method (FEM) is a general purpose method for solving differential equations [34]. The procedure works by projecting the solution onto simpler functions, called basis functions, resulting in a system of equations which, when solved, give the closest approximate fit to the solution. In some ways, it is similar to how PINNs estimate the solution by representing it as a neural network and minimizing the losses.

The differential equation in its standard form is known as the strong formulation. We project it by multiplying by a simpler function v(x,t)v(x,t) on both sides, which is referred to as a test function in the literature, and by integrating. This gives us the projected weak form of the problem:

∫Ω(∂u(x,t)∂t+N[u(x,t)])v(x,t) dx=0.(4)\int_{\Omega} \left( \frac{\partial u(x,t)}{\partial t} + N[u(x,t)] \right) v(x,t)\, dx = 0. \tag*{(4)}

Here N[u(x,t)]N[u(x,t)] is a linear differential operator which encodes the behavior of the system while u(x,t)u(x,t) is the solution we are looking for, defined over a domain Ω\Omega on some specific time interval [0,T][0,T] with respect to initial and boundary conditions. This description can be extended to any dimension, but we assume scalar variables here for simplicity.

There are many methods for how to proceed from Equation (4). One of the most well-established methods for deriving and solving the weak form is the Galerkin method [35], in which the same basis functions {ϕi(x)}i∈N\{\phi_i(x)\}_{i \in\mathbb{N}}, are used for both the test functions v(x,t)v(x,t) and trial functions u(x,t)u(x,t). We get a matrix equation with the following structure to solve:

Ma˙(t)+Aa(t)+Ba(t)=0(5)\mathbf{M}\dot{\mathbf{a}}(t) + \mathbf{A}\mathbf{a}(t) + \mathbf{B}\mathbf{a}(t) = 0 \tag*{(5)}

The matrices are defined by the exact problem and the specific approach taken. There are many variations of FEM which can be applied to solve problems both in space and time. The steps to derive the weak form depend on the equation to be solved and its boundary conditions. Custom algorithms are often necessary for acceptable performance on different problems [36].

Sequential Least Squares Programming

To estimate an unknown parameter in a differential equation using observed data, we first make an initial guess. With this guess, we solve the differential equation using the finite element method. The data loss, measuring the discrepancy between the solution and the training data, is used to update the guess. This process is repeated until convergence [3].

In the experiments, we use Sequential Least Squares Programming (SLSQP) [37], a more robust variation of Sequential Quadratic Programming (SQP). Both algorithms are efficient for non-linear programming problems. Our goal is to find the minimum of the function Ldata(x)\mathcal{L}_{\mathrm{data}}(x), which represents the data loss with regards to the unknown parameter xx. At the function minimum, the gradient ∇Ldata(x)\nabla\mathcal{L}_{\mathrm{data}}(x) will be zero. To seek this zero, the function (and any additional constraint functions) are linearized with Taylor expansions. In SQP, the problem is solved with Newton’s method which uses the gradient to find which direction to move and by how much. However, SLSQP uses a quasi-Newtonian method such as BFGS to avoid calculating the Hessian. Constraints are handled with a nonlinear least squares method.

Test problems

We will perform our experiments on the Burgers’ equation and the Navier-Stokes equations. The systems are related in the sense that Burgers’ equation is a simplification of the Navier-Stokes equations. All the systems used for our experiments are commonly used benchmark problems [4, 8, 20].

The actual implementations of each system used in the experiments, such as boundary conditions and parameters, will be described in Chapter 4.

Although we will specifically work on Burgers’ equation in 1D and Navier-Stokes in 2D and 3D, we will describe the systems in dd-dimensional form.

Burgers’ equation

Burgers’ equation is a non-linear partial differential equation. It models the evolution of a velocity field u(x,t)u(x,t) as a function of space and time:

∂u(x,t)∂t+u(x,t)⋅∇u(x,t)=ν∇2u(x,t),x∈Ω⊂Rn,t∈[0,∞),u∈Rn.(6)\frac{\partial\mathbf{u}(\mathbf{x},t)}{\partial t}+\mathbf{u}(\mathbf{x},t)\cdot\nabla\mathbf{u}(\mathbf{x},t)=\nu\nabla^{2}\mathbf{u}(\mathbf{x},t), \qquad\mathbf{x}\in\Omega\subset\mathbb{R}^{n},\quad t\in[0,\infty),\quad\mathbf{u}\in\mathbb{R}^{n}. \tag*{(6)}

Here ∂u(x,t)∂t\frac{\partial\mathbf{u}(\mathbf{x},t)}{\partial t} describes how the velocity field changes in time. u(x,t)⋅∇u(x,t)\mathbf{u}(\mathbf{x},t)\cdot\nabla\mathbf{u}(\mathbf{x},t) is the non-linear advection term which determines how the velocity field changes as a function of its current velocity and its position. ν∇2u(x,t)\nu\nabla^{2}\mathbf{u}(\mathbf{x},t) is the diffusion term which describes how the velocity is evened out in the flow. Depending on the initial conditions and viscosity, Burgers’ equation can develop shocks when discontinuities arise in the solution as ∂u(x,t)∂x→−∞\frac{\partial\mathbf{u}(\mathbf{x},t)}{\partial x}\to-\infty. Intuitively, shocks can be thought of as hard corners forming in the wave. Shocks pose a challenge for common approaches to solving differential equations since standard methods assume differentiable and smooth functions.

Incompressible Navier-Stokes equations

Like Burgers’ equation, the Navier-Stokes equations are a set of non-linear PDEs which model the evolution of a velocity field u(x,t)\mathbf{u}(\mathbf{x},t) as a function of time and space. The incompressible form is used to describe liquids whose density does not change with pressure and temperature. The momentum equation in the incompressible Navier-Stokes equations has the general form:

∂u(x,t)∂t+u(x,t)⋅∇u(x,t)=−1ρ∇p(x,t)+ν∇2u(x,t),x∈Ω⊂Rn,t∈[0,∞),u∈Rn,(7)\frac{\partial\mathbf{u}(\mathbf{x},t)}{\partial t}+\mathbf{u}(\mathbf{x},t)\cdot\nabla\mathbf{u}(\mathbf{x},t)=-\frac{1}{\rho}\nabla p(\mathbf{x},t)+\nu\nabla^{2}\mathbf{u}(\mathbf{x},t), \qquad\mathbf{x}\in\Omega\subset\mathbb{R}^{n},\quad t\in[0,\infty),\quad\mathbf{u}\in\mathbb{R}^{n}, \tag*{(7)}

with the requirement of continuity:

∇⋅u(x,t)=0.(8)\nabla\cdot\mathbf{u}(\mathbf{x},t)=0. \tag*{(8)}

As in Burgers’ equation, ∂u(x,t)∂t\frac{\partial\mathbf{u}(\mathbf{x},t)}{\partial t}, u(x,t)⋅∇u(x,t)\mathbf{u}(\mathbf{x},t)\cdot\nabla\mathbf{u}(\mathbf{x},t) and ν∇2u(x,t)\nu\nabla^{2}\mathbf{u}(\mathbf{x},t) describe the change in time, advection and diffusion, respectively. −1ρ∇p(x,t)-\frac{1}{\rho}\nabla p(\mathbf{x},t) is the pressure term which evens out differences in pressure. ∇⋅u(x,t)=0\nabla\cdot\mathbf{u}(\mathbf{x},t)=0 simply states the incompressibility in mathematical terms: The mass of the fluid in any volume does not change. Burgers’ equation can be seen as Navier-Stokes simplified, neglecting pressure and with no compressibility constraint.

Under some conditions, the Navier-Stokes equations exhibit turbulence. In turbulent flow, many small whirls form which interact in complex ways on small scales. We get a chaotic system where small changes in the initial conditions may lead to very large changes in the solution. The Reynolds number is a heuristic used to capture the turbulence of a system. Standard PINNs are not expected to work well with chaos [27]. For this reason, we only consider Navier-Stokes systems with low Reynolds numbers (<2000< 2000). However, for Taylor-Green Vortex this is sufficient for some turbulence.

General experimental setup

The most efficient traditional solver-based approaches require significant tailoring to each individual problem. Our goal is to compare PINNs to a traditional general-purpose method which is feasible to run on limited hardware using existing libraries. We want to consider how reasonable implementations would practically measure up against each other in real life application. This is not a controlled scaling study, which is not common in machine learning where the effects of different parameters may be highly irregular. For example, a larger neural network may not give higher accuracy. The results must be understood in light of our methodological limitations. We chose to follow a paper which specifically considered how to compare standard methods to machine learning [8]. To calibrate the models, we matched the models on accuracy when tested on noiseless data as well as we could.

As baseline, we employed a traditional Finite Element Method (FEM) solver combined with an SLSQP optimizer. For the optimizer, we used the SLSQP implementation from SciPy [38]. For the 1D Burgers’ equation, we used the optional FEniCS library pyadjoint [39] to differentiate the solver and calculate gradients for the optimizer. However, this approach required the data and solver grid structure to be identical, which was unfeasible for the Navier-Stokes test problems because of the more complex grids required to solve these problems. In these cases, we employed finite differences with a three-point stencil from SciPy. This produced more accurate gradients than alternative methods discussed in the FEniCS community [40].

The PINN models were implemented using PyTorch [41]. Note that for PINN, we provide results for two types of models. One is a pure PINN model (marked "PINN" in the plots in Figures 1, Figure 3, Figure 5) where we get predictions with a forward pass on the trained neural network. In the "PINN/FEM" model, we collect the estimated parameter from the trained model and solve the differential equation using the same FEM solver as for the baseline. As is standard in machine learning, a small amount of the training data was separated and used as validation data in the PINN experiments. To get prediction RMSEs, we only compared predicted velocities and not the pressure.

Image not extracted

Figure 3. Results for 2D Taylor-Green vortex on test set based on 30 runs of each model. Standard deviation (σ\sigma) is the noise added to the training data. Top left: Prediction accuracies, σ∈[0,0.05]\sigma\in[0,0.05]. Top right: Prediction accuracies, σ∈[0.1,1]\sigma\in[0.1,1]. Bottom left: Parameter accuracies, σ∈[0,0.05]\sigma\in[0,0.05]. Bottom right: Parameter accuracies, σ∈[0.1,1]\sigma\in[0.1,1].

Results for 3D Taylor-Green vortex on test set based on 30 runs of each model

Figure 5. Results for 3D Taylor-Green vortex on test set based on 30 runs of each model. For visibility, the x-axes are displayed in a symmetric logarithmic scale with a linear threshold of 0.01. Standard deviation (σ\sigma) is the noise added to the training data. Top left: Prediction accuracies with outliers removed. Top right: Parameter accuracies with outliers removed. Bottom left: Prediction accuracies with outliers included. Bottom right: Parameter accuracies with outliers included.

Note that we report the training losses in MSE, the quantity optimized during training. However, we report the predictions in RMSE. This is because the small variation in MSE makes the prediction graphs hard to read. For completeness, the same graphs with MSE are included in Appendix § 13.

The experiments were conducted on the Idun high-performance computing cluster at the Norwegian University of Science and Technology (NTNU) [42]. The compute nodes are mainly equipped with Intel Xeon processors and NVIDIA GPUs. For most of the PINN experiments, we used NVIDIA A100 GPUs. Training PINNs on Navier-Stokes problems may require days depending on the GPU and problem instance, making hyperparameter optimization computationally impractical [12]. For this reason, hyperparameters are commonly selected by manual tuning, and for Navier-Stokes, we used hyperparameter settings consistent with related literature. For the 1D Burgers’ equation, we performed extensive hyperparameter tuning using Optuna [43]. We also investigated the effect of individual hyperparameters. We observed a high degree of robustness with respect to hyperparameter selection. The final hyperparameters were chosen to favor performance on the inverse problem.

For 1D Burgers’ equation, we used the same data as in the original PINN paper2 [4]. For 1D Burgers’ equation with two parameters, we used an analytical solution based on the Cole-Hopf transform, which is described in additional detail in Appendix § 12. For 2D Navier-Stokes, we used an analytical solution. For the specific 3D Navier-Stokes problem that we tested, neither an analytical solution nor pre-existing ground truth data were available. For this reason, we generated reference data using the spectral/hp element framework Nektar++ [44]. We used a scheme similar to the quasi-3D scheme in the official Nektar++ 3D Taylor-Green Vortex tutorial [45].

The noise added to the data was Gaussian with zero mean, using the library NumPy [46]. In all the experiments, we only added noise to the labels in the training data, but never to the initial condition and boundary condition data. Noise was also added to the validation set labels. This is a standard choice in machine learning, since if you were to have data without noise available, it would be much more valuable as training data. By σ\sigma, we denote the standard deviation of the noise. By SNR, we denote the variance of the training data without noise divided by the variance of the noise.

Due to the Central Limit Theorem [47], Gaussian noise can be used to approximate sensor noise. However, many of the noise levels we tested, in particular for the 1D Burgers’ equation, are extreme. For 1D Burgers’ equation, all the values stay within [−1,1][-1,1]. In other words, with σ=1\sigma= 1 the magnitude of the noise should at least be as large as the original signal. With σ=25\sigma= 25, noise dominates the original signal by a large amount. In real life, noise on scales such as σ=0.1\sigma= 0.1 are more likely. However, we did not want to make any assumptions about the specific application of PINNs. Rather, we wanted to explore the practical properties and limits of PINNs in a general way.

For 1D Burgers’ equation, we chose not to use early stopping based on empirical experience. For the Navier-Stokes problems, we used the validation set to determine which model to use. However, for this to work we had to exclude early models from considerations. Especially with noise, PINNs may do well on the validation set before they have started learning the physics meaningfully. This means the optimizer does not have time to adjust the unknown physics parameter in the physics loss and the inverse problem is not solved. Additionally, both PINN and baseline model were bounded to output minimum ν=π/1000\nu= \pi/1000. This is because the FEM solver is unable to handle large Reynolds numbers and may break with low viscosity values.

Experiments

1D Burgers’ equation

The ground truth viscosity term to solve for was set to ν=0.01/π\nu= 0.01/\pi following the original PINN paper [4]. Likewise, t∈[0,0.99]t \in[0, 0.99] and x∈[−1,1]x \in[-1, 1]. The following initial conditions and Dirichlet boundary conditions were used:

u(x,0)=−sin⁡(xπ),u(−1,t)=u(1,t)=0.(9)u(x, 0) = -\sin(x\pi), \qquad u(-1,t) = u(1,t) = 0 . \tag*{(9)}

Under these conditions, the system develops shock-like formations around t=0.5t = 0.5, which are known to be difficult to solve [4]. Analytical solutions are possible for ν>0\nu> 0 using the Cole-Hopf transformation [1]. We also tested 1D Burgers’ equation with two unknown parameters, adding a second physical parameter aa to the equation, described more extensively in Appendix § 12.

Specific experimental setup

For the training and test data for Burgers’ equation, the same data was used as in the original PINN paper [4]. The total dataset consisted of 25600 observations of which 10% was used as training data and 2% as validation data. The training and validation data were randomly sampled from the full dataset. For each model, we trained 30 random initializations for which we present the mean and standard deviation of the performance. The noise levels tested for the inverse problem, both in σ\sigma and SNR, are given in Appendix F.§ 1. Remaining implementation details are in Appendix § 7.§ 1.

Results, 1D Burgers’ equation

Results for 1D Burgers’ equation

Figure 1. Results for 1D Burgers’ equation on test set based on 30 runs of each model. Standard deviation (σ\sigma) is the noise added to the training data. Top left: Prediction accuracies, σ∈[0,2]\sigma\in[0, 2]. Top right: Prediction accuracies, σ∈[3,25]\sigma\in[3, 25]. Bottom left: Parameter predictions, σ∈[0,3]\sigma\in[0, 3]. Bottom right: Parameter predictions, σ∈[5,10]\sigma\in[5, 10].

Figure 1 shows the results of the experiments. By inspecting the tables in Appendix § 9.§ 1, we see that for σ=0\sigma= 0, the PINN model has the lowest RMSE accuracy of the three models tested. However, the PINN model takes 110 seconds on average to run using GPUs while our not optimized FEM/SLSQP model takes 25 seconds on a single core CPU. The peak memory use of both models are similar, around 500 MB. As the noise increases, the variation in prediction accuracy for both models begins to increase as well. In particular, PINN tends to predict very high parameter values at higher frequency when the noise increases. As can be seen in Appendix C.§ 1, for PINN the skewness of the prediction distribution increase in a right-leaning direction. Without noise, all models perform well with RMSE of the order of 10−310^{-3}.

Despite many of our tests being on extreme levels of noise, we observe that FEM/SLSQP significantly outperforms the classical PINN at all noise levels. However, the results of FEM/SLSQP and the PINN/FEM model cannot be distinguished with statistical significance, aside from at σ=0.05\sigma= 0.05 where FEM/SLSQP is highly likely to have lower error than PINN/FEM.

PINN exhibits a small negative bias for all noise levels, including σ=0\sigma= 0, as seen in Appendix C.§ 1. However, the bias is only statistically significant for noise levels 0, 3, 5, 7, 25. FEM/SLSQP has a positive bias, but these results are only significant for the highest noise levels. With σ=25\sigma= 25, which can be seen in Appendix § 15, PINNs have the highest mean estimates. (However, PINN also has a very large variation.)

Figure 2 shows the loss history during training. The observed history for σ=0\sigma= 0 is typical for the 1D Burgers’ equation tests without noise. While running the Adam optimizer, we see characteristic spikes in both the data loss and physics loss as the model tries to balance them. When L-BFGS is turned on (after slightly less than 1 minute), we get a mostly smooth convergence towards a local minimum as would be expected from this optimizer.

Four-panel plots showing 1D Burgers' equation loss history and parameter history

Figure 2. 1D Burgers’ equation loss history from training and parameter history from training. Note that the validation data, like the training data, has noise added to it which can explain the similar performance of their losses on high noise levels. The test loss only evaluates predictive accuracy, not agreement with differential equation. Top left: Loss history during PINN training, σ=0\sigma= 0. Top right: Loss history for σ=1\sigma= 1. Bottom left: Loss history for σ=7\sigma= 7. Bottom right: Parameter learning history for σ=1\sigma= 1. All the results of the losses exclude the weights used for loss balancing.

With both σ=1\sigma= 1 and σ=7\sigma= 7, the data loss is largely monotonic. The physics loss develops in a more complicated way with more erratic spikes. For σ=7\sigma= 7, the physics loss makes a jump around 0.9 minutes. After the test loss has reached its lowest point, the model ends up converging to a worse state. However, there is no way to tell from the validation loss when this lowest point happened, which supports our choice to not use early stopping for this problem. The remaining loss histories from our experiments not included here show similar behavior in the presence of noise.

The minimum test loss approximately occurs when the parameter estimates are closest to the ground truth. After, PINN tends to estimate a lower viscosity on average and converge on a worse total loss. This is also supported by the previously mentioned statistical tests in Appendix C.§ 1, which show that PINN tends to have a slight negative bias. This behavior is prominent in the example loss history with σ=1\sigma= 1 and σ=7\sigma= 7. For σ=0\sigma= 0, this can only be weakly seen. However, as more noise is added to the training data, the model becomes increasingly likely to estimate the viscosity too high.

As described in Appendix § 12, we also tested 1D Burgers’ equation with two unknown parameters. As we can see in Appendix § 9.§ 2, the models had similar performance as with one unknown parameter with slightly worse overall performance. Some of the issues that we shall later see in the other experiments with more complex systems are more pronounced in the problem with two parameters, such as large physics losses. Solving for two parameters produces effects similar to increasing system complexity in other experiments.

2D Taylor-Green Vortex

For the 2D Navier-Stokes experiment, the Taylor-Green vortex problem was selected. This problem is defined by the initial conditions:

u(x,y,0)=sin⁡xcos⁡y,v(x,y,0)=−cos⁡xsin⁡y,p(x,y,0)=ρ4(cos⁡2x+cos⁡2y).(10)\begin{aligned} u(x,y,0) &= \sin x \cos y,\\ v(x,y,0) &= -\cos x \sin y,\\ p(x,y,0) &= \frac{\rho}{4}\left(\cos2x+\cos2y\right). \tag*{(10)} \end{aligned}

The ground truth viscosity term to be solved for was set to ν=0.1\nu= 0.1 while ρ=1\rho= 1 is known. The variables were restricted to t∈[0,2.5]t \in[0,2.5], x∈[0,2π]x \in[0,2\pi] and y∈[0,2π]y \in[0,2\pi]. This problem roughly corresponds to Re=100Re = 100 which is well within laminar flow. We assume q2D(x,y,t)\mathbf{q}_{2D}(x,y,t) is a state vector containing velocities uu, vv and the pressure pp at any given position and time in the 2D domain. The following periodic boundary conditions were used:

q2D(2π,y,t)=q2D(0,y,t),q2D(x,2π,t)=q2D(x,0,t).(11)\begin{aligned} \mathbf{q}_{2D}(2\pi,y,t) &= \mathbf{q}_{2D}(0,y,t),\\ \mathbf{q}_{2D}(x,2\pi,t) &= \mathbf{q}_{2D}(x,0,t). \tag*{(11)} \end{aligned}

The analytical solution for this system is as follows:

u(x,y,t)=e−2νtsin⁡xcos⁡y,v(x,y,t)=−e−2νtcos⁡xsin⁡y,p(x,y,t)=e−4νtρ4(cos⁡2x+cos⁡2y).(12)\begin{aligned} u(x,y,t) &= e^{-2\nu t}\sin x\cos y,\\ v(x,y,t) &= -e^{-2\nu t}\cos x\sin y,\\ p(x,y,t) &= e^{-4\nu t}\frac{\rho}{4}\left(\cos2x+\cos2y\right). \tag*{(12)} \end{aligned}

Intuitively, the system can be described as a set of four evenly spaced whirls with velocity dissipating according to the value of viscosity.

Specific experimental setup

Following [4], we trained the PINN model a(x;θ)\mathbf{a}(x;\theta) to learn the stream function, ψ(x,y,t;θ)\psi(x,y,t;\theta), along with the pressure field p(x,y,t;θ)p(x,y,t;\theta):

a(x;θ)=[ψ(x,y,t;θ)p(x,y,t;θ)].(13)\mathbf{a}(x;\theta) = \begin{bmatrix} \psi(x,y,t;\theta)\\ p(x,y,t;\theta) \end{bmatrix}. \tag*{(13)}

Instead of having to learn the velocity for each dimension xx and yy, the network only needs to learn a single stream output for all dimensions. This is achieved using the identities ∂ψ(x,y,t)∂y=u(x,y,t)\frac{\partial\psi(x,y,t)}{\partial y}=u(x,y,t) and ∂ψ(x,y,t)∂x=−v(x,y,t)\frac{\partial\psi(x,y,t)}{\partial x}=-v(x,y,t). The ψ(x)\psi(x) formulation assumes incompressibility, so we only need to impose the momentum equation to get the correct physics with this approach.

The total dataset consisted of 412 776 observations uniformly sampled in space and time from the analytical solution. We used 24% for training, 4.8% for validation and imitated hyperparameters used for successful training on 2D Navier-Stokes in the literature [4]. To consistently get acceptable performance, we had to use a gradient balancing algorithm on the weights of the losses [24]. Our implementation is standard, but more information can be found in Appendix § 7.§ 4. We trained 30 random initializations and present the mean and standard deviations of the results. The noise levels tested for the inverse problem, both in σ\sigma and SNR, are given in Appendix F.§ 3. Remaining exact implementation details can be found in Appendix § 7.§ 2.

Results, 2D Taylor-Green Vortex

Figure 3 shows that FEM/SLSQP on average outperforms the other models for 2D Taylor-Green Vortex on all tested noise levels. Where the distributions overlap, we have confirmed that FEM/SLSQP is superior with statistical significance in Appendix § 9.§ 3. On the other hand, PINN takes 6.6 hours on average while FEM takes 11.2 hours on average. PINN also had lower peak memory use than the baseline. We used 10 cores running in parallel for FEM/SLSQP and each individual core required 966 MBs at most. For PINN we only used one core, with a peak memory use of around 1088 MB.

We can see that the parameter estimation for 2D Taylor-Green Vortex has larger variation than for 1D Burgers’ equation on the noise levels for which we did experiments on both systems. As with RMSE predictions, FEM/SLSQP outperforms PINN at predicting the correct unknown parameter. As opposed to 1D Burgers’ equation, the skewness of the estimated parameter distribution leans left rather than right for PINN, and does not grow as evenly with the noise.

As with 1D Burgers’ equation, PINN tends to have a small negative bias on parameter estimation for 2D Taylor-Green Vortex. The bias is statistically significant from noise level σ=0.05\sigma= 0.05 to σ=1\sigma= 1. In contrast with the results for 1D Burgers’ equation, the mean of the predicted parameter continues to decrease with noise rather than to eventually blow up. FEM/SLSQP has a small positive bias on lower noise levels. Interestingly, PINN has the best performance at σ=0.01\sigma= 0.01. However, the advantage is slight and, for this noise level, PINN also has a larger range in accuracy than for lower noise levels.

In Figure 4, the loss histories fluctuate even more than they did for 1D Burgers’ equation. For visibility at σ=0\sigma= 0, we have applied weighted average exponential smoothing with α=0.001\alpha= 0.001 [48]. On lower noise levels, the loss history resembles the behavior of 1D Burgers’ equation: After switching to L-BFGS, we often see a smooth convergence towards a better model. At lower noise levels, PINNs rapidly converge near ground truth but show little subsequent improvement.

Image not extracted

Figure 4. 2D Taylor-Green vortex training history. Note that noise is added to the validation data which may explain poor validation loss performance. The test loss only evaluates predictive accuracy, not agreement with differential equation. Top left: Loss history during PINN training, σ=0\sigma= 0. Top right: Loss history for σ=1\sigma= 1. Bottom left: Parameter learning history, σ=0\sigma= 0. Bottom right: Parameter learning history, σ=1\sigma= 1. All the results of the losses exclude the weights used for loss balancing.

As noise increases, a different phenomenon becomes increasingly prevalent. While the PINN model first tends to balance physics and data training well, at some point it starts to disregard the physics loss. We can see for σ=1\sigma= 1 in Figure 4 that the physics loss, which is the mean square average over the data, approaches 10110^{1}. In the parameter estimation histories for higher numbered σ\sigma, it often stays near the ground truth for a while before training starts to fail. The best models are always from before this transition.

3D Taylor-Green Vortex

For the implementation of 3D Navier-Stokes, the Taylor-Green vortex problem was used. In 3D, it is defined by the initial conditions:

u(x,y,z,0)=sin⁡xcos⁡ycos⁡z,v(x,y,z,0)=−cos⁡xsin⁡ycos⁡z,w(x,y,z,0)=0,p(x,y,z,0)=0.(14)\begin{aligned} u(x,y,z,0) &= \sin x \cos y \cos z, \\ v(x,y,z,0) &= -\cos x \sin y \cos z, \\ w(x,y,z,0) &= 0, \\ p(x,y,z,0) &= 0. \tag*{(14)} \end{aligned}

The ground truth viscosity term was chosen to be ν=0.01\nu= 0.01. This roughly corresponds to the Reynolds number 1000, which for the Taylor-Green Vortex is starting to become turbulent. We wanted to increase the complexity of the problem as well as add a dimension to get more variety in the test systems. The variables were restricted to t∈[0,2.5]t \in[0, 2.5], x∈[−π,π]x \in[-\pi, \pi], y∈[−π,π]y \in[-\pi, \pi] and z∈[0,2π]z \in[0, 2\pi]. We assume q3D(x,y,z,t)\mathbf{q}_{3D}(x, y, z, t) is a state vector containing velocities uu, vv, ww and the pressure pp at any given position and time in the 3D domain. The following periodic boundary conditions were used:

q3D(−π,y,z,t)=q3D(π,y,z,t),q3D(x,−π,z,t)=q3D(x,π,z,t),q3D(x,y,0,t)=q3D(x,y,2π,t).(15)\begin{aligned} \mathbf{q}_{3D}(-\pi, y, z, t) &= \mathbf{q}_{3D}(\pi, y, z, t), \\ \mathbf{q}_{3D}(x, -\pi, z, t) &= \mathbf{q}_{3D}(x, \pi, z, t), \\ \mathbf{q}_{3D}(x, y, 0, t) &= \mathbf{q}_{3D}(x, y, 2\pi, t). \tag*{(15)} \end{aligned}

Intuitively, 3D Navier-Stokes consists of layers of the 2D version of the problem where the direction of the flow is opposite on regular intervals depending on the zz-coordinate. Due to small turbulence, velocity starts to build and creates some velocity in the zz direction making the layers of whirls interact with each other chaotically. There is no analytical solution for the chosen parameters.

Specific experimental setup

3D Navier-Stokes cannot be reduced to a single stream function, and alternative formulations that decrease the number of required functions are non-trivial. For this reason, we trained the PINN model to learn the individual velocities as outputs. The total dataset consisted of 1 100 000 observations randomly sampled from the data we generated using Nektar++ as described in § 3. We used 25% as training data and 5% as validation data which is similar as what we used for the 2D problem. The proportion of training and validation data was selected through manual experimentation to find what was required to reliably solve the problem with PINN at σ=0\sigma= 0. Here as well, we used a gradient balancing algorithm for the loss weights [24]. 30 random initializations were trained of each model and we present the mean and standard deviations of the results. The noise levels tested for the inverse problem, both in σ\sigma and SNR, are given in Appendix F.§ 4. Other implementation details can be found in Appendix § 7.§ 3.

Results, 3D Taylor-Green Vortex

For the 3D experiments, both PINN and FEM/SLSQP sometimes catastrophically fail. Both PINN and FEM/SLSQP occasionally get stuck on or near the initial guess. Note that these failures, in a practical setting, could easily be identified and excluded. We define outliers using interquartile range (IQR) [47]. Values below Q1−1.5⋅IQRQ_1 - 1.5 \cdot IQR and values above Q3+1.5⋅IQRQ_3 + 1.5 \cdot IQR were identified as outliers. For FEM/SLSQP these failures account for about 20% of the runs. For PINN, it is less than 1%. However, in more than 25% of runs, PINN is stuck on the lowest parameter allowed. This is not captured by our outlier definition and relative to the task it is a decent estimation, so we have chosen not to consider these results outliers.

No negative bias can be proven for PINN while FEM has a small positive bias on several noise levels. When we include the outlier runs, the standard deviations of FEM are also very large. For instance, it is 0.38 for σ=0.05\sigma= 0.05. According to our statistical tests in Appendix § 9.§ 4, FEM still outperforms both PINN and PINN/FEM with statistical significance. When we exclude the outliers for FEM, the same effect becomes larger. With outliers excluded, FEM/SLSQP’s parameter identification accuracy also becomes similar as in 2D. However, it must be noted that the baseline’s failure rate is very high. Even though FEM is very likely to outperform PINN in most cases, it is not straightforward to claim that the baseline is better than PINN.

For 3D Taylor-Green Vortex, FEM/SLSQP is a lot more expensive to run than PINN. On average, FEM/SLSQP takes 19.9 hours to solve the inverse problem. PINN takes 10.8 hours on average. The memory costs of FEM/SLSQP are also far greater. PINN’s peak memory use is similar to as for 2D Taylor-Green Vortex, about 1100 MB. FEM/SLSQP has an average peak memory of almost 2000 MB for each of the 55 cores we used. The baseline does not scale well and requires HPC resources for 3D.

Even with outliers removed, all the models generally perform worse than on the 2D problem. This result is expected since by adding a dimension with equal size as the others, the domain becomes much larger. The lower ground truth viscosity problem also increases the Reynolds number, making the system harder, potentially leading to small amounts of chaos as well.

Comparing the loss and parameter estimation histories in Figure 6, we observe the same phenomena as in 2D Taylor-Green Vortex. In the most successful runs, we may get a smooth improvement when L-BFGS is applied in the end. In the less successful runs, the physics loss tends to blow up. In the 2D case, these phenomena seem tied to the level of noise with the physics loss exploding becoming increasingly likely with higher levels of noise. For 3D, the phenomena seem to be independent of the noise.

3D Taylor-Green vortex training histories

Figure 6. 3D Taylor-Green vortex training histories. Note that the validation data may contain noise which can explain poor validation loss performance. The test loss only evaluates predictive accuracy, not agreement with differential equation. Left: Loss history for PINN, σ=1\sigma= 1. Weighted average exponential smoothing with α=0.001\alpha= 0.001 was used [48]. Right: Parameter learning history, σ=1\sigma= 1. All the results of the losses exclude the weights used for loss balancing.

Discussion

1D Burgers’ equation

The results from the 1D Burgers’ equation experiments raise an interesting question. As seen in Figure 2, for σ=1\sigma= 1, the PINN test loss exhibits typical overfitting behavior after around 1.25 minutes. Since we have much of the differential equation and noisy data, we could naively expect that minimizing both losses, would always approximate the solution better. The physics equation should prohibit the model from overfitting on the noise.

One contribution to this failure may be the tendency of PINNs to estimate the viscosity as lower than the ground truth. As the physics loss gets sufficiently small, PINNs may eventually decrease the loss significantly by making the unknown viscosity smaller. When the differential equation thus becomes incorrect, it is less compatible with the data. The model may not be able to improve anymore, stuck in a local minima, and it will converge on a worse solution than earlier in training. This is similar to the "reward hacking" problem common in reinforcement learning where the model finds a way to optimize its loss which does not serve the purpose of training [49]. As seen in Appendix C.§ 1, PINN often has a negative bias.

While test loss identifies the optimal model, using only training and validation losses is insufficient. Developing a method to detect optimal stopping points from these losses alone would enable early stopping tailored to PINNs in noisy inverse problems. Since the validation loss only tells us how the model is doing on the forward problem, it is not always indicative of the inverse problem. In many cases already, understanding and accommodating the difference of PINNs from ordinary machine learning has helped develop the competitiveness of PINNs in other aspects [24, 50].

Even if the FEM/SLSQP baseline is not significantly more accurate than PINN/FEM on the 1D Burgers’ equation, it is certainly faster. For some applications this difference may not make a practical difference. At the same time FEM/SLSQP is considerably more difficult to set up and requires more specialized knowledge. We need a weak form of our PDE, the grid must be specified and the matrix solver chosen. In many real-life settings, PINNs may be more appropriate given the existence of user-friendly ML packages.

2D Taylor-Green Vortex

We noted that for this problem PINN was faster than the FEM/SLSQP baseline. As discussed in § 3, a careful comparison of computational cost is outside of our scope. A significant part of the cost of FEM/SLSQP consists in calculating the points of the training set to compare against the solution. The large amount of training data may be biased towards PINN in that FEM/SLSQP may not need as much training data as a neural network to achieve equal results. This is a typical challenge in comparing traditional methods with machine learning methods: Both training on different data sets and training on the same data set may be considered unfair.

Shorter training times may also be achievable for PINN. With σ=0\sigma= 0, the model finds parameter values close to the ground truth in less than an hour, if we properly know when to stop training. With the applied gradient weighting algorithm, PINN may require less epochs than presented in the literature. This supports our claim in § 5.1 that we may be able to improve PINNs if we develop appropriate early stopping strategies.

For 2D Navier-Stokes, PINN has the same tendency to estimate the viscosity as lower than the ground truth as we observed with 1D Burgers’ equation. An example of this can be seen in the loss histories in Figure 4. This bias is also confirmed by statistical tests found in Appendix § 9.§ 3. However, with higher noise a different behavior begins to dominate. Increasingly with noise, PINNs will at some point during training disregard the physics loss and let it grow by a few magnitudes. For higher noise levels, such as σ=7\sigma= 7 for Burger’s equation in § 2, we can already see some of this tendency although less prominent. We also see this more frequently on lower noise levels in the 1D Burgers’ equation tests with two unknown parameters. However, we hypothesize this phenomena is exacerbated by the adaptive weight algorithm we used for the Navier-Stokes problems [24]. The weight on the data loss, calculated by dividing the maximum gradient of the physics loss by the average gradient of the data loss, may make it more profitable to overfit on the noise than to consider the physics. As the physics become incompatible with the model of the noisy data, the optimizer will eventually disregard the physics loss. Contrary to expectations, the physics loss does not effectively constrain the adverse effects of noise in PINNs, in particular when the adaptive weight gradient balancing algorithm is used.

3D Taylor-Green Vortex

Interestingly, many of the problems which we only saw for higher noise levels with 2D Taylor-Green Vortex are already present without noise with 3D Taylor-Green Vortex. A possible explanation is that increasing the difficulty of the problem has a similar effect as adding noise to the training data. This is also supported by the 1D Burgers’ equations experiments with two parameters instead of only one, where we see more of the problems that appear with 2D and 3D Taylor-Green Vortex. We may hypothesize that overfitting on a simplified model of a difficult physical system can become more profitable than learning the physics, just as when overfitting on noise.

PINN does not show a consistent negative bias for 3D Taylor-Green Vortex as it did with 1D Burgers’ equation and 2D Taylor-Green Vortex. The mean biases go both directions, but are only statistically significant at σ=1\sigma= 1 (where the bias is negative). A possible explanation is that learning issues due to the higher complexity in 3D occlude this simpler learning problem.

We observe that PINN seems to scale better with the computational complexity of the problem compared to FEM/SLSQP. The cost for 3D Taylor-Green Vortex is not much higher than it was for the 2D version of the problem, in particular the memory use. The baseline, FEM/SLSQP, scales much worse. Running it on the 3D problem is significantly more expensive and requires HPC resources. FEM/SLSQP is also highly likely to catastrophically fail compared to PINN. On the other hand, as with FEM, PINNs accuracy degrades with the 3D problem compared to the 2D problem. Our observation that PINNs may scale well supports previous claims in the literature [13].

Conclusion

Through three different experiments, we have observed that a simple baseline using FEM and an optimizer, mostly outperforms PINNs in accuracy on noisy inverse problems. However, this observation is complicated by the baseline’s memory costs for Navier-Stokes and its frequent catastrophic failure rate (about 20%) for the 3D test problem. The limits of our experiments also make discussion about time cost difficult. To the best of our knowledge, this is the first benchmark comparing PINNs with traditional solvers on inverse problems, in particular inverse Navier-Stokes problems with noise. However, PINNs do require less manual labor and specialized knowledge compared to the FEM-based baseline. We have also found that relative to the baseline, PINNs appear to scale better with the computational complexity of the problem as memory costs change little. This is consistent with findings in existing literature where PINNs excel in extremely high dimensions [13].

We have identified two difficulties which appear to limit the performance of PINNs on noisy inverse problems. There is a "reward hacking" phenomenon, the parameter learning is biased by the optimization process. This causes training to converge to a suboptimal solution. Adaptive weighting also tends to break down with noisy training data. Contrary to our expectations, the physics loss fails to constrain the adverse effects of noise.

Furthermore, we have found that it is not trivial to identify the best model during training. Neither the validation loss nor physics loss tends to give a good indication according to the test loss. Developing new early stopping strategies for PINNs may be crucial to make the models more competitive on noisy inverse problems. The future of PINN research appears to lie in better tailoring neural networks to the physics domain in particular [23].

Implementation details

1D Burgers' equation

In the FEM model, we used the following weak formulation:

∫Ω∂u(x,t)∂tv(x,t) dx+∫Ωu(x,t)∂u(x,t)∂xv(x,t) dx+ν∫Ω∂u(x,t)∂x∂v(x,t)∂x dx=0.(16)\begin{aligned} \int_{\Omega} \frac{\partial u(x,t)}{\partial t}v(x,t)\,dx + \int_{\Omega} u(x,t)\frac{\partial u(x,t)}{\partial x}v(x,t)\,dx \\ + \nu\int_{\Omega} \frac{\partial u(x,t)}{\partial x}\frac{\partial v(x,t)}{\partial x}\,dx = 0. \tag*{(16)} \end{aligned}

To represent the solution vector field u(x,t)u(x,t), we assume the PINN model a(xi;θ)a(\boldsymbol{x}_i;\boldsymbol{\theta}) where xi\boldsymbol{x}_i contains a single space and time coordinate from the physics training data xx and tt. Substituting this into the 1D Burgers’ partial differential equation, we get the following residual to measure:

Rθ(xi)=∂a(xi;θ)∂xi(t)+a(xi;θ)∂a(xi;θ)∂xi(x)−ν∂2a(xi;θ)∂xi(x)2.(17)\mathcal{R}_{\boldsymbol{\theta}}(\boldsymbol{x}_i) = \frac{\partial a(\boldsymbol{x}_i;\boldsymbol{\theta})}{\partial x_i(t)} + a(\boldsymbol{x}_i;\boldsymbol{\theta})\frac{\partial a(\boldsymbol{x}_i;\boldsymbol{\theta})}{\partial x_i(x)} - \nu\frac{\partial^2 a(\boldsymbol{x}_i;\boldsymbol{\theta})}{\partial x_i(x)^2}. \tag*{(17)}
AspectDetails
Repetitions30 runs per experiment.
Noise levels (σ\sigma)0, 0.5, 1, 2, 3, 5, 7, 10, 25.
Spatial resolutiondx=2255≈0.00784dx = \frac{2}{255} \approx0.00784.
Temporal resolutiondt=0.01dt = 0.01.
DatasetFrom PINN paper [4].
Total data points100 time steps · 256 positions = 25600.
Training data10 time steps · 256 = 2560 (10%).
Validation data2 time steps · 256 = 512.
Test data88 time steps · 256 = 22528 (88%).
Initial condition data160 randomly generated points.
Boundary condition data80 randomly generated points.
Physics data2540 random (x,tx,t) points.

Table A.1. 1D Burgers’ equation, shared setup.

AspectDetails
Basis functions1st degree polynomials.
Training dataTraining and validation sets.
DiscretizationSame as training data.
SolverNewton, max 50 iter, no preconditioner.
Initial guessν=5\nu= 5.
SLSQP settingsFtol=1e−16Ftol=1e-16, max iter 100, bounds [−5,5][-5,5].
Gradient calc.Adjoint method (pyadjoint).

Table A.2. 1D Burgers’ model setups, FEM/SLSQP setup.

AspectDetails
Architecture4 layers, 20 neurons/layer.
ActivationTanh.
Batch sizeFull-batch.
Training dataTraining data, BC, IC, physics points.
Training epochs4000 (ADAM) + 4000 (L-BFGS).
Early stoppingBest results without.
Adaptive weightsBest results without.
Parameter searchLogarithmic scale.
Initial guessν=5\nu= 5.
Loss weightw=1w = 1.
Adam, learning rate0.001 (default).
L-BFGS, learning rate1 (default).
L-BFGS, history size50.
L-BFGS, line searchStrong Wolfe.
L-BFGS, rtol and atol1e-16, 1e-16.

Table A.3. 1D Burgers’ model setups, PINN setup.

2D Navier-Stokes equations

For FEM, we used the algorithm Chorin’s projection [51]. Each time step requires solving three systems. First we solve for utempu^{temp}, ignoring pressure:

∫Ωutemp−unΔt⋅v dx+∫Ω(un⋅∇un)⋅v dx+ν∫Ω∇utemp:∇v dx=0.(18)\int_{\Omega} \frac{\mathbf{u}^{temp}-\mathbf{u}^{n}}{\Delta t} \cdot\mathbf{v}\,dx + \int_{\Omega} \left(\mathbf{u}^{n}\cdot\nabla\mathbf{u}^{n}\right)\cdot\mathbf{v}\,dx + \nu\int_{\Omega}\nabla\mathbf{u}^{temp}:\nabla\mathbf{v}\,dx = 0. \tag*{(18)}

Next, we solve for the pressure pp using the velocity utempu^{temp} from last step:

∫Ω∇p⋅∇q dx+1Δt∫Ω(∇⋅utemp)q dx=0,(19)\int_{\Omega}\nabla p\cdot\nabla q\,dx+\frac{1}{\Delta t}\int_{\Omega}\left(\nabla\cdot\mathbf{u}^{temp}\right)q\,dx=0, \tag*{(19)}

where qq is a test function. With pp we can get the corrected, real un+1u^{n+1}:

∫Ωun+1⋅v dx−∫Ωutemp⋅v dx+Δt∫Ω∇pn+1⋅v dx=0.(20)\int_{\Omega}\mathbf{u}^{n+1}\cdot\mathbf{v}\,dx-\int_{\Omega}\mathbf{u}^{temp}\cdot\mathbf{v}\,dx+\Delta t\int_{\Omega}\nabla p^{n+1}\cdot\mathbf{v}\,dx=0. \tag*{(20)}

For incompressible Navier-Stokes systems with periodic boundary conditions, the pressure is only unique up to an additive constant. In our implementation, this gauge condition is not enforced during the discrete solve. We obtain a pressure field with a fixed gauge by subtracting the spatial average of the pressure, thus enforcing a zero-mean pressure condition. Since we only compare velocity predictions in our study, this choice does not affect the results.

By using the identities from 4.2.1, by substitution we have ui=∂a(xi;θ)∂xi(y)u_i = \frac{\partial a(\mathbf{x}_i;\theta)}{\partial x_i(y)}, vi=−∂a(xi;θ)∂xi(x)v_i = -\frac{\partial a(\mathbf{x}_i;\theta)}{\partial x_i(x)} and the residual functions for the physics loss:

R1,θ(xi)=∂ui∂xi(t)+ui∂ui∂xi(x)+vi∂ui∂xi(y)+∂p(xi;θ)∂xi(x)−ν(∂2ui∂xi(x)2+∂2ui∂xi(y)2),R_{1,\theta}(\mathbf{x}_i) = \frac{\partial u_i}{\partial x_i(t)} + u_i\frac{\partial u_i}{\partial x_i(x)} + v_i\frac{\partial u_i}{\partial x_i(y)} + \frac{\partial p(\mathbf{x}_i;\theta)}{\partial x_i(x)} - \nu\left(\frac{\partial^2 u_i}{\partial x_i(x)^2} + \frac{\partial^2 u_i}{\partial x_i(y)^2}\right),
R2,θ(xi)=∂vi∂xi(t)+ui∂vi∂xi(x)+vi∂vi∂xi(y)+∂p(xi;θ)∂xi(y)−ν(∂2vi∂xi(x)2+∂2vi∂xi(y)2),R_{2,\theta}(\mathbf{x}_i) = \frac{\partial v_i}{\partial x_i(t)} + u_i\frac{\partial v_i}{\partial x_i(x)} + v_i\frac{\partial v_i}{\partial x_i(y)} + \frac{\partial p(\mathbf{x}_i;\theta)}{\partial x_i(y)} - \nu\left(\frac{\partial^2 v_i}{\partial x_i(x)^2} + \frac{\partial^2 v_i}{\partial x_i(y)^2}\right),
Rθ(xi)=(R1,θ(xi),R2,θ(xi))T.(21)R_{\theta}(\mathbf{x}_i) = (R_{1,\theta}(\mathbf{x}_i), R_{2,\theta}(\mathbf{x}_i))^T. \tag*{(21)}

Here we assumed that each xi\mathbf{x}_i contains the coordinates xx, yy and time tt.

AspectDetails shared
Repetitions30 runs per experiment.
Noise levels (σ\sigma)0, 0.001, 0.01, 0.05, 0.1, 0.25, 0.5, 1.
Spatial resolutiondx=dy=0.05dx = dy = 0.05.
Temporal resolutiondt=0.1dt = 0.1.
DatasetGenerated from analytical solution.
Total data points26 time steps ⋅\cdot 126 ⋅\cdot 126 =412776= 412776 observations.
Training data99225 random samples, ≈24%\approx24\% of total.
Validation data19845 random samples, ≈4.8%\approx4.8\% of total.
Test data293706 random samples, ≈71.12%\approx71.12\% of total.
Initial condition data4000 randomly generated positions.
Boundary condition data4000 randomly generated positions.
Physics dataThe x\mathbf{x} inputs from the training data.

Table A.4. 2D Navier-Stokes equation, shared setup.

AspectDetails for FEM/SLSQP
Basis functionsCubic polynomial for velocity, quadratic polynomial for pressure.
Training dataTraining and validation sets.
Discretization642=409664^2 = 4096, dx=dy=2π64≈0.098175dx = dy = \frac{2\pi}{64} \approx0.098175, dt=0.01dt = 0.01.
SolverGmres solver with amg preconditioner. Abstol = 1e-12, rtol=1e-10, maximum iterations 5000.
Initial guessν=5\nu= 5.
SLSQP settingsFtol=1e−16, max iter 100, bounds [0.003141, 5].
Gradient calc.3-point finite difference (SciPy).

Table A.5. 2D Navier-Stokes, FEM/SLSQP setup.

AspectDetails for PINN
Architecture9 layers, 20 neurons/layer.
ActivationTanh.
Batch sizeFull-batch.
Training dataTraining data, BC, IC, physics points.
Training epochs200000 (ADAM) + 50000 (L-BFGS).
Early stoppingBest model, based on the validation set, is loaded at end of training. Models from the first 10,000 epochs are ignored.
Adaptive weightsYes, was needed [24].
Parameter searchSoftplus used to bound to [0.003141, ∞\infty).
Initial guessν=5\nu= 5.
Adam, learning rate0.001 (default).
L-BFGS, learning rate1 (default).
L-BFGS, history size50.
L-BFGS, line searchStrong Wolfe.
L-BFGS, rtol and atol1e-16, 1e-16.

Table A.6. 2D Navier-Stokes, PINN setup.

3D Navier-Stokes equations

As for 2D Navier-Stokes, we used Chorin’s projection described in dd-dimensional form. For the physics loss, the residual function was implemented as follows, assuming each xix_i has coordinates xx, yy, zz and time tt:

R1,θ(xi)=∂u(xi;θ)∂xi(t)+u(xi;θ)∂u(xi;θ)∂xi(x)+v(xi;θ)∂u(xi;θ)∂xi(y)+w(xi;θ)∂u(xi;θ)∂xi(z)+1ρ∂p(xi;θ)∂xi(x)−ν(∂2u(xi;θ)∂xi(x)2+∂2u(xi;θ)∂xi(y)2+∂2u(xi;θ)∂xi(z)2),(22)\mathcal{R}_{1,\theta}(x_i) = \frac{\partial u(x_i;\theta)}{\partial x_i(t)} + u(x_i;\theta)\frac{\partial u(x_i;\theta)}{\partial x_i(x)} + v(x_i;\theta)\frac{\partial u(x_i;\theta)}{\partial x_i(y)} + w(x_i;\theta)\frac{\partial u(x_i;\theta)}{\partial x_i(z)} + \frac{1}{\rho}\frac{\partial p(x_i;\theta)}{\partial x_i(x)} - \nu\left(\frac{\partial^2u(x_i;\theta)}{\partial x_i(x)^2}+\frac{\partial^2u(x_i;\theta)}{\partial x_i(y)^2}+\frac{\partial^2u(x_i;\theta)}{\partial x_i(z)^2}\right), \tag*{(22)}
R2,θ(xi)=∂v(xi;θ)∂xi(t)+u(xi;θ)∂v(xi;θ)∂xi(x)+v(xi;θ)∂v(xi;θ)∂xi(y)+w(xi;θ)∂v(xi;θ)∂xi(z)+1ρ∂p(xi;θ)∂xi(y)−ν(∂2v(xi;θ)∂xi(x)2+∂2v(xi;θ)∂xi(y)2+∂2v(xi;θ)∂xi(z)2),(23)\mathcal{R}_{2,\theta}(x_i) = \frac{\partial v(x_i;\theta)}{\partial x_i(t)} + u(x_i;\theta)\frac{\partial v(x_i;\theta)}{\partial x_i(x)} + v(x_i;\theta)\frac{\partial v(x_i;\theta)}{\partial x_i(y)} + w(x_i;\theta)\frac{\partial v(x_i;\theta)}{\partial x_i(z)} + \frac{1}{\rho}\frac{\partial p(x_i;\theta)}{\partial x_i(y)} - \nu\left(\frac{\partial^2v(x_i;\theta)}{\partial x_i(x)^2}+\frac{\partial^2v(x_i;\theta)}{\partial x_i(y)^2}+\frac{\partial^2v(x_i;\theta)}{\partial x_i(z)^2}\right), \tag*{(23)}
R3,θ(xi)=∂w(xi;θ)∂xi(t)+u(xi;θ)∂w(xi;θ)∂xi(x)+v(xi;θ)∂w(xi;θ)∂xi(y)+w(xi;θ)∂w(xi;θ)∂xi(z)+1ρ∂p(xi;θ)∂xi(z)−ν(∂2w(xi;θ)∂xi(x)2+∂2w(xi;θ)∂xi(y)2+∂2w(xi;θ)∂xi(z)2),(24)\mathcal{R}_{3,\theta}(x_i) = \frac{\partial w(x_i;\theta)}{\partial x_i(t)} + u(x_i;\theta)\frac{\partial w(x_i;\theta)}{\partial x_i(x)} + v(x_i;\theta)\frac{\partial w(x_i;\theta)}{\partial x_i(y)} + w(x_i;\theta)\frac{\partial w(x_i;\theta)}{\partial x_i(z)} + \frac{1}{\rho}\frac{\partial p(x_i;\theta)}{\partial x_i(z)} - \nu\left(\frac{\partial^2w(x_i;\theta)}{\partial x_i(x)^2}+\frac{\partial^2w(x_i;\theta)}{\partial x_i(y)^2}+\frac{\partial^2w(x_i;\theta)}{\partial x_i(z)^2}\right), \tag*{(24)}
R4,θ(xi)=∂u(xi;θ)∂xi(x)+∂v(xi;θ)∂xi(y)+∂w(xi;θ)∂xi(z),(25)\mathcal{R}_{4,\theta}(x_i) = \frac{\partial u(x_i;\theta)}{\partial x_i(x)} + \frac{\partial v(x_i;\theta)}{\partial x_i(y)} + \frac{\partial w(x_i;\theta)}{\partial x_i(z)}, \tag*{(25)}
Rθ(xi)=(R1,θ(xi),R2θ(xi),R3,θ(xi),R4,θ(xi))T.(26)\mathcal{R}_{\theta}(x_i) = \left(\mathcal{R}_{1,\theta}(x_i),\mathcal{R}_{2\theta}(x_i),\mathcal{R}_{3,\theta}(x_i),\mathcal{R}_{4,\theta}(x_i)\right)^T. \tag*{(26)}
AspectDetails
Repetitions30 runs per experiment.
Noise levels (σ\sigma)0, 0.001, 0.01, 0.05, 0.1, 0.25, 0.5, 1.
Spatial resolutiondx=dy=2π128≈0.0490876dx = dy = \frac{2\pi}{128} \approx0.0490876.
Temporal resolutionSolved with dt=0.005dt = 0.005, saved with dt=0.25dt = 0.25.
DatasetGenerated using Nektar++ [44].
Total data points1100000 random samples from generated solution.
Training data275000 random samples, = 25% of total.
Test data770000 random samples, = 75% of total.
Validation data55000 random samples, = 5% of total.
Initial condition data16000 randomly generated positions.
Boundary condition data8000 random positions from generated solution.
None shared with training data.
Physics dataThe xx inputs from the training data.

Table A.7. 3D Navier-Stokes equation, shared setup.

AspectDetails
Basis functionsQuadratic polynomial for velocity, 1st degree polynomial for pressure.
Training dataTraining and validation sets.
Discretization323=3276832^3 = 32768, dx=dy=dz=2π32≈0.1963495dx = dy = dz = \frac{2\pi}{32} \approx0.1963495, dt=0.05dt = 0.05.
SolverGmres solver with amg preconditioner. Abstol = 1e-12, rtol=1e-10, maximum iterations 50000.
Initial guessν=5\nu= 5.
SLSQP settingsFtol=1e-16, max iter 100, bounds [0.003141, 5].
Gradient calc.3-point finite difference (SciPy).

Table A.8. 3D Navier-Stokes equation, FEM/SLSQP setup.

AspectDetails
Architecture9 layers, 20 neurons/layer.
ActivationTanh.
Batch sizeFull-batch.
Training dataTraining data, BC, IC, physics points.
Training epochs200000 (ADAM) + 50000 (L-BFGS).
Early stoppingBest model, based on the validation set, is loaded at end of training. Models from the first 10,000 epochs are ignored.
Adaptive weightsYes, was needed. [24].
Parameter searchSoftplus used to bound to [0.003141, ∞\infty).
Initial guessν=5\nu= 5.
Adam, learning rate0.001 (default).
L-BFGS, learning rate1 (default).
L-BFGS, history size50.
L-BFGS, line searchStrong Wolfe.
L-BFGS, rtol and atol1e-16, 1e-16.

Table A.9. 3D Navier-Stokes equation, PINN setup.

Adaptive weights

Adaptive weights

For the Navier-Stokes problems, we used a gradient balancing algorithm [24]. At regular intervals, the algorithm calculates the difference in magnitude of the maximum physics loss gradients and the average data loss gradients:

w^=max⁡(∣∇Lphys(x)∣)∣∇Ldata(x)∣.(27)\hat{w} = \frac{\max(\lvert\nabla\mathcal{L}_{\mathrm{phys}}(x)\rvert)}{\lvert\nabla\mathcal{L}_{\mathrm{data}}(x)\rvert}. \tag*{(27)}

The data weight ww is updated using w^\hat{w}, with α\alpha commonly chosen as 0.9:

w←(1−α)w+αw^.(28)w \leftarrow(1-\alpha)w+\alpha\hat{w}. \tag*{(28)}

The weights are updated every 10 epochs, which is the frequency discussed in the paper [24]. We tried lower update frequencies, but this did not give satisfactory performance. Note that the weights are detached from the computational graph. Boundary condition and initial condition data is combined with the rest of the training data and they are all weighted together as one. The physics loss is not weighted. Below are some sample graphs demonstrating how the weights evolve during training.

Image not extracted

Figure 7 (PDF p. 38).

Weight histories for 2D and 3D Taylor-Green vortices

Figure 8 (PDF p. 39).

Weight history for 3D Taylor-Green Vortex with $\sigma=1$

Figure 9 (PDF p. 40).

Visually the graphs are similar to what is shown in the original paper [24], but the weight values are much larger. Note that for 2D Taylor-Green Vortex with σ=1\sigma= 1 and 3D Taylor-Green Vortex, where there was struggle with apparent overfitting, the weights tend to grow during training.

Prediction results

In this section we report the mean and standard deviation for each model’s RMSE performance on the test set for all the problems studied. The difference between each pair of models is quantified using the Vargha-Delaney AA statistic [52]. Its value indicates a probability between 0 and 1 that the reference model (the first model in each set of tables) has larger RMSE than the other model. To quantify whether the distribution of the results of each pair of models are different to a statistically significant degree, we used the Brunner-Munzel test [53]. This is justified by the results from the tests generally not being normally distributed. Note that Brunner-Munzel is undefined when the distributions of each model do not overlap at all. For the 1D Burgers’ problems with σ=0\sigma= 0, FEM is deterministic, and variance is 0 which also means Brunner-Munzel becomes undefined. For both these cases, tests are simply reported as N/A. We provide the original pp values, as well as the pp values adjusted using the Holm family-wise error rate correction [54] and the Benjamini-Hochberg false discovery rate correction [55]. For the Holm-derived pp values, all the values below 0.05% have been marked in bold.

1D Burgers' equation

NoisePINNFEMVargha–Delaney A
00.005366 ± 0.0005950.006515 ± 00.067
0.0010.022701 ± 0.0068440.010623 ± 0.0043500.956
0.010.049035 ± 0.0092640.020314 ± 0.0160210.914
0.050.095079 ± 0.0236050.026936 ± 0.0198830.974
0.10.144118 ± 0.0391720.058326 ± 0.0450870.914
0.250.205870 ± 0.0575950.110683 ± 0.1481790.832
0.50.257481 ± 0.0808850.160393 ± 0.1713010.791
10.452368 ± 0.2125030.187883 ± 0.1925280.822

Table B.10. PINN and FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0N/AN/AN/AN/A
0.001−15.3982.537e − 161.522e − 158.879e − 16
0.01−9.0324.370e − 101.748e − 097.647e − 10
0.05−20.9201.023e − 197.160e − 197.160e − 19
0.1−10.2803.039e − 121.520e − 117.091e − 12
0.25−5.2071.280e − 052.560e − 051.493e − 05
0.5−4.3241.427e − 041.427e − 041.427e − 04
1−5.7121.243e − 063.729e − 061.740e − 06

Table B.11. Brunner-Munzel tests and adjusted p-values for PINN and FEM results.

NoiseFEMPINN/FEMVargha–Delaney AA
00.006515±00.006515 \pm00.006799±0.0001400.006799 \pm0.0001400.033
0.0010.010623±0.0043500.010623 \pm0.0043500.012746±0.0066230.012746 \pm0.0066230.462
0.010.020314±0.0160210.020314 \pm0.0160210.027380±0.0130510.027380 \pm0.0130510.311
0.050.026936±0.0198830.026936 \pm0.0198830.040960±0.0190250.040960 \pm0.0190250.263
0.10.058326±0.0450870.058326 \pm0.0450870.049486±0.0212130.049486 \pm0.0212130.507
0.250.110683±0.1481790.110683 \pm0.1481790.056091±0.0318970.056091 \pm0.0318970.504
0.50.160393±0.1713010.160393 \pm0.1713010.060345±0.0357930.060345 \pm0.0357930.677
10.187883±0.1925280.187883 \pm0.1925280.147435±0.1759610.147435 \pm0.1759610.529

Table B.12. FEM and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0N/AN/AN/AN/A
0.0010.4820.63211.00000.9565
0.012.5730.01270.07630.0445
0.053.4050.00138.800e−038.800\mathrm{e}{-03}0.0088
0.1−0.083-0.0830.93421.00000.9565
0.25−0.055-0.0550.95651.00000.9565
0.5−2.347-2.3470.02450.12230.0571
1−0.373-0.3730.71101.00000.9565

Table B.13. Brunner-Munzel tests and adjusted p-values for FEM and PINN/FEM results.

NoisePINNPINN/FEMVargha–Delaney AA
00.005366±0.0005950.005366 \pm0.0005950.006799±0.0001400.006799 \pm0.0001400.013
0.0010.022701±0.0068440.022701 \pm0.0068440.012746±0.0066230.012746 \pm0.0066230.847
0.010.049035±0.0092640.049035 \pm0.0092640.027380±0.0130510.027380 \pm0.0130510.927
0.050.095079±0.0236050.095079 \pm0.0236050.040960±0.0190250.040960 \pm0.0190250.969
0.10.144118±0.0391720.144118 \pm0.0391720.049486±0.0212130.049486 \pm0.0212130.989
0.250.205870±0.0575950.205870 \pm0.0575950.056091±0.0318970.056091 \pm0.0318970.988
0.50.257481±0.0808850.257481 \pm0.0808850.060345±0.0357930.060345 \pm0.0357930.998
10.452368±0.2125030.452368 \pm0.2125030.147435±0.1759610.147435 \pm0.1759610.878

Table B.14. PINN and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
038.3042.931e−292.931e-291.466e−28\mathbf{1.466e-28}5.863e−295.863e-29
0.001−6.798-6.7982.676e−082.676e-082.676e−08\mathbf{2.676e-08}2.676e−082.676e-08
0.01−12.038-12.0382.228e−152.228e-156.685e−15\mathbf{6.685e-15}2.971e−152.971e-15
0.05−26.468-26.4687.718e−347.718e-344.631e−33\mathbf{4.631e-33}2.058e−332.058e-33
0.1−52.720-52.7202.058e−442.058e-441.441e−43\mathbf{1.441e-43}8.233e−448.233e-44
0.25−38.771-38.7719.420e−299.420e-293.768e−28\mathbf{3.768e-28}1.507e−281.507e-28
0.5−183.956-183.9561.679e−741.679e-741.343e−73\mathbf{1.343e-73}1.343e−731.343e-73
1−7.514-7.5141.142e−081.142e-082.284e−08\mathbf{2.284e-08}1.305e−081.305e-08

Table B.15. Brunner-Munzel tests and adjusted p-values for PINN and PINN/FEM results.

1D Burgers' equation with two parameters

NoisePINNFEMVargha–Delaney AA
00.006525±0.0000480.006525 \pm0.0000480.004709±00.004709 \pm01.000
0.0010.028642±0.0078880.028642 \pm0.0078880.013563±0.0068900.013563 \pm0.0068900.922
0.010.063422±0.0139700.063422 \pm0.0139700.022783±0.0099900.022783 \pm0.0099900.988
0.050.123037±0.0306580.123037 \pm0.0306580.043907±0.0196440.043907 \pm0.0196440.991
0.10.194091±0.0815610.194091 \pm0.0815610.060095±0.0369790.060095 \pm0.0369790.976
0.250.334672±0.2088180.334672 \pm0.2088180.128104±0.1178910.128104 \pm0.1178910.938
0.50.484270±0.2671890.484270 \pm0.2671890.183947±0.2095370.183947 \pm0.2095370.870
10.733201±0.4051810.733201 \pm0.4051810.287932±0.2460030.287932 \pm0.2460030.849

Table B.16. PINN and FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0N/AN/AN/AN/A
0.001−12.126-12.1263.650e−153.650e-151.460e−141.460e-146.387e−156.387e-15
0.01−52.367-52.3672.276e−442.276e-441.594e−431.594e-431.594e−431.594e-43
0.05−52.803-52.8031.488e−341.488e-348.928e−348.928e-345.208e−345.208e-34
0.1−23.988-23.9882.868e−222.868e-221.434e−211.434e-216.692e−226.692e-22
0.25−12.536-12.5363.325e−143.325e-149.975e−149.975e-144.655e−144.655e-14
0.5−6.778-6.7781.138e−071.138e-071.138e−071.138e-071.138e−071.138e-07
1−6.632-6.6323.979e−083.979e-087.957e−087.957e-084.642e−084.642e-08

Table B.17. Brunner-Munzel tests and adjusted p-values for PINN and FEM results.

NoiseFEMPINN/FEMVargha–Delaney AA
00.004709±00.004709 \pm00.004786±0.0000330.004786 \pm0.0000330.000
0.0010.013563±0.0068900.013563 \pm0.0068900.017567±0.0104220.017567 \pm0.0104220.382
0.010.022783±0.0099900.022783 \pm0.0099900.044224±0.0225870.044224 \pm0.0225870.180
0.050.043907±0.0196440.043907 \pm0.0196440.093578±0.0439490.093578 \pm0.0439490.169
0.10.060095±0.0369790.060095 \pm0.0369790.106218±0.0564910.106218 \pm0.0564910.206
0.250.128104±0.1178910.128104 \pm0.1178910.135507±0.0771780.135507 \pm0.0771780.428
0.50.183947±0.2095370.183947 \pm0.2095370.176300±0.1182120.176300 \pm0.1182120.372
10.287932±0.2460030.287932 \pm0.2460030.236276±0.1620770.236276 \pm0.1620770.517

Table B.18. FEM and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0N/AN/AN/AN/A
0.0011.6070.11350.39740.1589
0.015.4673.183e−063.183e-061.910e−051.910e-051.114e−051.114e-05
0.056.1472.347e−072.347e-071.643e−061.643e-061.643e−061.643e-06
0.15.0805.725e−065.725e-062.862e−052.862e-051.336e−051.336e-05
0.250.9590.34160.68310.3985
0.51.6860.09930.39740.1589
1−0.212-0.2120.83290.83290.8329

Table B.19. Brunner-Munzel tests and adjusted p-values for FEM and PINN/FEM results.

NoisePINNPINN/FEMVargha–Delaney AA
00.006525±0.0000480.006525 \pm0.0000480.004786±0.0000330.004786 \pm0.0000331.0001.000
0.0010.028642±0.0078880.028642 \pm0.0078880.017567±0.0104220.017567 \pm0.0104220.8320.832
0.010.063422±0.0139700.063422 \pm0.0139700.044224±0.0225870.044224 \pm0.0225870.7840.784
0.050.123037±0.0306580.123037 \pm0.0306580.093578±0.0439490.093578 \pm0.0439490.7060.706
0.10.194091±0.0815610.194091 \pm0.0815610.106218±0.0564910.106218 \pm0.0564910.8660.866
0.250.334672±0.2088180.334672 \pm0.2088180.135507±0.0771780.135507 \pm0.0771780.8960.896
0.50.484270±0.2671890.484270 \pm0.2671890.176300±0.1182120.176300 \pm0.1182120.9210.921
10.733201±0.4051810.733201 \pm0.4051810.236276±0.1620770.236276 \pm0.1620770.9440.944

Table B.20. PINN and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0N/AN/AN/AN/A
0.001−5.694-5.6942.003e−062.003e-066.008e−066.008e-062.804e−062.804e-06
0.01−4.561-4.5613.622e−053.622e-057.243e−057.243e-054.225e−054.225e-05
0.05−3.024-3.0240.00390.00393.900e−033.900e-030.00390.0039
0.1−7.310-7.3108.469e−098.469e-093.388e−083.388e-081.482e−081.482e-08
0.25−9.102-9.1025.139e−115.139e-112.569e−102.569e-101.199e−101.199e-10
0.5−11.928-11.9281.006e−151.006e-157.040e−157.040e-156.471e−156.471e-15
1−13.773-13.7731.849e−151.849e-151.109e−141.109e-146.471e−156.471e-15

Table B.21. Brunner-Munzel tests and adjusted p-values for PINN and PINN/FEM results.

2D Taylor-Green Vortex

NoisePINNFEMVargha–Delaney AA
00.000426±0.0006860.000426 \pm0.0006860.000151±0.0000050.000151 \pm0.0000050.257
0.0010.000372±0.0005970.000372 \pm0.0005970.000151±0.0000050.000151 \pm0.0000050.268
0.010.000364±0.0004100.000364 \pm0.0004100.000152±0.0000060.000152 \pm0.0000061.000
0.050.001304±0.0004000.001304 \pm0.0004000.000189±0.0000420.000189 \pm0.0000421.000
0.10.002315±0.0004030.002315 \pm0.0004030.000240±0.0000930.000240 \pm0.0000931.000
0.250.005814±0.0006030.005814 \pm0.0006030.000499±0.0002510.000499 \pm0.0002511.000
0.50.009809±0.0010760.009809 \pm0.0010760.000902±0.0005880.000902 \pm0.0005881.000
10.015522±0.0024750.015522 \pm0.0024750.001336±0.0008220.001336 \pm0.0008221.000

Table B.22. PINN and FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
03.0550.00489.576e−039.576\mathrm{e}{-03}0.0083
0.0012.8320.00839.576e−039.576\mathrm{e}{-03}0.0083
0.01N/AN/AN/AN/A
0.05N/AN/AN/AN/A
0.1N/AN/AN/AN/A
0.25N/AN/AN/AN/A
0.5N/AN/AN/AN/A
1N/AN/AN/AN/A

Table B.23. Brunner-Munzel tests and adjusted p-values for PINN and FEM results.

NoiseFEMPINN/FEMVargha–Delaney AA
000.000151±0.0000050.000151 \pm0.0000050.000755±0.0012810.000755 \pm0.0012810.0590.059
0.0010.0010.000151±0.0000050.000151 \pm0.0000050.000719±0.0012670.000719 \pm0.0012670.0770.077
0.010.010.000152±0.0000060.000152 \pm0.0000060.000449±0.0007450.000449 \pm0.0007450.0610.061
0.050.050.000189±0.0000420.000189 \pm0.0000420.001320±0.0009500.001320 \pm0.0009500.0790.079
0.10.10.000240±0.0000930.000240 \pm0.0000930.002353±0.0015310.002353 \pm0.0015310.0640.064
0.250.250.000499±0.0002510.000499 \pm0.0002510.005645±0.0025470.005645 \pm0.0025470.0200.020
0.50.50.000902±0.0005880.000902 \pm0.0005880.009270±0.0034370.009270 \pm0.0034370.0000.000
110.001336±0.0008220.001336 \pm0.0008220.010382±0.0069040.010382 \pm0.0069040.0430.043

Table B.24. FEM and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0014.20814.2081.936e−161.936e-161.162e−15\mathbf{1.162e-15}6.252e−166.252e-16
0.0010.00110.19310.1931.985e−111.985e-113.971e−11\mathbf{3.971e-11}2.316e−112.316e-11
0.010.0112.76012.7601.606e−141.606e-146.423e−14\mathbf{6.423e-14}2.810e−142.810e-14
0.050.059.3519.3512.130e−102.130e-102.130e−10\mathbf{2.130e-10}2.130e−102.130e-10
0.10.111.30811.3081.842e−121.842e-125.525e−12\mathbf{5.525e-12}2.578e−122.578e-12
0.250.2528.87328.8733.530e−253.530e-252.471e−24\mathbf{2.471e-24}2.471e−242.471e-24
0.50.5N/AN/AN/AN/A
1115.44415.4442.679e−162.679e-161.340e−15\mathbf{1.340e-15}6.252e−166.252e-16

Table B.25. Brunner-Munzel tests and adjusted p-values for PINN and PINN/FEM results.

NoisePINNPINN/FEMVargha–Delaney AA
000.000426±0.0006860.000426 \pm0.0006860.000755±0.0012810.000755 \pm0.0012810.1970.197
0.0010.0010.000372±0.0005970.000372 \pm0.0005970.000719±0.0012670.000719 \pm0.0012670.1960.196
0.010.010.000364±0.0004100.000364 \pm0.0004100.000449±0.0007450.000449 \pm0.0007450.5310.531
0.050.050.001304±0.0004000.001304 \pm0.0004000.001320±0.0009500.001320 \pm0.0009500.5670.567
0.10.10.002315±0.0004030.002315 \pm0.0004030.002353±0.0015310.002353 \pm0.0015310.5180.518
0.250.250.005814±0.0006030.005814 \pm0.0006030.005645±0.0025470.005645 \pm0.0025470.5270.527
0.50.50.009809±0.0010760.009809 \pm0.0010760.009270±0.0034370.009270 \pm0.0034370.5820.582
110.015522±0.0024750.015522 \pm0.0024750.010382±0.0069040.010382 \pm0.0069040.8760.876

Table B.26. PINN and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
04.5357.323e−057.323e - 054.394e−044.394e - 041.953e−041.953e - 04
0.0014.6874.668e−054.668e - 053.267e−043.267e - 041.867e−041.867e - 04
0.01−0.3850.70221.00000.8330
0.05−0.8060.42601.00000.6816
0.1−0.2130.83301.00000.8330
0.25−0.3140.75531.00000.8330
0.5−0.9640.34241.00000.6816
1−6.9517.953e−087.953e - 086.362e−076.362e - 076.362e−076.362e - 07

Table B.27. Brunner-Munzel tests and adjusted p-values for PINN and PINN/FEM results.

3D Taylor-Green Vortex, with outliers

NoisePINNFEMVargha–Delaney AA
00.032690±0.0095980.032690 \pm0.0095980.002516±0.0025960.002516 \pm0.0025961.000
0.0010.039061±0.0098500.039061 \pm0.0098500.011303±0.0496500.011303 \pm0.0496500.967
0.010.036553±0.0105520.036553 \pm0.0105520.030073±0.0828020.030073 \pm0.0828020.896
0.050.036072±0.0115410.036072 \pm0.0115410.082922±0.3815180.082922 \pm0.3815180.923
0.10.036208±0.0106800.036208 \pm0.0106800.012680±0.0496220.012680 \pm0.0496220.956
0.250.038357±0.0101490.038357 \pm0.0101490.020584±0.0689310.020584 \pm0.0689310.933
0.50.040931±0.0090310.040931 \pm0.0090310.003665±0.0050020.003665 \pm0.0050021.000
10.051235±0.0041870.051235 \pm0.0041870.013193±0.0495240.013193 \pm0.0495240.967

Table B.28. PINN and FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0N/AN/AN/AN/A
0.001−14.0001.964e−141.964e-141.178e−131.178e-135.891e−145.891e-14
0.01−7.1087.914e−087.914e-087.914e−087.914e-087.914e−087.914e-08
0.05−9.1054.065e−104.065e-108.786e−108.786e-104.878e−104.878e-10
0.1−13.3393.640e−143.640e-141.456e−131.456e-137.281e−147.281e-14
0.25−9.3552.929e−102.929e-108.786e−108.786e-104.393e−104.393e-10
0.5N/AN/AN/AN/A
1−14.0001.964e−141.964e-141.178e−131.178e-135.891e−145.891e-14

Table B.29. Brunner-Munzel tests and adjusted p-values for PINN and FEM results.

NoiseFEMPINN/FEMVargha–Delaney AA
00.002516±0.0025960.002516 \pm0.0025960.006938±0.0061730.006938 \pm0.0061730.050
0.0010.011303±0.0496500.011303 \pm0.0496500.012048±0.0092030.012048 \pm0.0092030.061
0.010.030073±0.0828020.030073 \pm0.0828020.009024±0.0075700.009024 \pm0.0075700.194
0.050.082922±0.3815180.082922 \pm0.3815180.011518±0.0100640.011518 \pm0.0100640.171
0.10.012680±0.0496220.012680 \pm0.0496220.010777±0.0084860.010777 \pm0.0084860.123
0.250.020584±0.0689310.020584 \pm0.0689310.013890±0.0228050.013890 \pm0.0228050.138
0.50.003665±0.0050020.003665 \pm0.0050020.012939±0.0094450.012939 \pm0.0094450.103
10.013193±0.0495240.013193 \pm0.0495240.010349±0.0097720.010349 \pm0.0097720.247

Table B.30. FEM and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
012.6871.423e−131.423e-131.138e−121.138e-121.138e−121.138e-12
0.00111.5011.300e−121.300e-129.097e−129.097e-125.198e−125.198e-12
0.014.5334.887e−054.887e-059.774e−059.774e-055.585e−055.585e-05
0.055.3596.640e−066.640e-061.992e−051.992e-058.853e−068.853e-06
0.17.4821.137e−081.137e-085.683e−085.683e-082.273e−082.273e-08
0.256.1071.825e−071.825e-077.300e−077.300e-072.920e−072.920e-07
0.58.7553.877e−123.877e-122.326e−112.326e-111.034e−111.034e-11
13.9632.105e−042.105e-042.105e−042.105e-042.105e−042.105e-04

Table B.31. Brunner-Munzel tests and adjusted p-values for PINN and PINN/FEM results.

NoisePINNPINN/FEMVargha–Delaney AA
00.032690±0.0095980.032690 \pm0.0095980.006938±0.0061730.006938 \pm0.0061730.9820.982
0.0010.039061±0.0098500.039061 \pm0.0098500.012048±0.0092030.012048 \pm0.0092030.9690.969
0.010.036553±0.0105520.036553 \pm0.0105520.009024±0.0075700.009024 \pm0.0075700.9640.964
0.050.036072±0.0115410.036072 \pm0.0115410.011518±0.0100640.011518 \pm0.0100640.9400.940
0.10.036208±0.0106800.036208 \pm0.0106800.010777±0.0084860.010777 \pm0.0084860.9390.939
0.250.038357±0.0101490.038357 \pm0.0101490.013890±0.0228050.013890 \pm0.0228050.9670.967
0.50.040931±0.0090310.040931 \pm0.0090310.012939±0.0094450.012939 \pm0.0094450.9860.986
10.051235±0.0041870.051235 \pm0.0041870.010349±0.0097720.010349 \pm0.0097721.0001.000

Table B.32. PINN and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0−40.513-40.5133.069e−443.069e - 442.149e−432.149e - 432.149e−432.149e - 43
0.001−24.209-24.2091.402e−291.402e - 297.012e−297.012e - 293.272e−293.272e - 29
0.01−24.061-24.0612.796e−312.796e - 311.678e−301.678e - 309.787e−319.787e - 31
0.05−15.114-15.1141.070e−211.070e - 213.209e−213.209e - 211.498e−211.498e - 21
0.1−14.911-14.9112.022e−192.022e - 194.044e−194.044e - 192.359e−192.359e - 19
0.25−14.000-14.0001.964e−141.964e - 141.964e−141.964e - 141.964e−141.964e - 14
0.5−32.882-32.8825.131e−265.131e - 262.053e−252.053e - 258.980e−268.980e - 26
1N/AN/AN/AN/A

Table B.33. Brunner-Munzel tests and adjusted p-values for PINN and PINN/FEM results.

3D Taylor-Green Vortex, without outliers

NoisePINNFEMVargha–Delaney AA
00.032690 ± 0.0095980.001923 ± 0.0000191.000
0.0010.039061 ± 0.0098500.001920 ± 0.0000011.000
0.010.036553 ± 0.0105520.001924 ± 0.0000181.000
0.050.036072 ± 0.0115410.001919 ± 0.0000011.000
0.10.036208 ± 0.0106800.001922 ± 0.0000061.000
0.250.037936 ± 0.0100590.001935 ± 0.0000221.000
0.50.040479 ± 0.0088370.001996 ± 0.0001111.000
10.051235 ± 0.0041870.002609 ± 0.0013271.000

Table B.34. PINN and FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0N/AN/AN/AN/A
0.001N/AN/AN/AN/A
0.01N/AN/AN/AN/A
0.05N/AN/AN/AN/A
0.1N/AN/AN/AN/A
0.25N/AN/AN/AN/A
0.5N/AN/AN/AN/A
1N/AN/AN/AN/A

Table B.35. Brunner-Munzel tests and adjusted p-values for PINN and FEM results.

NoiseFEMPINN/FEMVargha–Delaney AA
00.001923 ±\pm 0.0000190.006938 ±\pm 0.0061730.000
0.0010.001920 ±\pm 0.0000010.012048 ±\pm 0.0092030.000
0.010.001924 ±\pm 0.0000180.009024 ±\pm 0.0075700.036
0.050.001919 ±\pm 0.0000010.011518 ±\pm 0.0100640.000
0.10.001922 ±\pm 0.0000060.010777 ±\pm 0.0084860.000
0.250.001935 ±\pm 0.0000220.013890 ±\pm 0.0228050.033
0.50.001996 ±\pm 0.0001110.012939 ±\pm 0.0094450.041
10.002609 ±\pm 0.0013270.010349 ±\pm 0.0097720.155

Table B.36. FEM and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0N/AN/AN/AN/A
0.001N/AN/AN/AN/A
0.0113.8681.880e −- 147.518e −- 143.927e −- 14
0.05N/AN/AN/AN/A
0.1N/AN/AN/AN/A
0.2514.0001.964e −- 147.518e −- 143.927e −- 14
0.513.4734.124e −- 148.247e −- 145.498e −- 14
16.5462.526e −- 082.526e −- 082.526e −- 08

Table B.37. Brunner-Munzel tests and adjusted p-values for FEM and PINN/FEM results.

NoisePINNPINN/FEMVargha–Delaney AA
00.032690 ±\pm 0.0095980.006938 ±\pm 0.0061730.982
0.0010.039061 ±\pm 0.0098500.012048 ±\pm 0.0092030.969
0.010.036553 ±\pm 0.0105520.009024 ±\pm 0.0075700.964
0.050.036072 ±\pm 0.0115410.011518 ±\pm 0.0100640.940
0.10.036208 ±\pm 0.0106800.010777 ±\pm 0.0084860.939
0.250.037936 ±\pm 0.0100590.013890 ±\pm 0.0228050.967
0.50.040479 ±\pm 0.0088370.012939 ±\pm 0.0094450.985
10.051235 ±\pm 0.0041870.010349 ±\pm 0.0097721.000

Table B.38. PINN and PINN/FEM results with Vargha-Delaney A effect size.

NoiseBrunner–Munzel WWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0−40.5133.069e−443.069e-442.149e−432.149e-432.149e−432.149e-43
0.001−24.2091.402e−291.402e-297.012e−297.012e-293.272e−293.272e-29
0.01−24.0612.796e−312.796e-311.678e−301.678e-309.787e−319.787e-31
0.05−15.1141.070e−211.070e-213.209e−213.209e-211.498e−211.498e-21
0.1−14.9112.022e−192.022e-194.044e−194.044e-192.359e−192.359e-19
0.25−14.0001.964e−141.964e-141.964e−141.964e-141.964e−141.964e-14
0.5−31.7711.608e−251.608e-256.434e−256.434e-252.815e−252.815e-25
1N/AN/AN/AN/A

Table B.39. Brunner-Munzel tests and adjusted p-values for PINN and PINN/FEM results.

Parameter results

In this section we report the mean and median bias of each model. The bias is defined as the deviation between the parameter predicted by each model and the ground truth parameter. Bootstrap 95% confidence intervals [56] are also reported for each bias value as well. We characterize the distribution of the bias by reporting IQR, skewness and minimum/maximum values. To determine whether the bias deviates from 0 in a statistically significant manner, we used a two-sided Wilcoxon signed-rank test [57] as the distributions of the biases are substantially skewed, in particular at high noise levels. The resulting pp-values are reported both in the original form and adjusted using the Holm family-wise error rate correction [54] as well as the Benjamini-Hochberg false discovery rate correction [55]. The Holm-based pp values below 0.05% are shown in bold. Note that for the deterministic FEM results for 1D Burgers’ equation with σ=0\sigma= 0, skewness is undefined and the Wilcoxon test is uninformative. In these cases, we report the results as N/A.

1D Burgers’ equation

Ground truth value for unknown parameter in this problem is 0.01/π0.01/\pi.

NoiseModelMean BiasMedian Bias95% Bootstrap CI for Bias
0PINN−0.000098−0.000115[−0.000115, −0.000081]
0.5PINN−0.000342−0.000256[−0.000670, 0.000002]
1PINN−0.000394−0.001424[−0.001181, 0.000473]
2PINN−0.000536−0.002193[−0.001672, 0.000756]
3PINN−0.000644−0.002362[−0.002166, 0.001495]
5PINN0.000348−0.002549[−0.002431, 0.004670]
7PINN−0.001204−0.002519[−0.002468, 0.000703]
10PINN0.360167−0.001840[0.030164, 0.854303]
25PINN25.4459836.099766[9.407605, 49.872131]
0FEM0.0001310.000131[0.000131, 0.000131]
0.5FEM0.0004250.000489[0.000112, 0.000760]
1FEM0.000276−0.000314[−0.000482, 0.001151]
2FEM0.000763−0.001043[−0.000637, 0.002663]
3FEM0.0066580.002247[0.002790, 0.010983]
5FEM0.1508660.002336[0.009093, 0.359999]
7FEM0.3710690.007848[0.020457, 0.811692]
10FEM0.4879350.008563[0.075662, 1.016190]
25FEM2.0029000.254178[1.200200, 2.825378]

Table C.40. Bias statistics for 1D Burgers’ equation tests.

NoiseModelIQRSkewnessMinimumMaximum
0PINN0.0000500.928−0.0001880.000038
0.5PINN0.0014400.589−0.0018410.002329
1PINN0.0021271.410−0.0027220.005951
2PINN0.0022752.044−0.0028820.010696
3PINN0.0007903.815−0.0028740.023615
5PINN0.0005584.101−0.0028970.048962
7PINN0.0007834.262−0.0031130.021295
10PINN0.0285014.200−0.0030226.106257
25PINN20.2022634.443−0.003014322.680601
0FEM0.000000N/A0.0001310.000131
0.5FEM0.0012050.827−0.0011990.003356
1FEM0.0024421.824−0.0027890.008702
2FEM0.0032053.426−0.0024920.021940
3FEM0.0117531.847−0.0031010.043822
5FEM0.0291213.654−0.0031472.155088
7FEM0.0465793.314−0.0031624.996817
10FEM0.1367853.025−0.0031504.996817
25FEM4.9807150.463−0.0031854.996817

Table C.41. Characteristics of distribution for 1D Burgers’ equation test errors.

NoiseModelWWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0PINN5.01.863e−081.863e - 083.166e−073.166e - 071.583e−071.583e - 07
0.5PINN138.00.05230.26130.0683
1PINN170.00.20540.79230.2328
2PINN126.00.02770.16643.930e−023.930e - 02
3PINN96.00.00404.032e−024.032e - 028.568e−038.568e - 03
5PINN59.01.529e−041.529e - 042.140e−032.140e - 036.412e−046.412e - 04
7PINN61.01.886e−041.886e - 042.452e−032.452e - 036.412e−046.412e - 04
10PINN169.00.19810.79230.2328
25PINN10.08.009e−088.009e - 081.201e−061.201e - 064.539e−074.539e - 07
0FEMN/AN/AN/AN/A
0.5FEM116.00.01550.10822.389e−022.389e - 02
1FEM223.00.85531.00000.8553
2FEM215.00.73031.00000.7760
3FEM110.00.01060.08941.802e−021.802e - 02
5FEM109.00.00990.08941.802e−021.802e - 02
7FEM71.05.055e−045.055e - 046.066e−036.066e - 031.432e−031.432e - 03
10FEM73.06.084e−046.084e - 046.692e−036.692e - 031.478e−031.478e - 03
25FEM5.01.863e−081.863e - 083.166e−073.166e - 071.583e−071.583e - 07

Table C.42. Wilcoxon signed-rank test statistics for 1D Burgers’ equation.

1D Burgers' equation with two parameters

Ground truth value for viscosity parameter in this problem is 0.01/π0.01/\pi.

Ground truth value for advection parameter in this problem is 0.55.

NoiseModelMean BiasMedian Bias95% Bootstrap CI for Bias
0PINN0.0001930.000198[0.000181, 0.000206]
0.5PINN−0.000583−0.000531[−0.000947, −0.000226]
1PINN−0.001792−0.002553[−0.002328, −0.001170]
2PINN−0.002821−0.003034[−0.003021, −0.002507]
3PINN−0.002983−0.003060[−0.003058, −0.002897]
5PINN−0.000553−0.003103[−0.003084, 0.004446]
7PINN0.061586−0.003117[−0.003069, 0.186921]
10PINN0.256946−0.002925[0.009165, 0.617358]
25PINN9.4942771.600279[3.282049, 17.611974]
0FEM0.0001520.000152[0.000152, 0.000152]
0.5FEM0.000162−0.000221[−0.000148, 0.000493]
1FEM0.0008240.000817[0.000287, 0.001370]
2FEM−0.000152−0.001336[−0.001145, 0.000951]
3FEM0.0018380.000419[0.000374, 0.003564]
5FEM0.102238−0.000375[0.002358, 0.297202]
7FEM0.562331−0.000480[0.077626, 1.136288]
10FEM1.1829310.002623[0.511359, 1.932753]
25FEM1.2687330.005378[0.566791, 2.052807]

Table C.43. Bias statistics for viscosity prediction in 1D Burgers’ equation tests.

NoiseModelIQRSkewnessMinimumMaximum
0PINN0.000060−0.1160.0001340.000247
0.5PINN0.001390−0.360−0.0027450.001270
1PINN0.0014502.036−0.0031820.003860
2PINN0.0003924.604−0.0031830.001037
3PINN0.0002871.605−0.003183−0.002260
5PINN0.0002665.475−0.0031830.071309
7PINN0.0002635.469−0.0031831.856309
10PINN0.0059343.645−0.0031833.807769
25PINN5.7154383.112−0.00318394.327661
0FEM0.000000N/A0.0001520.000152
0.5FEM0.0010130.987−0.0011840.002327
1FEM0.0023020.289−0.0017830.003989
2FEM0.0040601.225−0.0029870.008324
3FEM0.0052261.853−0.0028940.018063
5FEM0.0158985.470−0.0033462.883208
7FEM0.0142152.555−0.0033504.996817
10FEM2.2894491.209−0.0033434.996817
25FEM1.4942291.246−0.0033554.996817

Table C.44. Characteristics of distribution for viscosity prediction in 1D Burgers’ equation.

NoiseModelWWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0PINN0.01.863e−091.863e-093.166e−083.166e-081.583e−081.583e-08
0.5PINN103.00.00660.05981.254e−021.254e-02
1PINN45.03.050e−053.050e-053.659e−043.659e-048.640e−058.640e-05
2PINN1.03.725e−093.725e-095.588e−085.588e-082.111e−082.111e-08
3PINN0.01.863e−091.863e-093.166e−083.166e-081.583e−081.583e-08
5PINN30.03.790e−063.790e-064.928e−054.928e-051.289e−051.289e-05
7PINN87.00.00202.020e−022.020e-024.293e−034.293e-03
10PINN199.00.50281.00000.5342
25PINN27.02.349e−062.349e-063.288e−053.288e-059.982e−069.982e-06
0FEMN/AN/AN/AN/A
0.5FEM219.00.79221.00000.7922
1FEM107.00.00870.06961.480e−021.480e-02
2FEM191.00.40451.00000.4584
3FEM155.00.11420.68510.1618
5FEM178.00.27101.00000.3291
7FEM173.00.22861.00000.2989
10FEM116.00.01550.10822.389e−022.389e-02
25FEM85.00.00171.890e−021.890e-024.174e−034.174e-03

Table C.45. Wilcoxon signed-rank tests for viscosity prediction in 1D Burgers’ equation.

NoiseModelMean BiasMedian Bias95% Bootstrap CI for Bias
0PINN0.0021670.002157[0.002081, 0.002256]
0.5PINN−0.0054460.001737[−0.014797, 0.003965]
1PINN−0.038576−0.040225[−0.059253, −0.018518]
2PINN−0.080095−0.059642[−0.115137, −0.046315]
3PINN−0.111381−0.106573[−0.173821, −0.051607]
5PINN−0.217104−0.199926[−0.277550, −0.157503]
7PINN−0.231970−0.256692[−0.314987, −0.148053]
10PINN0.083602−0.322993[−0.299589, 0.727331]
25PINN0.742806−0.085778[0.031044, 1.713769]
0FEM0.0012910.001291[0.001291, 0.001291]
0.5FEM−0.004102−0.006705[−0.012382, 0.004651]
1FEM0.0094550.001445[−0.005047, 0.025132]
2FEM−0.002922−0.006106[−0.034036, 0.028089]
3FEM−0.023035−0.038868[−0.068635, 0.022375]
5FEM0.027904−0.047465[−0.056421, 0.114937]
7FEM−0.017300−0.059742[−0.109609, 0.075631]
10FEM0.0073440.007008[−0.134890, 0.144440]
25FEM−0.046517−0.007033[−0.192782, 0.101437]

Table C.46. Bias statistics for advection prediction in 1D Burgers’ equation tests.

NoiseModelIQRSkewnessMinimumMaximum
0PINN0.0004390.0360.0017030.002567
0.5PINN0.0384670.064−0.0527010.057314
1PINN0.071762−0.135−0.1533870.079692
2PINN0.115984−0.403−0.2583590.068890
3PINN0.132001−0.376−0.5406910.177078
5PINN0.190839−0.605−0.5500000.050067
7PINN0.4135680.201−0.5500000.203899
10PINN0.4545975.029−0.5500008.383439
25PINN1.1742513.109−0.55000010.835713
0FEM0.000000N/A0.0012910.001291
0.5FEM0.0207290.996−0.0499990.063621
1FEM0.0633690.543−0.0515690.105989
2FEM0.1111490.183−0.1586310.171450
3FEM0.170158−0.113−0.3496690.268005
5FEM0.3656960.558−0.3372370.450000
7FEM0.280093−0.057−0.5500000.450000
10FEM0.809269−0.266−0.5500000.450000
25FEM0.991587−0.040−0.5500000.450000

Table C.47. Distribution characteristics for advection prediction in 1D Burgers’ equation.

NoiseModelWWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0PINN0.01.863e−091.863e - 093.166e−083.166e - 083.166e−083.166e - 08
0.5PINN184.00.32851.00000.5584
1PINN83.00.00151.895e−021.895e - 024.955e−034.955e - 03
2PINN56.01.106e−041.106e - 041.548e−031.548e - 034.700e−044.700e - 04
3PINN94.00.00354.170e−024.170e - 029.846e−039.846e - 03
5PINN8.04.657e−084.657e - 087.451e−077.451e - 073.958e−073.958e - 07
7PINN45.03.050e−053.050e - 054.574e−044.574e - 041.728e−041.728e - 04
10PINN106.00.00810.08961.978e−021.978e - 02
25PINN209.00.64081.00000.7781
0FEMN/AN/AN/AN/A
0.5FEM157.00.12411.00000.2638
1FEM191.00.40451.00000.5730
2FEM223.00.85531.00000.8553
3FEM189.00.38181.00000.5730
5FEM220.00.80781.00000.8553
7FEM205.00.58381.00000.7634
10FEM221.00.82361.00000.8553
25FEM177.00.26211.00000.4951

Table C.48. Wilcoxon signed-rank tests for advection prediction in 1D Burgers’ equation.

2D Taylor-Green Vortex

Ground truth value for unknown parameter in this problem is 0.1.

NoiseModelMean BiasMedian Bias95% Bootstrap CI for Bias
0PINN−0.0005580.000026[−0.001053, −0.000148]
0.001PINN−0.000524−0.000053[−0.001027, −0.000145]
0.01PINN−0.000184−0.000039[−0.000496, 0.000041]
0.05PINN−0.000889−0.000776[−0.001333, −0.000462]
0.1PINN−0.002075−0.002093[−0.002659, −0.001515]
0.25PINN−0.005316−0.005385[−0.006187, −0.004425]
0.5PINN−0.008795−0.007857[−0.009940, −0.007670]
1PINN−0.009789−0.009772[−0.012159, −0.007637]
0FEM0.0001040.000104[0.000104, 0.000104]
0.001FEM0.0001040.000104[0.000103, 0.000104]
0.01FEM0.0000990.000099[0.000092, 0.000106]
0.05FEM0.0000840.000065[0.000041, 0.000126]
0.1FEM0.0001700.000208[0.000101, 0.000237]
0.25FEM0.0001170.000077[−0.000069, 0.000304]
0.5FEM0.0000870.000007[−0.000284, 0.000469]
1FEM0.000120−0.000161[−0.000419, 0.000667]

Table C.49. Bias statistics for viscosity prediction in 2D Taylor-Green Vortex tests.

NoiseModelIQRSkewnessMinimumMaximum
0PINN0.000214−2.142-2.142−0.004590-0.0045900.000106
0.001PINN0.000161−2.803-2.803−0.005437-0.0054370.000245
0.01PINN0.000244−3.614-3.614−0.003713-0.0037130.000770
0.05PINN0.001946−0.077-0.077−0.002847-0.0028470.001184
0.1PINN0.002040−0.125-0.125−0.005230-0.0052300.001228
0.25PINN0.0027740.181−0.010830-0.0108300.000586
0.5PINN0.004669−0.353-0.353−0.015567-0.015567−0.003149-0.003149
1PINN0.006672−1.402-1.402−0.028845-0.028845−0.000376-0.000376
0FEM0.000000−0.187-0.1870.0001030.000104
0.001FEM0.0000020.8160.0001010.000108
0.01FEM0.000018−0.331-0.3310.0000440.000142
0.05FEM0.0001620.006−0.000164-0.0001640.000283
0.1FEM0.000229−0.413-0.413−0.000343-0.0003430.000570
0.25FEM0.0009980.056−0.000906-0.0009060.000950
0.5FEM0.0016260.389−0.001635-0.0016350.002386
1FEM0.0024150.416−0.002274-0.0022740.002966

Table C.50. Distribution characteristics for viscosity prediction in 2D Taylor-Green vortex.

NoiseModelWWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0PINN189.00.38181.00000.4699
0.001PINN112.00.01200.07231.752e−021.752e-02
0.01PINN200.00.51581.00000.5895
0.05PINN79.00.00107.267e−037.267e-031.661e−031.661e-03
0.1PINN16.03.148e−073.148e-073.148e−063.148e-067.195e−077.195e-07
0.25PINN1.03.725e−093.725e-094.098e−084.098e-089.934e−099.934e-09
0.5PINN0.01.863e−091.863e-092.980e−082.980e-085.960e−095.960e-09
1PINN0.01.863e−091.863e-092.980e−082.980e-085.960e−095.960e-09
0FEM0.01.863e−091.863e-092.980e−082.980e-085.960e−095.960e-09
0.001FEM0.01.863e−091.863e-092.980e−082.980e-085.960e−095.960e-09
0.01FEM0.01.863e−091.863e-092.980e−082.980e-085.960e−095.960e-09
0.05FEM72.05.548e−045.548e-044.439e−034.439e-039.864e−049.864e-04
0.1FEM51.06.287e−056.287e-055.658e−045.658e-041.257e−041.257e-04
0.25FEM184.00.32851.00000.4380
0.5FEM217.00.76111.00000.8118
1FEM224.00.87121.00000.8712

Table C.51. Wilcoxon signed-rank tests for viscosity prediction in 2D Taylor-Green Vortex.

3D Taylor-Green Vortex, with outliers

Ground truth value for unknown parameter in this problem is 0.01.

NoiseModelMean BiasMedian Bias95% Bootstrap CI for Bias
0PINN0.0006350.001489[−0.000784, 0.001951]
0.001PINN−0.001079−0.000067[−0.002878, 0.000755]
0.01PINN0.0005410.000964[−0.001189, 0.002187]
0.05PINN−0.001477−0.000084[−0.003103, 0.000115]
0.1PINN−0.001392−0.000025[−0.003050, 0.000192]
0.25PINN0.0036270.001124[−0.002388, 0.013921]
0.5PINN−0.000971−0.001744[−0.003294, 0.001756]
1PINN−0.002776−0.002551[−0.003954, −0.001583]
0FEM−0.0000410.000294[−0.000603, 0.000295]
0.001FEM0.1663390.000294[−0.000220, 0.499160]
0.01FEM0.4939190.000293[−0.000174, 0.996085]
0.05FEM0.1651540.000289[−0.001825, 0.498047]
0.1FEM0.1656840.000313[−0.001089, 0.498493]
0.25FEM0.3292660.000243[−0.000306, 0.824930]
0.5FEM−0.0004250.000278[−0.001250, 0.000238]
1FEM0.1654200.000112[−0.001460, 0.498441]

Table C.52. Bias statistics for viscosity prediction in 3D Taylor-Green Vortex tests.

NoiseModelIQRSkewnessMinimumMaximum
0PINN0.004957−0.759−0.0068580.006629
0.001PINN0.0097340.065−0.0068580.008105
0.01PINN0.005261−0.349−0.0068580.009968
0.05PINN0.009177−0.185−0.0068580.004507
0.1PINN0.009413−0.173−0.0068580.005126
0.25PINN0.0095515.233−0.0068580.141775
0.5PINN0.0090391.981−0.0068580.026158
1PINN0.0066020.210−0.0068580.004680
0FEM0.000000−4.316−0.0064750.000316
0.001FEM0.0000015.477−0.0037514.988961
0.01FEM0.0000082.810−0.0068584.987737
0.05FEM0.0000545.477−0.0068584.985034
0.1FEM0.0000935.477−0.0068584.985623
0.25FEM0.0001733.661−0.0050904.989280
0.5FEM0.000429−2.445−0.0068580.001019
1FEM0.0015265.477−0.0068584.988669

Table C.53. Distribution characteristics for viscosity prediction in 3D Taylor-Green vortex.

NoiseModelWWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0PINN173.00.22861.00000.2813
0.001PINN167.00.18401.00000.2453
0.01PINN192.00.41611.00000.4756
0.05PINN160.00.14031.00000.2209
0.1PINN162.00.15191.00000.2209
0.25PINN205.00.58381.00000.5838
0.5PINN155.00.11421.00000.2209
1PINN65.02.833e−042.833e-043.682e−033.682e-031.133e−031.133e-03
0FEM59.01.529e−041.529e-042.140e−032.140e-038.153e−048.153e-04
0.001FEM57.01.233e−041.233e-041.850e−031.850e-038.153e−048.153e-04
0.01FEM78.09.518e−049.518e-041.142e−021.142e-023.046e−033.046e-03
0.05FEM159.00.13471.00000.2209
0.1FEM135.00.04490.49400.1198
0.25FEM56.01.106e−041.106e-041.769e−031.769e-038.153e−048.153e-04
0.5FEM144.00.06990.69890.1598
1FEM204.00.56991.00000.5838

Table C.54. Wilcoxon signed-rank tests for viscosity prediction in 3D Taylor-Green Vortex.

3D Taylor-Green Vortex, without outliers

Ground truth value for unknown parameter in this problem is 0.01.

NoiseModelMean BiasMedian Bias95% Bootstrap CI for Bias
0PINN0.0006350.001489[−0.000784, 0.001951]
0.001PINN−0.001079−0.000067[−0.002878, 0.000755]
0.01PINN0.0005410.000964[−0.001189, 0.002187]
0.05PINN−0.001477−0.000084[−0.003103, 0.000115]
0.1PINN−0.001392−0.000025[−0.003050, 0.000192]
0.25PINN−0.0011370.001096[−0.002790, 0.000444]
0.5PINN−0.001906−0.002592[−0.003642, −0.000099]
1PINN−0.002776−0.002551[−0.003955, −0.001550]
0FEM0.0002940.000294[0.000294, 0.000294]
0.001FEM0.0002940.000294[0.000293, 0.000294]
0.01FEM0.0002940.000294[0.000292, 0.000296]
0.05FEM0.0002980.000298[0.000289, 0.000307]
0.1FEM0.0003120.000329[0.000291, 0.000331]
0.25FEM0.0002460.000243[0.000198, 0.000294]
0.5FEM0.0003500.000328[0.000217, 0.000474]
1FEM−0.0002060.000133[−0.000704, 0.000229]

Table C.55. Bias statistics for viscosity prediction in 3D Taylor-Green Vortex tests.

NoiseModelIQRSkewnessMinimumMaximum
0PINN0.004957−0.759−0.0068580.006629
0.001PINN0.0097340.065−0.0068580.008105
0.01PINN0.005261−0.349−0.0068580.009968
0.05PINN0.009177−0.185−0.0068580.004507
0.1PINN0.009413−0.173−0.0068580.005126
0.25PINN0.009237−0.319−0.0068580.005572
0.5PINN0.0090020.411−0.0068580.009137
1PINN0.0066020.210−0.0068580.004680
0FEM0.0000000.6580.0002940.000294
0.001FEM0.000001−0.4740.0002920.000295
0.01FEM0.0000050.2980.0002830.000307
0.05FEM0.000031−0.2640.0002550.000334
0.1FEM0.000074−0.2000.0002160.000423
0.25FEM0.0001050.141−0.0000240.000554
0.5FEM0.000356−0.336−0.0004170.001019
1FEM0.000897−1.577−0.0032650.001259

Table C.56. Distribution characteristics for viscosity prediction in 3D Taylor-Green vortex.

NoiseModelWWpppHolmp_{\mathrm{Holm}}pFDRp_{\mathrm{FDR}}
0PINN173.00.22860.98200.2813
0.001PINN167.00.18400.98200.2453
0.01PINN192.00.41611.00000.4439
0.05PINN160.00.14030.98200.2209
0.1PINN162.00.15190.98200.2209
0.25PINN175.00.36921.00000.4220
0.5PINN125.00.04550.36420.0809
1PINN65.02.833e−042.833e-042.549e−03\mathbf{2.549e-03}5.665e−045.665e-04
0FEM0.09.537e−079.537e-071.049e−05\mathbf{1.049e-05}2.543e−062.543e-06
0.001FEM0.05.960e−085.960e-089.537e−07\mathbf{9.537e-07}4.768e−074.768e-07
0.01FEM0.04.768e−074.768e-075.722e−06\mathbf{5.722e-06}1.526e−061.526e-06
0.05FEM0.02.384e−072.384e-073.099e−06\mathbf{3.099e-06}9.537e−079.537e-07
0.1FEM0.01.192e−071.192e-071.669e−06\mathbf{1.669e-06}6.358e−076.358e-07
0.25FEM1.05.960e−085.960e-089.537e−07\mathbf{9.537e-07}4.768e−074.768e-07
0.5FEM30.06.032e−056.032e-056.032e−04\mathbf{6.032e-04}1.379e−041.379e-04
1FEM151.00.77121.00000.7712

Table C.57. Wilcoxon signed-rank tests for viscosity prediction in 3D Taylor-Green Vortex.

1D Burgers' equation with two unknown parameters

We also tested 1D Burgers’ equation with two parameters.

∂u(x,t)∂t+a⋅u(x,t)⋅∇u(x,t)=ν∇2u(x,t),x∈Ω⊂Rn,t∈[0,∞),u∈Rn.(29)\frac{\partial\mathbf{u}(x,t)}{\partial t} + a \cdot\mathbf{u}(x,t) \cdot\nabla\mathbf{u}(x,t) = \nu\nabla^{2}\mathbf{u}(x,t), \qquad x \in\Omega\subset\mathbb{R}^{n}, \quad t \in[0,\infty), \quad\mathbf{u} \in\mathbb{R}^{n}. \tag*{(29)}

The parameter added was an advection parameter, aa, which scales the advection. These tests may give insight in how the behavior of the models is affected when identifying multiple parameters. The ground truth data was created from an analytical solution using the Cole-Hopf transformation. To be consistent with the scaling, the equation was reformulated in terms of the rescaled variable v=auv = au. After applying the standard Cole-Hopf transform to vv, the physical velocity field was recovered using u=v/au = v/a. The ground truth solutions for the unknown parameters were a=0.55a = 0.55 and ν=0.01/π\nu= 0.01/\pi.

The ranking between the models were very similar as with one parameter, however overall all the models performed slightly worse. Compared to the Navier-Stokes experiments, the standard 1D Burgers’ equation tests showed less of the biased parameter learning and exploding physics loss that we saw in the more complex problems. However, in the 1D Burgers’ equation tests with two unknown parameters, these phenomena were more pronounced.

Results for 1D Burgers’ equation with two unknown parameters

Figure D.7. Results for 1D Burgers’ equation with two unknown parameters on test set based on 30 runs of each model. Standard deviation (σ\sigma) is the noise added to the training data. Top left: Prediction accuracies, σ∈[0,2]\sigma\in[0, 2]. Top right: Prediction accuracies, σ∈[3,25]\sigma\in[3, 25]. Center left: Viscosity identification accuracies, σ∈[0,3]\sigma\in[0, 3]. Center right: Viscosity identification accuracies, σ∈[5,10]\sigma\in[5, 10]. Bottom left: Advection identification accuracies, σ∈[0,3]\sigma\in[0, 3]. Bottom right: Advection identification accuracies, σ∈[5,25]\sigma\in[5, 25]

Predictions graphs in MSE

1D Burgers' equation

Figure E.8. Results for 1D Burgers’ equation on test set based on 30 runs of each model. Left: Prediction accuracies, σ∈[0,2]\sigma\in[0,2]. Right: Prediction accuracies, σ∈[3,25]\sigma\in[3,25].

1D Burgers' equation with two parameters

Figure E.9. Results for 1D Burgers’ equation with two parameters on test set based on 30 runs of each model. Left: Prediction accuracies, σ∈[0,2]\sigma\in[0,2]. Right: Prediction accuracies, σ∈[3,25]\sigma\in[3,25].

2D Taylor-Green Vortex

Figure E.10. Results for 2D Taylor-Green Vortex on test set based on 30 runs of each model. Left: Prediction accuracies, σ∈[0,0.05]\sigma\in[0,0.05]. Right: Prediction accuracies, σ∈[0.05,1]\sigma\in[0.05,1].

3D Taylor-Green Vortex

Image not extracted

(a)

Image not extracted

(b)

Figure E.11. Results for 3D Taylor-Green Vortex on test set based on 30 runs of each model. Left: Prediction accuracies with outliers excluded. Right: Prediction accuracies with outliers included.

Noise levels tested

1D Burgers’ equation

σ\sigmaSNRSNR (dB)
0∞\infty—
0.51.50481.77
10.3762-4.25
20.0940-10.27
30.0418-13.79
50.0150-18.23
70.0077-21.15
100.0038-24.25
250.0006-32.20

Table 2.

1D Burgers’ equation with two parameters

σ\sigmaSNRSNR (dB)
0∞\infty—
0.51.84762.67
10.4619-3.35
20.1155-9.38
30.0513-12.90
50.0185-17.33
70.0094-20.26
100.0046-23.35
250.0007-31.31

Table 3.

2D Taylor-Green Vortex

σ\sigmaSNRSNR (dB)
0∞\infty—
0.001114885.259650.60
0.011148.852630.60
0.0545.954116.62
0.111.488510.60
0.251.83822.64
0.50.4595-3.38
10.1149-9.40

Table 4.

3D Taylor-Green Vortex

σ\sigmaSNRSNR (dB)
0∞\infty—
0.00169943.582248.45
0.01699.435828.45
0.0527.977414.47
0.16.99448.45
0.251.11910.49
0.50.2798-5.53
10.0699-11.55

Table 1.

Additional graph

Image not extracted

Figure 1 (PDF p. 76). 1D Burger's equation, parameter discovered

References

  1. [1]L. Evans, Partial Differential Equations, Graduate studies in mathematics, 2 ed., American Mathematical Society, 2010.
  2. [2]V. Ryaben’kii, et al., A Theoretical Introduction to Numerical Analysis, CRC Press, 2006.
  3. [3]C. R. Vogel, Computational Methods for Inverse Problems, volume 23 of Frontiers in Applied Mathematics, SIAM, 2002.DOI
  4. [4]M. Raissi, et al., Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378 (2019) 686–707.DOI
  5. [5]Z. Miao, Y. Chen, Vc-pinn: Variable coefficient physics-informed neural network for forward and inverse problems of pdes with variable coefficient, Physica D: Nonlinear Phenomena 456 (2023) 133945. URL: https://www.sciencedirect.com/science/article/pii/S0167278923002993. doi:https://doi.org/10.1016/j.physd.2023.133945.
  6. [6]S. Thakur, E. Esmaili, S. Libring, L. Solorio, A. Ardekani, Inverse resolution of spatially varying diffusion coefficient using physics-informed neural networks, Physics of Fluids 36 (2024). doi:10.1063/5.0207453.DOI
  7. [7]S. Cuomo, et al., Scientific machine learning through physics-informed neural networks: Where we are and what’s next, 2022. URL: https://arxiv.org/abs/2201.05624. arXiv:2201.05624.
  8. [8]N. McGreivy, et al., Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations, 2024. URL: https://arxiv.org/abs/2407.07218. arXiv:2407.07218.
  9. [9]G. Karniadakis, et al., Physics-informed machine learning, Nature Reviews Physics 3 (2021) 422–440. doi:10.1038/s42254-021-00314-5.DOI
  10. [10]T. G. Grossmann, et al., Can physics-informed neural networks beat the finite element method?, 2023. URL: https://arxiv.org/abs/2302.04107. arXiv:2302.04107.
  11. [11]F. Fernández de la Mata, et al., Physics-informed neural networks for data-driven simulation: Advantages, limitations, and opportunities, Physica A: Statistical Mechanics and its Applications 610 (2023). doi:10.1016/j.physa.2022.128415.DOI
  12. [12]P.-Y. Chuang, et al., Experience report of physics-informed neural networks in fluid simulations: pitfalls and frustration, 2022. URL: https://arxiv.org/abs/2205.14249. arXiv:2205.14249.
  13. [13]Z. Hu, et al., Tackling the curse of dimensionality with physics-informed neural networks, Neural Networks 176 (2024). doi:https://doi.org/10.1016/j.neunet.2024.106369.
  14. [14]X. Jin, et al., Nsf nets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations, Journal of Computational Physics 426 (2021) 109951. URL: https://www.sciencedirect.com/science/article/pii/S0021999120307257. doi:https://doi.org/10.1016/j.jcp.2020.109951.
  15. [15]M. Raissi, et al., Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science 367 (2020) 1026–1030. URL: https://www.science.org/doi/abs/10.1126/science.aaw4741. doi:10.1126/science.aaw4741.DOI
  16. [16]R. Sharma, et al., Inversion of dc resistivity data using physics-informed neural networks, 2024. URL: https://arxiv.org/abs/2408.02420. arXiv:2408.02420.
  17. [17]X. Yang, et al., A two-stage imaging framework combining cnn and physics-informed neural networks for full-inverse tomography: A case study in electrical impedance tomography (eit), 2025. URL: https://arxiv.org/abs/2407.17721. arXiv:2407.17721.
  18. [18]K. Yokota, et al., Identification of physical properties in acoustic tubes using physics-informed neural networks, 2024. URL: https://arxiv.org/abs/2406.11119. arXiv:2406.11119.
  19. [19]M. Berardi, et al., Inverse physics-informed neural networks for transport models in porous materials, 2024. URL: https://arxiv.org/abs/2407.10654. arXiv:2407.10654.
  20. [20]Y. Luo, et al., Cfdbench: A large-scale benchmark for machine learning methods in fluid dynamics, 2024. URL: https://arxiv.org/abs/2310.05963. arXiv:2310.05963.
  21. [21]Z. Hao, J. Yao, C. Su, H. Su, Z. Wang, F. Lu, Z. Xia, Y. Zhang, S. Liu, L. Lu, J. Zhu, Pinnacle: A comprehensive benchmark of physics-informed neural networks for solving pdes, 2023. URL: https://arxiv.org/abs/2306.08827. arXiv:2306.08827.
  22. [22]P. Karnakov, et al., Solving inverse problems in physics by optimizing a discrete loss: Fast and accurate learning without neural networks, PNAS Nexus 3 (2023). URL: http://dx.doi.org/10.1093/pnasnexus/pgae005. doi:10.1093/pnasnexus/pgae005.
  23. [23]S. Wang, S. Sankaran, H. Wang, P. Perdikaris, An expert’s guide to training physics-informed neural networks, 2023. URL: https://arxiv.org/abs/2308.08468. arXiv:2308.08468.
  24. [24]S. Wang, et al., Understanding and mitigating gradient pathologies in physics-informed neural networks, 2020. URL: https://arxiv.org/abs/2001.04536. arXiv:2001.04536.
  25. [25]E. Kharazmi, Z. Zhang, G. E. Karniadakis, hp-vpinns: Variational physics-informed neural networks with domain decomposition, Computer Methods in Applied Mechanics and Engineering 374 (2021) 113547. URL: http://dx.doi.org/10.1016/j.cma.2020.113547. doi:10.1016/j.cma.2020.113547.
  26. [26]M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, R. Ng, Fourier features let networks learn high frequency functions in low dimensional domains, 2020. URL: https://arxiv.org/abs/2006.10739. arXiv:2006.10739.
  27. [27]S. Wang, et al., Respecting causality is all you need for training physics-informed neural networks, 2022. URL: https://arxiv.org/abs/2203.07404. arXiv:2203.07404.
  28. [28]A. New, M. Villafañe-Delgado, C. Shugert, Equation identification for fluid flows via physics-informed neural networks, 2024. URL: https://arxiv.org/abs/2408.17271. arXiv:2408.17271.
  29. [29]M. Aliakbari, M. Soltany Sadrabadi, P. Vadasz, A. Arzani, Ensemble physics informed neural networks: A framework to improve inverse transport modeling in heterogeneous domains, Physics of Fluids 35 (2023) 053616. URL: https://doi.org/10.1063/5.0150016. doi:10.1063/5.0150016.
  30. [30]I. Goodfellow, et al., Deep Learning, MIT Press, 2016. http://www.deeplearningbook.org.
  31. [31]D. P. Kingma, et al., Adam: A method for stochastic optimization, 2017. URL: https://arxiv.org/abs/1412.6980. arXiv:1412.6980.
  32. [32]J. F. Urbán, P. Stefanou, J. A. Pons, Unveiling the optimization process of physics informed neural networks: How accurate and competitive can pinns be?, Journal of Computational Physics 523 (2025) 113656. URL: https://www.sciencedirect.com/science/article/pii/S0021999124009045. doi:https://doi.org/10.1016/j.jcp.2024.113656.
  33. [33]A. S. Lewis, S. Zhang, Nonsmoothness and the bfgs method, 2013. URL: https://people.orie.cornell.edu/aslewis/publications/sub-nonsmoothness.pdf, preprint, Cornell University.
  34. [34]R. Courant, Variational methods for the solution of problems of equilibrium and vibrations, Bulletin of the American Mathematical Society 49 (1943) 1–23.
  35. [35]P. Seshu, Textbook of Finite Element Analysis, PHI Learning, 2003.
  36. [36]W. K. Liu, et al., Eighty years of the finite element method: Birth, evolution, and future, Archives of Computational Methods in Engineering 29 (2022) 4431–4453. doi:10.1007/s11831-022-09740-9.DOI
  37. [37]D. Kraft, A Software Package for Sequential Quadratic Programming, Technical Report DFVLR-FB 88-28, DLR German Aerospace Center, Köln, Germany, 1988.
  38. [38]P. Virtanen, et al., SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods 17 (2020) 261–272.
  39. [39]S. K. Mitusch, et al., dolfin-adjoint 2018.1: automated adjoints for FEniCS and Firedrake, Journal of Open Source Software 4 (2019) 1292. doi:10.21105/joss.01292.DOI
  40. [40]dolfin-adjoint developers, Inverse parameter identification fails - how to use discrete measurements?, 2021. URL: https://github.com/dolfin-adjoint/dolfin-adjoint/issues/, gitHub issue #7.
  41. [41]A. Paszke, et al., Pytorch: An imperative style, high-performance deep learning library, 2019. URL: https://arxiv.org/abs/1912.01703.
  42. [42]Norwegian University of Science and Technology, Idun hpc cluster, https://www.hpc.ntnu.no/idun/, 2025. Accessed: 2025-08-25.
  43. [43]T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in: KDD ’19: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 2623–2631.arxiv.org/abs/1907.10902
  44. [44]C. D. Cantwell, et al., Nektar++: An open-source spectral/hp element framework, Computer Physics Communications 192 (2015) 205–219. URL: https://www.sciencedirect.com/science/article/pii/S0010465515000533.
  45. [45]Department of Aeronautics, Imperial College London, UK, et al., Quasi-3D Computation of the Taylor-Green Vortex Flow, 2024. URL: https://doc.nektar.info/tutorials/latest/incns/taylor-green-vortex/incns-taylor-green-vortex.pdf.
  46. [46]C. R. Harris, et al., Array programming with NumPy, Nature 585 (2020) 357–362. URL: https://doi.org/10.1038/s41586-020-2649-2. doi:10.1038/s41586-020-2649-2.
  47. [47]R. E. Walpole, R. H. Myers, S. L. Myers, K. Ye, Probability and Statistics for Engineers and Scientists, 9 ed., Pearson, 2016.
  48. [48]R. J. Hyndman, G. Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., OTexts, Melbourne, Australia, 2021. URL: https://OTexts.com/fpp3, accessed on 2026-02-13.
  49. [49]A. Irpan, Deep reinforcement learning doesn’t work yet, https://www.alexirpan.com/2018/02/14/rl-hard.html, 2018.
  50. [50]J. Taylor, et al., Optimizing the optimizer for data driven deep neural networks and physics informed neural networks, 2022. URL: https://arxiv.org/abs/2205.07430. arXiv:2205.07430.
  51. [51]A. J. Chorin, A numerical method for solving incompressible viscous flow problems, Journal of Computational Physics 2 (1967) 12–26. doi:https://doi.org/10.1016/0021-9991(67)90037-X.
  52. [52]A. Vargha, H. D. Delaney, A critique and improvement of the cl common language effect size statistics of mcgraw and wong, Journal of Educational and Behavioral Statistics 25 (2000) 101 – 132.
  53. [53]E. Brunner, U. Munzel, The nonparametric behrens-fisher problem: Asymptotic theory and a small-sample approximation, Biometrical Journal 42 (2000) 17–25.doi.org/10.1002/(sici)1521-4036(200001)42:1<17::aid-bimj17>3.0.co;2-u

Paper details

vlnc:W_cenjq4rh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents