betaFeedback

WorksMathematics

About rectified sigmoid function for enhancing the accuracy of Physics-Informed Neural Networks

Abstract

The article is devoted to the study of neural networks with one hidden layer and a modified activation function for solving physical problems. A rectified sigmoid activation function has been proposed to solve physical problems described by the ODE with neural networks. Algorithms for physics-informed data-driven initialization of a neural network and a neuron-by-neuron gradient-free fitting method have been presented for the neural network with this activation function. Numerical experiments demonstrate the superiority of neural networks with a rectified sigmoid function over neural networks with a sigmoid function in the accuracy of solving physical problems (harmonic oscillator, relativistic slingshot, and Lorentz system).

Introduction

The field of machine learning related to science and engineering [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] has seen significant advancements in recent decades, particularly with the integration of physical laws and principles into machine learning models, leading to the development of physics-informed machine learning techniques [3, 4, 5]. In the PINN (Physics-Informed Neural Network) approach, a neural network is trained to approximate the dependences of physical values on spatial and temporal variables for a given physical problem, described by a set of physical equations, together with additional constraints such as initial and boundary conditions. In some cases, data from numerical simulations or experiments can also be used to help train the network. PINNs have been used to solve a wide range of problems described by differential equations in various fields, including thermodynamics, hydrodynamics, mechanics, finance, systems biology, etc. [3, 9, 11, 12, 13, 14, 15, 16, 17, 18].

Most current works on the subject using the PINN approach for scientific problems employ small neural networks, with only a few layers, or specific architectures with many layers, which can mitigate the vanishing gradients problem [19, 20]. However, this issue reduces the representativeness and effectiveness of deep PINNs. In contrast, neural networks with one or two hidden layers avoid this drawback and, despite their simplicity, their potential has not yet been fully explored.

The past article [21] discusses the development of various methods and techniques for initialization and training neural networks with a single hidden layer, as well as the training of a separable physics-informed neural network consisting of neural networks with a single hidden layer to solve physical problems described by ordinary differential equations (ODEs) and partial differential equations (PDEs). Neural networks initialized and trained by the methods proposed in this article demonstrate outstanding accuracy of solutions (at a relatively short learning time) and generalizing properties for parameterized differential equations. In this article [21], the sigmoid function was taken as the activation function for neural networks. At the same time, we do not think that this activation function is the best for neural networks used to solve physical problems described by differential equations.

This work is devoted to studying neural networks with one hidden layer and modified activation function for solving physical problems. We will briefly list the contributions made in the paper:

  1. A rectified sigmoid function is proposed for solving physical problems described by the ODE;

  2. Algorithms for physics-informed data-driven (PIDD) initialization of a neural network and gradient-free neuron-by-neuron (NbN) fitting method for a neural network with one hidden layer has been modified with rectified sigmoid function as activation function;

  3. The accuracy of the ODE solutions of neural networks with a sigmoid activation function is compared with the accuracy of the ODE solutions provided by neural networks with a rectified sigmoid activation function.

The paper is structured as follows. In § 2, a rectified sigmoid function is proposed for solving physical problems described by the ODE. The algorithms of PIDD initialization of a neural network and NbN fitting method for a neural network with one hidden layer and rectified sigmoid function as activation function are presented. § 3 shows the results of numerical experiments. Finally, in § 4 concluding remarks are given.

State of the problem

We consider a neural network with a single hidden layer for solving problems which are described with ordinary differential equations (ODEs). Consider a system of nonlinear ordinary differential equations, which depend on coordinates xx, and in general take the following form

∂u∂x+N[u,x]=0,x∈[0,X](1)\frac{\partial\mathbf{u}}{\partial x}+\mathcal{N}[\mathbf{u},x]=0,\qquad x\in[0,X] \tag*{(1)}

under the initial conditions

u(0)=g,(2)\mathbf{u}(0)=\mathbf{g}, \tag*{(2)}

where u(x)\mathbf{u}(x) denotes the latent solution that is governed by the ODE system of equations (1) and consists of nn components u=(u1,u2,…,un)\mathbf{u}=(u_1,u_2,\ldots,u_n), N\mathcal{N} is a nonlinear differential operator, g\mathbf{g} is initial distribution of u\mathbf{u}.

According to PINN approach [3] we approximate the unknown solution u(x)\mathbf{u}(x) with neural networks uθ(x)\mathbf{u}_{\theta}(x), every of which the component uθ;l(x)u_{\theta;l}(x) is the separated neural network with a single hidden layer, as follows:

uθ;l(x)=∑k=0N−1Wk(2)σ(Wk(1)x+bk(1))+b0(2),(3)u_{\theta;l}(x)=\sum_{k=0}^{N-1}W_k^{(2)}\sigma\left(W_k^{(1)}x+b_k^{(1)}\right)+b_0^{(2)}, \tag*{(3)}

where Wk(j)W_k^{(j)} is kkth weight of the jjth layer, bk(j)b_k^{(j)} is kkth bias of the jjth layer, θ\boldsymbol{\theta} denote all trainable parameters (weights and biases) of the neural network uθ;lu_{\theta;l}, σ\sigma is an activation function, NN is number neurons in the hidden layer.

In the article [21], the sigmoid function was used as an activation function, because it is the most suitable among all the presented activation functions [22] for Euler’s method of integrating differential equations. In this paper, we propose using a new activation function, that we call “rectified sigmoid”, which corresponds better to Euler’s approach for integrating differential equations than the original sigmoid function. Rectified sigmoid function is defined as follows

Re ⁣- ⁣⁡σ(x)={0,x<−1,12(x+1),−1≤x≤1,1,x>1.(4)\operatorname{Re\!-\!}\sigma(x)= \begin{cases} 0, & x<-1,\\ \frac{1}{2}(x+1), & -1\le x\le1,\\ 1, & x>1. \end{cases} \tag*{(4)}

This function can be written as a combination of two rectified linear unit (ReLU [22]) functions as follows

Re⁡-σ(x)=12[ReLU⁡(x+1)−ReLU⁡(x−1)].(5)\operatorname{Re}\text{-}\sigma(x) = \frac{1}{2}\left[\operatorname{ReLU}(x+1)-\operatorname{ReLU}(x-1)\right]. \tag*{(5)}

Perhaps this circumstance is the reason why neural networks with the ReLU activation function demonstrate relatively good results in solving many practical problems [23].

To verify the accuracy of a neural network with an activation function Re-σ\sigma, we use the physics-informed data-driven (PIDD) initialization of neural networks algorithm proposed in paper [21], which is provided in the Appendix A for convenience as algorithm 3. We used this algorithm for a number of physical problems described by equations (1). Algorithm 3 was proposed for the sigmoid activation function. Some changes should be made to it when using the activation function Re-σ\sigma. Based on the behaviour of the activation function Re-σ\sigma, we must assume Δζ\Delta\zeta to be equal to 1. With this choice of Δζ\Delta\zeta, the derivatives of the outputs (Re-σ′(x)\sigma'(x)) of each neuron in the hidden layer do not contribute to those of neighbouring neurons in the same layer. Therefore, the values of κk\kappa_k in algorithm 3 are equal to 1.

Let us give an equation for calculating the weights Wm(2)W_m^{(2)}. The weights of the hidden layer have already been initialized using algorithm 3 based on the requirements given above. Find the derivative of the component of solution uθ;l(x)u_{\theta;l}(x) at the point with the coordinate xmx_m (xm=mΔxx_m=m\Delta x, where Δx=X/N\Delta x=X/N):

∂uθ;l(x)∂x∣x=xm=∑k=0N−12ΔζΔxWk(2)Re⁡-σ′(2ΔζΔx[xm−xk]).(6)\left.\frac{\partial u_{\theta;l}(x)}{\partial x}\right|_{x=x_m} = \sum_{k=0}^{N-1}\frac{2\Delta\zeta}{\Delta x}W_k^{(2)}\operatorname{Re}\text{-}\sigma'\left(\frac{2\Delta\zeta}{\Delta x}[x_m-x_k]\right). \tag*{(6)}

Here, Re-σ′(x)\sigma'(x) means a derivate on the argument of the function Re-σ(x)\sigma(x). Pay attention Re-σ′(0)=1/2\sigma'(0)=1/2. We have following equation of derivative of uθ;l(x)u_{\theta;l}(x)

∂uθ;l(x)∂x∣x=xm=ΔζΔxWm(2).(7)\left.\frac{\partial u_{\theta;l}(x)}{\partial x}\right|_{x=x_m} = \frac{\Delta\zeta}{\Delta x}W_m^{(2)}. \tag*{(7)}

On the other hand, derivative of uθ;l(x)u_{\theta;l}(x) must satisfy the Equations (1). Using (1) and (7) we have the following equation for the weights Wm(2)W_m^{(2)}

Wm(2)=−ΔxN[um,xm]/Δζ.(8)W_m^{(2)}=-\Delta x\mathcal{N}[u_m,x_m]/\Delta\zeta. \tag*{(8)}

Here values um\mathbf{u}_m is solution u\mathbf{u} in the point xmx_m, N[u,xm]\mathcal{N}[u,x_m] is ll component of the result of operator action N[u,xm]\mathcal{N}[u,x_m].

As a result, we have the following physics-informed data-driven (PIDD) initialization of neural networks with activation function Re-σ\sigma (Algorithm 1).

Algorithm 1: Physics-informed data-driven initialization of neural networks with activation function Re-σ\sigma
Data: {uk}k=0N−1\{\mathbf{u}_k\}_{k=0}^{N-1} for {xk}k=0N−1\{x_k\}_{k=0}^{N-1} (uniform grid with a step Δx←X/N\Delta x\leftarrow X/N)
Result: Initialized neural network uθ;l(x)u_{\theta;l}(x) of PINN uθ\mathbf{u}_{\theta}, which consists of NN neurons on hidden layer
Δζ←1\Delta\zeta\leftarrow1;
for k=0,…,N−1k=0,\ldots,N-1 do
Wk(1)←2Δζ/ΔxW_k^{(1)}\leftarrow2\Delta\zeta/\Delta x;
bk(1)←−2kΔζb_k^{(1)}\leftarrow-2k\Delta\zeta;
Wk(2)←−ΔxΔζN[uk,xk]W_k^{(2)}\leftarrow-\dfrac{\Delta x}{\Delta\zeta}\mathcal{N}[\mathbf{u}_k,x_k];
b0(2)←ul;0−∑k=0N−1Wk(2)Re⁡-σ(Wk(1)x0+bk(1))b_0^{(2)}\leftarrow u_{l;0}-\sum_{k=0}^{N-1}W_k^{(2)}\operatorname{Re}\text{-}\sigma\left(W_k^{(1)}x_0+b_k^{(1)}\right);
where ul;0=ul(0)u_{l;0}=u_l(0).

Table 1.

Similarly way, we modify the physical-informed neuron-by-neuron training for neural networks with a sigmoid activation function (Algorithm 4 in Appendix B; see the article for details [21]). Neuron-by-neuron training for neural networks with a rectified sigmoid activation is presented in Algorithm 2. It is worth recalling that NbN training and PINN are unsupervised learning techniques for neural networks.

Numerical experiments

For our experiments we used Pytorch [24] version 2.1.2 and the training was carried out on a node with a GPU with characteristics similar to industry-leading GPU and CPU.

Algorithm 2: Neuron-by-neuron training of neural networks with activation function Re-σ\sigma

Data: —

Result: Trained for the EE epoch NNs uθu_\theta, each of which consists of NN neurons on hidden layer

Δx←X/N\Delta x \leftarrow X/N;

Δζ←1\Delta\zeta\leftarrow1;

Initialization of uθ;l(x)u_{\theta;l}(x) of PINN uθu_\theta with following steps;

for k=0,…,N−1k = 0,\ldots,N-1 do

Wk(1)←2Δζ/ΔxW_k^{(1)} \leftarrow2\Delta\zeta/\Delta x;

bk(1)←−2kΔζb_k^{(1)} \leftarrow-2k\Delta\zeta;

Wk(2)←−ΔxN[u(0),xk]/ΔζW_k^{(2)} \leftarrow-\Delta xN[u(0),x_k]/\Delta\zeta;

b0(2)←u1(0)b_0^{(2)} \leftarrow u_1(0);

Neuron-by-neuron improving weights Wk(2)W_k^{(2)} with following steps;

for e=0,…,E−1e = 0,\ldots,E-1 do

for k=0,…,N−1k = 0,\ldots,N-1 do

Wk(2)=−ΔxN[uθ(xk),xk]/ΔζW_k^{(2)} = -\Delta xN[u_\theta(x_k),x_k]/\Delta\zeta.

In our experiments, it was enough to use 3 epochs (E=3E = 3) of training to obtain high-precision results.

In the examples given in this section, we compare the accuracy of solutions predicted by a neural network using a sigmoid activation function (hereinafter referred to as σ\sigma) and a neural network that uses an activation function Re-σ\sigma. We use Algorithms 3 and 1 for PIDD initialization of neural networks with σ\sigma and Re-σ\sigma activation functions, respectively. For all experiments, we use a uniform distribution for the coordinates of the collocation points. The reference solutions were obtained using the odeint solver of scipy.integrate library. As a measure of accuracy, we use the relative L2_2 error, which is defined in Appendix C.

Set of physical problems

To compare the accuracy of neural networks, we consider the physical problems given in this subsection which are described by linear, nonlinear, and chaotic systems of differential equations.

Harmonic Oscillator

The harmonic oscillator is governed by a system of two hidden-order equations for the t∈[0,T]t \in[0,T]

du1dt=u2,du2dt=−ω2u1,t∈[0,T],(9)\frac{du_1}{dt}=u_2,\qquad\frac{du_2}{dt}=-\omega^2u_1,\quad t\in[0,T], \tag*{(9)}
u1(0)=1,u2(0)=0,(10)u_1(0)=1,\qquad u_2(0)=0, \tag*{(10)}

where ω\omega is the frequency of the considered system. We used the following parameters: T=100T = 100 and ω=1\omega= 1. The exact analytical solution of this problem is u1exact=cos⁡(ωt)u_1^{\mathrm{exact}}=\cos(\omega t), u2exact=−sin⁡(ωt)u_2^{\mathrm{exact}}=-\sin(\omega t).

Relativistic slingshot

Consider the problem of a source for single circularly polarized attosecond x-ray pulses [25]. The problem is formulated by following a system of ordinary differential equations in the absolute coordinate tt (see details in [21])

∂h∂t=(Ex−εu⊥21+u⊥2)11+b,∂x∂t=b1+b,∂y∂t=uyh(1+b),∂z∂t=uzh(1+b),b=1+u⊥2−h22h2,uy=ay,L−εy,uz=az,L−εz,(11)\begin{aligned} \frac{\partial h}{\partial t} &= \left(E_x-\varepsilon\frac{u_{\perp}^{2}}{1+u_{\perp}^{2}}\right)\frac{1}{1+b},\\ \frac{\partial x}{\partial t} &= \frac{b}{1+b},\\ \frac{\partial y}{\partial t} &= \frac{u_y}{h(1+b)},\\ \frac{\partial z}{\partial t} &= \frac{u_z}{h(1+b)},\\ b &= \frac{1+u_{\perp}^{2}-h^{2}}{2h^{2}},\\ u_y &= a_{y,L}-\varepsilon y,\\ u_z &= a_{z,L}-\varepsilon z, \tag*{(11)} \end{aligned}

where h=γ−uxh=\gamma-u_x, γ2=1+u⊥2+ux2\gamma^{2}=1+u_{\perp}^{2}+u_x^{2} is the relativistic gamma factor, ux,y,zu_{x,y,z} are the space components of the four-velocity, u⊥2=uy2+uz2u_{\perp}^{2}=u_y^{2}+u_z^{2}, ε=2πe2n′l′/mωLc\varepsilon=2\pi e^{2}n^{\prime}l^{\prime}/m\omega_Lc (l′l^{\prime} is foil thickness), ay,La_{y,L} and az,La_{z,L} are the yy and zz components of the laser vector potential A\mathbf{A}. It is assumed that the electron is initially at x=0x=0, y=0y=0 and z=0z=0, Ex=εth(x/4l′)E_x=\varepsilon_{\mathrm{th}}(x/4l^{\prime}). For the calculations we take ε=4π\varepsilon=4\pi, l′=0.01λLl^{\prime}=0.01\lambda_L (λL=2πc/ωL\lambda_L=2\pi c/\omega_L). The period of impulse is TL=1T_L=1 (ωL=2π/TL\omega_L=2\pi/T_L) and duration of impulse is T=4TLT=4T_L. In our calculation the value l′l^{\prime} was given l′=0.012πl^{\prime}=0.012\pi. The amplitude of the vector potential components of the initiating pulse is described by

ay=a0sin⁡(2πt/TL)sin⁡2(πt/T),t∈[0,T],az=a0cos⁡(2πt/TL)sin⁡2(πt/T),t∈[0,T].(12)\begin{aligned} a_y &= a_0\sin(2\pi t/T_L)\sin^{2}(\pi t/T),\quad t\in[0,T],\\ a_z &= a_0\cos(2\pi t/T_L)\sin^{2}(\pi t/T),\quad t\in[0,T]. \tag*{(12)} \end{aligned}

Lorentz system

As a further example, let us consider the chaotic Lorenz system. This system of equations arises in studies of convection and instability in planetary atmospheric convection, in which variables describe convective intensity and horizontal and vertical temperature differences [26]. This system is given by the following set of ordinary differential equations:

dxdt=σ~(y−x),dydt=x(ρ−z)−y,dzdt=xy−βz,(13)\begin{aligned} \frac{dx}{dt} &= \tilde{\sigma}(y-x),\\ \frac{dy}{dt} &= x(\rho-z)-y,\\ \frac{dz}{dt} &= xy-\beta z, \tag*{(13)} \end{aligned}

where ρ\rho, σ~\tilde{\sigma}, and β\beta are the Prandtl number, Rayleigh number, and a geometric factor, respectively. We take the classical parameters σ~=10\tilde{\sigma}=10, ρ=28\rho=28 and β=8/3\beta=8/3. The max time is T=20T=20, and initial conditions are x(0)=1x(0)=1, y(0)=1y(0)=1, and z(0)=1z(0)=1.

Results of numerical experiments for the PIDD initialization

In our experiments number of neurons of the hidden layer of neural networks is 20000, as the number of collocation points equals 20000. Results of experiments are presented in the Table 1. Column 3 (relative L2L_2 errors) shows the relative errors between solutions, provided by neural networks and solutions obtained using odeint solver of scipy.integrate library (u1u_1 and u2u_2 for harmonic oscillator; hh, xx, yy and zz for relativistic slingshot; xx, yy and zz for Lorentz system). The data presented herein provides compelling evidence that substituting the sigmoid activation function with the rectified sigmoid function results in a reduction of the relative error by at least an order of magnitude for any given value. Moreover, the duration of PIDD initialization is taken short time.

Note, that calculating the relative errors between the exact solution of the harmonic oscillator problem and the solution by odeint for a selected set of points, we have the following values: ϵ[u1exact,u1ref]=1.03×10−6\epsilon[u_1^{\mathrm{exact}},u_1^{\mathrm{ref}}]=1.03\times10^{-6}, ϵ[u2exact,u2ref]=1.04×10−6\epsilon[u_2^{\mathrm{exact}},u_2^{\mathrm{ref}}]=1.04\times10^{-6}. Thus, these values are very close to the values obtained by a neural network with a rectified sigmoid activation function.

ExperimentsPerformance
ProblemActivation functionRelative L2_2 errors (ϵ[uθ;l,ulref]\epsilon[u_{\theta;l},u_l^{\mathrm{ref}}])Run time (s)
Harmonic Oscillatorσ\sigma(5.67×10−5,6.82×10−4)(5.67 \times10^{-5}, 6.82 \times10^{-4})0.0020
Harmonic OscillatorRe-σ\sigma(3.88×10−6,2.63×10−6)(3.88 \times10^{-6}, 2.63 \times10^{-6})0.0017
Relativistic slingshotσ\sigma(1.88×10−5,1.92×10−5,3.48×10−5,3.43×10−5)(1.88 \times10^{-5}, 1.92 \times10^{-5}, 3.48 \times10^{-5}, 3.43 \times10^{-5})0.004
Relativistic slingshotRe-σ\sigma(4.80×10−7,3.57×10−7,2.52×10−6,2.70×10−6)(4.80 \times10^{-7}, 3.57 \times10^{-7}, 2.52 \times10^{-6}, 2.70 \times10^{-6})0.004
Lorentz systemσ\sigma(5.52×10−5,3.07×10−4,4.27×10−5)(5.52 \times10^{-5}, 3.07 \times10^{-4}, 4.27 \times10^{-5})0.0026
Lorentz systemRe-σ\sigma(5.14×10−6,8.10×10−6,3.03×10−6)(5.14 \times10^{-6}, 8.10 \times10^{-6}, 3.03 \times10^{-6})0.0025

Table 1 (PDF p. 6). Relative L2_2 error and run time of the PIDD initialization for the different physical problems and both neural networks with sigmoid activation function and Re-σ\sigma activation function.

Results of numerical experiments for the NbN training

The whole time domains [0;T][0; T] for all problems were split into 20 disjoint equivalent time windows of size Δt=T/20\Delta t = T / 20. In our experiments for each time window we taken neural networks with 10000 neurons on the hidden layer. Results of experiments are presented in Table 2 (the values are the same as in Table 1). It is evident from this table that neural networks with a rectified sigmoid activation function, trained using the NbN method exhibit superior accuracy compared to neural networks utilizing a sigmoid activation function. As NbN represents an unsupervised approach to training, it requires more time compared to PIDD initialization but less than vanilla PINN techniques [3, 4, 21].

ExperimentsPerformance
ProblemActivation functionRelative L2_2 errors (ϵ[uθ;l,ulref]\epsilon[u_{\theta;l},u_l^{\mathrm{ref}}])Run time (s)
Harmonic Oscillatorσ\sigma(3.35×10−5,3.17×10−5)(3.35 \times10^{-5}, 3.17 \times10^{-5})302
Harmonic OscillatorRe-σ\sigma(3.35×10−6,3.30×10−6)(3.35 \times10^{-6}, 3.30 \times10^{-6})254
Relativistic slingshotσ\sigma(6.40×10−5,5.19×10−5,5.44×10−5,6.08×10−5)(6.40 \times10^{-5}, 5.19 \times10^{-5}, 5.44 \times10^{-5}, 6.08 \times10^{-5})982
Relativistic slingshotRe-σ\sigma(5.79×10−6,4.18×10−6,5.16×10−7,3.16×10−7)(5.79 \times10^{-6}, 4.18 \times10^{-6}, 5.16 \times10^{-7}, 3.16 \times10^{-7})903
Lorentz systemσ\sigma(3.47×10−3,5.04×10−3,2.12×10−3)(3.47 \times10^{-3}, 5.04 \times10^{-3}, 2.12 \times10^{-3})1077
Lorentz systemRe-σ\sigma(2.63×10−3,3.83×10−3,1.62×10−3)(2.63 \times10^{-3}, 3.83 \times10^{-3}, 1.62 \times10^{-3})888

Table 2. Relative L2_2 error and run time of the NbN training for the different physical problems and both neural networks with sigmoid activation function and Re-σ\sigma activation function.

Results of NbN training for neural networks with rectified sigmoid function as an activation function are shown on the Figure 1. Over the all-time interval, the absolute errors for all values obtained by such neural networks by NbN training are significantly lower than the absolute errors obtained by the vanilla PINN [21].

Comparison of predicted and reference solutions and absolute errors

Figure 1. (a), (b), (c) and (d) are comparison of the predicted (red dash lines) and reference solutions (blue solid lines) corresponding to hh, xx, yy and zz, respectively. (e), (f), (g) and (h) are absolute errors ∣href−hθ∣|h^{\mathrm{ref}}-h_\theta|, ∣xref−xθ∣|x^{\mathrm{ref}}-x_\theta|, ∣yref−yθ∣|y^{\mathrm{ref}}-y_\theta|, and ∣zref−zθ∣|z^{\mathrm{ref}}-z_\theta|, respectively.

Conclusions

In this paper, a study has been carried out on neural networks with one hidden layer and a rectified sigmoid activation function for solving physical problems. Algorithms of physics-informed data-driven (PIDD) initialization of a neural network and gradient-free neuron-by-neuron (NbN) fitting method for neural networks with one hidden layer and rectified sigmoid function have been proposed. The results show that the accuracy of the ODE solutions which is provided by neural networks with a rectified sigmoid activation function outperforms the accuracy of the ODE solution which is provided by neural networks with a sigmoid activation function. This superiority is observed for the neural networks both after PIDD initialization and NbN training, for linear, nonlinear, and chaotic problems. Using the example of a harmonic oscillator, it is shown that the accuracy of solving the ODE using a neural network with such an activation function as a rectified sigmoid is close in values to standard numerical solvers. Our research continues to unlock the potential of neural networks with one or two hidden layers, and we hope that it will allow us to develop more accurate, effective, and reliable methods to solve complex physical problems for them.

Physics-informed data-driven (PIDD) initialization of neural networks

Algorithm 3: Physics-informed data-driven initialization of neural networks

Data: {uk}k=0N\{u_k\}_{k=0}^{N} for {xk}k=0N\{x_k\}_{k=0}^{N} (uniform grid with a step Δx←X/N\Delta x \leftarrow X/N)

Result: Initialized neural network uθ;l(x)u_{\theta;l}(x) of PINN uθu_\theta, which consists of NN neurons on hidden layer

Δζ←ln⁡(2+3)/2\Delta\zeta\leftarrow\ln(2+\sqrt{3})/2
for k=0,…,N−1 do\textbf{for } k=0,\ldots,N-1\textbf{ do}
Wk(1)←2Δζ/Δx;W_k^{(1)} \leftarrow2\Delta\zeta/\Delta x;
bk(1)←−2kΔζ;b_k^{(1)} \leftarrow-2k\Delta\zeta;
κk←∑m=k−Lk+Lσ′(2Δζ[k−m]);\kappa_k \leftarrow\sum_{m=k-L}^{k+L}\sigma'(2\Delta\zeta[k-m]);
Wk(2)←−Δx2ΔζN[uk,xk]κk;W_k^{(2)} \leftarrow-\frac{\Delta x}{2\Delta\zeta}\frac{\mathcal{N}[u_k,x_k]}{\kappa_k};
b0(2)←ul;0−∑k=0N−1Wk(2)Tσ(Wk(1)Tx0+bk(1)).b_0^{(2)} \leftarrow u_{l;0}-\sum_{k=0}^{N-1}W_k^{(2)\mathrm{T}}\sigma\left(W_k^{(1)\mathrm{T}}x_0+b_k^{(1)}\right).

Neuron-by-neuron training of neural networks

Measure of accuracy

To evaluate the accuracy of the approximate solution obtained with the help of PINN method, the values of the solution of (1) predicted by the neural network at given points are compared with the values calculated on the basis of classical high-precision numerical methods. As a measure of accuracy, the relative total L2L_2 error of prediction is taken, which can be expressed with the following relation

ϵ[uθ,u]={1Ne∑i=1Ne[uθ(xi)−u(xi)]2}1/2×{1Ne∑i=1Ne[u(xi)]2}−1/2(14)\epsilon[u_\theta,u]=\left\{\frac{1}{N_e}\sum_{i=1}^{N_e}[u_\theta(x_i)-u(x_i)]^2\right\}^{1/2}\times\left\{\frac{1}{N_e}\sum_{i=1}^{N_e}[u(x_i)]^2\right\}^{-1/2} \tag*{(14)}

where {xi}i=1Ne\{x_i\}_{i=1}^{N_e} is the set of evaluation points taken from the domain Ω\Omega, uθu_\theta and uu are the predicted and reference solutions respectively.

Algorithm 4: Neuron-by-neuron training

Data: — Result: Trained for the EE epoch NNs uθ;l(x)\mathbf{u}_{\theta;l}(x) of PINN uθ\mathbf{u}_{\theta}, each of which consists of NN neurons on hidden layer

Δx←X/N\Delta x \leftarrow X/N;

Δζ←ln⁡(2+3)/2\Delta\zeta\leftarrow\ln(2+\sqrt{3})/2;

Initialization of uθ;l(x)\mathbf{u}_{\theta;l}(x) with following steps;

for k=0,…,N−1k=0,\ldots,N-1 do

Wk(1)←2Δζ/ΔxW_k^{(1)} \leftarrow2\Delta\zeta/\Delta x;

bk(1)←−2kΔζb_k^{(1)} \leftarrow-2k\Delta\zeta;

Wk(2)←−2ΔxNl[u(0),xk]/ΔζW_k^{(2)} \leftarrow-2\Delta xN_l[\mathbf{u}(0),x_k]/\Delta\zeta;

b0(2)←ul(0)b_0^{(2)} \leftarrow u_l(0);

Neuron-by-neuron improving weights Wk(2)W_k^{(2)} with following steps;

for e=0,…,E−1e=0,\ldots,E-1 do

for k=0,…,N−1k=0,\ldots,N-1 do

Wk(2)=−2ΔxNl[uθ(xk),xk]/ΔζW_k^{(2)}=-2\Delta x\mathcal{N}_l[\mathbf{u}_{\theta}(x_k),x_k]/\Delta\zeta.

In our experiments, it was enough to use 3 epochs (E=3E=3) of training to obtain high-precision results.

References

  1. [1]J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Beghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis, “Highly accurate protein structure prediction with AlphaFold,” Nature, vol. 596, no. 7873, pp. 583–589, aug 2021. [Online]. Available: https://www.nature.com/articles/s41586-021-03819-2DOI
  2. [2]C. Bodnar, W. P. Bruinsma, A. Lucic, M. Stanley, A. Vaughan, J. Brandstetter, P. Garvan, M. Riechert, J. A. Weyn, H. Dong, J. K. Gupta, K. Thambiratnam, A. T. Archibald, C.-C. Wu, E. Heider, M. Welling, R. E. Turner, and P. Perdikariš, “A foundation model for the earth system,” 2024. [Online]. Available: https://arxiv.org/abs/2405.13063
  3. [3]M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” Journal of Computational Physics, vol. 378, pp. 686–707, 2019. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0021999118307125DOI
  4. [4]S. Wang, S. Sankaran, and P. Perdikaris, “Respecting causality is all you need for training physics-informed neural networks,” 2022. [Online]. Available: http://arxiv.org/abs/2203.07404
  5. [5]H. Wang, X. Qian, Y. Sun, and S. Song, “A Modified Physics Informed Neural Networks for Solving the Partial Differential Equation with Conservation Laws,” —. [Online]. Available: https://ssrn.com/abstract=4274376
  6. [6]L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis, “Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators,” Nature Machine Intelligence, vol. 3, no. 3, pp. 218–229, mar 2021. [Online]. Available: https://doi.org/10.1038%2Fs42256-021-00302-5
  7. [7]M. A. Krinitsky, V. M. Stepanenko, A. O. Malkhanov, and M. E. Smorkalov, “A General Neural-Networks-Based Method for Identification of Partial Differential Equations, Implemented on a Novel AI Accelerator,” Supercomputing Frontiers and Innovations, vol. 9, no. 3, sep 2022. [Online]. Available: https://superfri.org/index.php/superfri/article/view/439
  8. [8]V. Fanaskov and I. Oseledets, “Spectral Neural Operators,” 2022. [Online]. Available: https://arxiv.org/abs/2205.10573
  9. [9]M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations,” no. Part I, pp. 1–22. [Online]. Available: https://arxiv.org/abs/1711.10561
  10. [10]V. A. Es’kin, D. V. Davydov, E. D. Egorova, A. O. Malkhanov, M. A. Akhukov, and M. E. Smorkalov, “About optimal loss function for training physics-informed neural networks under respecting causality,” 2023, arXiv:2304.02282.DOI
  11. [11]E. Kharazmi, Z. Zhang, and G. E. Karniadakis, “hp-VPINNs: Variational physics-informed neural networks with domain decomposition,” Computer Methods in Applied Mechanics and Engineering, vol. 374, p. 113547, 2021. [Online]. Available: https://doi.org/10.1016/j.cma.2020.113547
  12. [12]L. Yang, X. Meng, and G. E. Karniadakis, “B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data,” Journal of Computational Physics, vol. 425, p. 109913, 2021. [Online]. Available: https://doi.org/10.1016/j.jcp.2020.109913
  13. [13]S. Cuomo, V. S. di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli, “Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What’s next,” jan 2022. [Online]. Available: http://arxiv.org/abs/2201.05624
  14. [14]R. G. Patel, I. Manziarck, N. A. Trask, M. A. Wood, M. Lee, I. Tomas, and E. C. Cyr, “Thermodynamically consistent physics-informed neural networks for hyperbolic systems,” Journal of Computational Physics, vol. 449, p. 110754, jan 2022. [Online]. Available: https://doi.org/10.1016/j.jcp.2021.110754https://linkinghub.elsevier.com/retrieve/pii/S0021999121006495
  15. [15]S. Cai, Z. Mao, Z. Wang, M. Yin, and G. E. Karniadakis, “Physics-informed neural networks (PINNs) for fluid mechanics: a review,” Acta Mechanica Sinica, vol. 37, no. 12, pp. 1727–1738, dec 2021. [Online]. Available: https://link.springer.com/10.1007/s10409-021-01148-1arxiv.org/abs/2105.09506
  16. [16]B. Moseley, A. Markham, and T. Nissen-Meyer, “Solving the wave equation with physics-informed deep learning,” jun 2020. [Online]. Available: http://arxiv.org/abs/2006.11894
  17. [17]L. Yuan, Y.-Q. Ni, X.-Y. Deng, and S. Hao, “A-PINN: Auxiliary physics informed neural networks for forward and inverse problems of nonlinear integro-differential equations,” Journal of Computational Physics, vol. 462, p. 111260, aug 2022. [Online]. Available: https://doi.org/10.1016/j.jcp.2022.111260https://linkinghub.elsevier.com/retrieve/pii/S0021999122003229
  18. [18]N. A. Daryakenari, M. D. Florio, K. Shukla, and G. E. Karniadakis, “Ai-aristotle: A physics-informed framework for systems biology gray-box identification,” 2023. [Online]. Available: https://arxiv.org/abs/2310.01433
  19. [19]S. Wang, B. Li, Y. Chen, and P. Perdikaris, “Piratenets: Physics-informed deep learning with residual adaptive networks,” 2024. [Online]. Available: https://arxiv.org/abs/2402.00326
  20. [20]F. Jiang, X. Hou, and M. Xia, “Element-wise multiplication based deeper physics-informed neural networks,” 2024. [Online]. Available: https://arxiv.org/abs/2406.04170
  21. [21]V. A. Es’kin, A. O. Malkhanov, and M. E. Smorkalov, “Are two hidden layers still enough for the physics-informed neural networks?” 2024. [Online]. Available: https://arxiv.org/abs/2412.19235
  22. [22]V. Kunc and J. Kléma, “Three decades of activations: A comprehensive survey of 400 activation functions for neural networks,” 2024. [Online]. Available: https://arxiv.org/abs/2402.09092
  23. [23]V. A. Es’kin, D. V. Davydov, J. V. Gur’eva, A. O. Malkhanov, and M. E. Smorkalov, “Separable physics-informed neural networks for the solution of elasticity problems,” 2024. [Online]. Available: https://arxiv.org/abs/2401.13486
  24. [24]A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köp, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, PyTorch: An Imperative Style, High-Performance Deep Learning Library. Red Hook, NY, USA: Curran Associates Inc., 2019.arxiv.org/abs/1912.01703
  25. [25]J. Wang, S. V. Bulanov, M. Chen, B. Lei, Y. Zhang, R. Zagidullin, V. Zorina, W. Yu, Y. Leng, R. Li, M. Zepf, and S. G. Rykovanov, “Relativistic slingshot: A source for single circularly polarized attosecond x-ray pulses,” Phys. Rev. E, vol. 102, p. 061201, Dec 2020. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevE.102.061201DOI
  26. [26]E. N. Lorenz, “Deterministic nonperiodic flow,” Journal of the Atmospheric Sciences, vol. 20, pp. 130–141, 1963. [Online]. Available: https://api.semanticscholar.org/CorpusID:15359559

Paper details

vlnc:W_nwb2ywsg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents