WorksPhysics and Astronomy
Is it time to swish? Comparing activation functions in solving the Helmholtz equation using physics-informed neural networks
Abstract
Solving the wave equation numerically constitutes the majority of the computational cost for applications like seismic imaging and full waveform inversion. An alternative approach is to solve the frequency domain Helmholtz equation, since it offers a reduction in dimensionality as it can be solved per frequency. However, computational challenges with the classical Helmholtz solvers such as the need to invert a large stiffness matrix can make these approaches computationally infeasible for large 3D models or for modeling high frequencies. Moreover, these methods do not have a mechanism to transfer information gained from solving one problem to the next. This becomes a bottleneck for applications like full waveform inversion where repeated modeling is necessary. Therefore, recently a new approach based on the emerging paradigm of physics informed neural networks (PINNs) has been proposed to solve the Helmholtz equation. The method has shown promise in addressing several challenging associated with the conventional algorithms, including flexibility to model additional physics and the use of transfer learning to speed up computations. However, the approach still needs further developments to be fully practicable. Foremost amongst the challenges is the slow convergence speed and reduced accuracy, especially in presence of sharp heterogeneities in the velocity model. Therefore, with an eye on exploring how improved convergence can be obtained for the PINN Helmholtz solvers, we study different activation functions routinely used in the PINN literature, in addition to the swish activation function - a variant of ReLU that has shown improved performance on a number of data science problems. Through a comparative study, we find that swish yields superior performance compared to the other activation functions routinely used in the PINN literature.
Introduction
Forward modeling lies at the heart of most inversion algorithms. Solving the wave equation numerically constitutes the majority of the computational cost for applications like seismic imaging and full waveform inversion (FWI). Frequency-domain solutions of the wave equation allow a reduction in dimensionality as they provide solutions of the Helmholtz equation per frequency. This approach to model wavefields gained popularity with the rise of waveform inversion [1]; however, such solutions require inverting a stiffness matrix that can become computationally intractable for large 3D models or for modeling high frequencies. Moreover, conventional algorithms do not have a mechanism to transfer information gained from solving one problem to the next. For example, the same amount of computational effort is needed to model wavefields corresponding to a small perturbation in the velocity model. This results in a computational bottleneck when repeated modeling is needed for updated velocity models, as in the case of FWI.
Recently, [2] demonstrated the use of physics-informed neural networks (PINNs) to address some of the challenges associated with the conventional Helmholtz solvers, including the flexibility of modeling additional physics. By considering a homogeneous background model, for which the wavefield solution is obtained analytically, [2] trained a neural network to solve for the scattered wavefield. Contrary to conventional deep learning approaches, instead of a labeled set, the training of the neural network is done through a loss function, which embeds the residual of the underlying Helmholtz equation. For an input given by a location in the computational domain, the neural network learns to map it to the scattered wavefield solution at the input location, thanks to the universal approximation theorem. The key enabler of PINNs is automatic differentiation that allows for fast and accurate derivative computation of the neural network’s output w.r.t. to the inputs, which is needed to evaluate the loss function. By using machine learning techniques like transfer learning and surrogate modeling, the method can be used to speed up repeated wavefield computations, as suggested by [3] for the eikonal equation. The PINN-based Helmholtz solver has recently been extended for the vertically transversely isotropic (VTI) case [4].
However, several challenges remain to be addressed in making the PINN-based Helmholtz solver fully practicable. Chief among them is the slow convergence of the neural network in the presence of sharp heterogeneities in the velocity model. Although fully-connected neural networks can approximate arbitrary functions, they have been shown to prioritize learning low frequency features of the underlying function, and hence exhibit a bias towards smooth functions. This poses difficulty in training them to approximate, for example, solutions corresponding to high frequencies or solutions for sharply varying velocity models that generate high-frequency features in the wavefield solution. These observations have been confirmed both in theory and practice [5, 6].
Therefore, in this study, we explore the role of activation functions in improving convergence speed for the PINN-based Helmholtz solver. We do a comparative study of different activation functions that are routinely used in the PINN literature, namely hyperbolic tangent (tanh), inverse tangent (atan), and exponential linear unit (elu). In addition, we consider another variant of the ReLU activation function, named ‘swish’, which has shown superior performance over other activation functions on a number of challenging image classification and machine translation problems [7]. Through tests on a synthetic model, we find that the swish activation function shows markedly improved convergence rate and accuracy in solving the Helmholtz equation.
Theory
The Helmholtz equation in an acoustic, isotropic medium with constant density is given as:
where is the wave velocity, is the angular frequency. The goal is to solve equation (1) for the complex valued wavefield as a function of the angular frequency . Then the time domain solution can be found using inverse Fourier transform of the superposition of the mono-frequency solutions. To avoid the problem of sampling near the source location, [2] suggests solving the Lippmann-Schwinger form of the Helmholtz equation:
for the scattered wavefield , where is the background wavefield satisfying equation (1) for the background velocity , and is the velocity model perturbation. By using a background model with a constant velocity in which we can calculate the background wavefield analytically, we use equation (2) to calculate the scattered wavefield per frequency, from which we find the solution to the Helmholtz equation (1) as:
To solve equation (2) using a deep neural network (DNN), we consider a network with two neurons in the input layer for the spatial coordinates , two output neurons for the real and imaginary parts of the scattered wavefield, and a number of hidden layers. The second derivatives of the outputs w.r.t. the inputs can be computed using automatic differentiation. We seek to minimize the following loss function:
for the real and imaginary parts of the scattered wavefield . Here is the number of training samples, which consists of a randomly selected collocation points from the computational domain, and is the sample index. The role of this loss function is to impose the validity of the Helmholtz equation on a given number of training points. The set of network parameters that minimize the loss function given in equation (3) on this set of training samples can be found by solving the following optimization problem:
Once the DNN is trained, we evaluate it on a regular grid to obtain the predicted scattered wavefields, which are then added to the background wavefields to obtain the final wavefield solution.
Activation functions
The choice of activation function in DNNs has a significant impact on the training dynamics and convergence of the network. Activation functions are sources of non-linearity as they transform the neural network function to a superposition of non-linear basis functions. Choosing the right basis function can make a significant impact in efficiently representing a function. Therefore, in this study, we consider activation functions that are routinely used in the PINN literature (atan, tanh, and elu) in addition to the swish activation function, which is given as . Swish is a smooth, non-monotonic function that has consistently matched or outperformed ReLU on DNNs applied to a variety of challenging problems, including image classification and machine translation. It is believed that the reason for this improvement is that swish is able to better alleviate the problem of vanishing gradients during backpropagation.
Results
In this section, we present a comparative study of different activation functions to solve equation (2) for a two-scatter velocity model, shown in Figure 1. We keep all the hyper-parameters fixed to ensure fairness of the comparison. We use a DNN architecture with 8 hidden layers and 20 neurons in each layer, as used by [2] for the considered velocity model. We randomly select 5000 locations from the computational domain and train the PINN model using the Adam optimizer for 15,000 epochs with mini-batch optimization having a batch size of 256. The PINN model is implemented using the SciANN package [8, ] – a high-level Tensorflow wrapper for scientific computations.
Figure 1. A two-scatter velocity model taken from [2] for the tests. The background has a homogeneous velocity of 2 km/s.
Figure 2 shows the loss history for training the PINN model using different activation functions considered. We observe that elu performs the worst with tanh and atan yielding similar performances, while swish has a markedly improved convergence compared to all the considered activation functions. For all activation functions, the convergence plateaus after an initial reduction of the loss value; however, swish recovers faster than the other activation functions. This can be observed by comparing the loss curves for the first 2000 epochs. We also tabulate the and errors for both the real and imaginary components of the wavefield solution using these activation functions (see Table 1). We clearly observe improved accuracy for the swish activation function compared to the rest.
| Activation functions | ||||
| Real | Imag. | Real | Imag. | |
| tanh | ||||
| atan | ||||
| elu | ||||
| swish |
Table 1. and errors for the real and imaginary parts of the PINN Helmholtz solution obtained using different activation functions.
In Figure 3, we show the scattered wavefield solution obtained from the PINN model trained using the swish activation function. We also plot the wavefield errors on the same color scale. The overall errors are considerably smaller than for the solutions computed using other activation functions as confirmed by Table 1.
Figure 3. The predicted scattered wavefield solution by the PINN model for the real (a) and imaginary (b) parts. Also shown are the wavefield errors for the real (c) and imaginary (d) parts.
Figure 2. Loss history for training of the PINN model using different activation functions in the hidden layer.
Conclusions
We studied the routinely used activation functions from the PINN literature in solving the Helmholtz equation. In addition, we also considered the swish activation function, which is a variant of ReLU. Swish had shown improved performance compared to other activation functions in training DNNs for a variety of data science problems, thanks to its robustness against the problem of vanishing gradients. We also observed superior convergence using the swish activation function in solving the Helmholtz equation. Although these results are preliminary, they demonstrate the potential of selecting better activation functions to improve the convergence of PINN solvers. To focus on the activation functions in this study, we only used the Adam optimizer for all cases. However, to converge faster, typically the optimization process begins with a first-order optimizer like Adam and then switches to a second-order optimizer like L-BFGS after a suitably low value of the loss function. Seeing the loss curves, it can be noted that swish allows early switching to a second-order optimizer compared to other studied activation functions, which would compound its efficacy.
Acknowledgments
We extend gratitude to Prof. Tariq Alkhalifah and Dr. Ehsan Haghighat for helpful discussions.
References
- [1]R Gerhard Pratt. Seismic waveform inversion in the frequency domain, part 1: Theory and verification in a physical scale model. Geophysics, 64(3):888–901, 1999.DOI
- [2]Alkhalifah, T., Song, C., Waheed, U., & Hao, Q. (2020). Wavefield solutions from machine learned functions that approximately satisfy the wave equation. 82nd EAGE Annual Conference & Exhibition, 2020, 1–5.
- [3]Waheed, U., Haghighat, E., Alkhalifah, T., Song, C., & Hao, Q. (2020). Eikonal solution using physics-informed neural networks. 82nd EAGE Annual Conference & Exhibition, 2020, 1–5.
- [4]Song, C., Alkhalifah, T., & Waheed, U. B. (2021). Solving the frequency-domain acoustic vti wave equation using physics-informed neural networks. Geophysical Journal International, 225, 846–859.
- [5]Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301–5310. PMLR, 2019.arxiv.org/abs/1806.08734
- [6]Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint arXiv:1912.01198, 2019.arXiv
- [7]Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.arXiv