betaFeedback

WorksPhysics and Astronomy

Towards Accelerating Particle-Resolved Direct Numerical Simulation with Neural Operators

Abstract

We present our ongoing work aimed at accelerating a particle-resolved direct numerical simulation model designed to study aerosol-cloud-turbulence interactions. The dynamical model consists of two main components - a set of fluid dynamics equations for air velocity, temperature, and humidity, coupled with a set of equations for particle (i.e., cloud droplet) tracing. Rather than attempting to replace the original numerical solution method in its entirety with a machine learning (ML) method, we consider developing a hybrid approach. We exploit the potential of neural operator learning to yield fast and accurate surrogate models and, in this study, develop such surrogates for the velocity and vorticity fields. We discuss results from numerical experiments designed to assess the performance of ML architectures under consideration as well as their suitability for capturing the behavior of relevant dynamical systems.

Introduction

Particle-resolved direct numerical simulations (PR-DNS), which resolve not only the smallest turbulent eddies but also track the development and motion of individual particles, are essential tools for studying aerosol-cloud-turbulence interactions [1, 2]. The associated mathematical models comprise systems of partial differential equations (PDEs), including the Navier-Stokes equations for fluid flow. In turbulent regimes, the numerical solution of the Navier-Stokes equations comes at a high computational cost. To reduce the cost of such high-fidelity direct numerical simulations (DNS), surrogate models (of lower fidelity) are often employed. With judicious algorithmic choices, high-fidelity models may be used in conjunction with surrogate models to reduce computing costs, while retaining acceptable levels of accuracy in the simulation results [3, 4].

The confluence of machine learning and computational fluid dynamics has led to important progress in increasing the speed of DNS [5], turbulence modeling [6], developing surrogate models [7], and learning patterns and PDEs from data [8, 9, 10]. Among many recent advancements in machine learning (ML) and artificial intelligence, deep neural networks for operator learning, which learn mappings between function spaces [11, 12, 13, 14], have emerged as new scientific computing tools with much potential to aid in the development of fast and accurate surrogate models for dynamical systems [12, 15]. Research with such neural operator networks is currently very active in many application areas, including climate sciences [16, 17].

In the present study, we consider the use of such neural operator methods in order to assess their performance as surrogates of a numerical PR-DNS model. Due to the complexity of the PR-DNS model of interest to us, as a first step we focus on a subset of equations used in the PR-DNS model rather than attempting to replace the physical model in its entirety with a machine learning surrogate. This will allow us to study the potential benefits of a hybrid approach, where the neural operator surrogates are judiciously incorporated within the numerical PR-DNS model. We proceed to describe the PR-DNS model in Section 2. Our initial test case and data set are presented in Section 3. The methodology and choice of neural operator method appears in Section 4, followed by a discussion of the obtained results in Section 5. Concluding remarks and future research directions are given in Section 6.

Particle-Resolved DNS Model

A summary of the physical model ultimately of interest to us appears next. Readers may refer to [1] for a thorough description of the model.

Fluid Dynamics Equations

The incompressible Boussinesq system

∂u∂t+(u⋅∇)u=−1ρ0∇p+ν∇2u+fb+fe(1)\frac{\partial\mathbf{u}}{\partial t}+(\mathbf{u}\cdot\nabla)\mathbf{u}=-\frac{1}{\rho_0}\nabla p+\nu\nabla^2\mathbf{u}+\mathbf{f}_b+\mathbf{f}_e \tag*{(1)}
∇⋅u=0(2)\nabla\cdot\mathbf{u}=0 \tag*{(2)}

is adopted to model the fluid velocity. In equations (1)–(2), u\mathbf{u} is the fluid velocity field, pp denotes pressure, ν\nu is the kinematic viscosity, and ρ0\rho_0 is the density of dry air. The buoyancy force fb\mathbf{f}_b in (1) is given by

fb=−g[T−T0T0+0.608(qv−qv0)−qc](3)\mathbf{f}_b=-\mathbf{g}\left[\frac{T-T_0}{T_0}+0.608(q_v-q_{v0})-q_c\right] \tag*{(3)}

where g\mathbf{g} denotes the gravitational acceleration vector, TT is temperature, qvq_v is the water vapor mixing ratio, and qcq_c is the liquid water mixing ratio. T0T_0 and qv0q_{v0} denote reference values. As the buoyancy force fb\mathbf{f}_b in (3) depends on the temperature TT and the water vapor mixing ratio qvq_v, it couples the equations (1)–(2) for fluid velocity with those below for temperature (4) and water vapor (5). Finally, the external force fe\mathbf{f}_e in (1) is introduced to maintain a statistically stationary homogeneous turbulence [1].

The time evolution for the temperature TT and water vapor mixing ratio qvq_v is modeled by the equations

∂T∂t+(u⋅∇)T=LhcpCd+μT∇2T(4)\frac{\partial T}{\partial t}+(\mathbf{u}\cdot\nabla)T=\frac{L_h}{c_p}C_d+\mu_T\nabla^2T \tag*{(4)}
∂qv∂t+(u⋅∇)qv=−Cd+μv∇2qv(5)\frac{\partial q_v}{\partial t}+(\mathbf{u}\cdot\nabla)q_v=-C_d+\mu_v\nabla^2q_v \tag*{(5)}

where LhL_h is the latent heat of water vapor condensation, cpc_p is the specific heat at constant pressure. μT\mu_T and μv\mu_v are the molecular diffusivities for temperature and water vapor, respectively. The rate of exchange between liquid and vapor, a.k.a. the condensation rate CdC_d, as defined in equation (9), depends on droplet radii and thus couples the fluid dynamics model with the equations for droplet growth and motion depicted in Section 2.2 below.

Droplet Growth and Motion

The equations describing the condensation/evaporation of cloud droplets and their motion are given by

Ri(t)dRi(t)dt=A⋅S(Xi,t)(6)R_i(t)\frac{\mathrm{d}R_i(t)}{\mathrm{d}t}=A\cdot S(X_i,t) \tag*{(6)}
dXi(t)dt=Vi(t)(7)\frac{\mathrm{d}X_i(t)}{\mathrm{d}t}=V_i(t) \tag*{(7)}
dVi(t)dt=1τp[u(Xi,t)−Vi(t)]+g.(8)\frac{\mathrm{d}V_i(t)}{\mathrm{d}t}=\frac{1}{\tau_p}\left[\mathbf{u}(X_i,t)-V_i(t)\right]+\mathbf{g}. \tag*{(8)}

Here, Ri(t)R_i(t), Xi(t)X_i(t), and Vi(t)V_i(t) denote, respectively, the radius, position coordinate, and velocity of the ii-th droplet; AA is a function of temperature and pressure; S(X,t)S(X,t) is the supersaturation; g\mathbf{g} is the gravitational acceleration vector; and τp\tau_p is a measure of the droplet inertial effect.

The condensation rate CdC_d in equations (4)–(5) is defined as

Cd(X,t)=4πρlAρ0a3∑i=1nS(Xi,t)Ri(t),(9)C_d(X,t)=\frac{4\pi\rho_l A}{\rho_0a^3}\sum_{i=1}^{n}S(X_i,t)R_i(t), \tag*{(9)}

where aa is the size of a grid cell, nn is the number of droplets in the grid cell, ρl\rho_l and ρ0\rho_0 are the densities of water and air, and Ri(t)R_i(t) is the radius of the ii-th droplet. Note that the condensation rate CdC_d acts as a source (or sink) term for equations (4)–(5) and thus couples the particle equations with the fluid dynamics model. Finally, the full model (1)–(8) is augmented with suitable initial and boundary conditions. Typical simulations of the PR-DNS model from Gao et al. [1] track evolution of approximately 10 million cloud particles in a physical domain of 0.512m×0.512m×0.512m0.512m \times0.512m \times0.512m meshed onto a grid of 256×256×256256 \times256 \times256.

Data Set

As discussed in Section 2, the PR-DNS model captures the basic fields of fluid dynamics like velocity, temperature, and pressure as well as the cloud physics fields of vapor mixing ratio, supersaturation, and condensation rate. It also tracks the location and size of cloud particles. As the first step to model such a complex system, this paper aims to ascertain whether the Navier–Stokes solver governing the velocity field can be replaced by a surrogate model. To this effect, we select a simpler test case of doubly periodic shear layer where the entire dynamics can be captured by a vorticity or a velocity field.

The doubly periodic shear layer is a well-known test case of two dimensional incompressible flow simulations. This test has been extensively employed to assess the errors and stability of fluid dynamics models [18, 19]. Here, an incompressible fluid of kinematic viscosity ν\nu is confined in a two dimensional square domain of side L=1L=1. Periodic boundary conditions are applied in both xx and yy spatial directions. The velocity fields at the initial time t=0t = 0 are given by

ux(y)={U0tanh⁡[(4y−1)/w],y≤1/2U0tanh⁡[(3−4y)/w],y>1/2(10)u_x(y) = \begin{cases} U_0 \tanh[(4y-1)/w], & y \le1/2 \\ U_0 \tanh[(3-4y)/w], & y > 1/2 \end{cases} \tag*{(10)}
uy(x)=U0δsin⁡[2π(x+1/4)],(11)u_y(x) = U_0\delta\sin[2\pi(x+1/4)], \tag*{(11)}

where ux,uyu_x, u_y are the velocities in x,yx, y directions respectively. The parameters ww and δ\delta control the width of the shear layer and the magnitude of the perturbation. The Reynolds number (Re) defined as Re=U0L/ν\mathrm{Re} = U_0L/\nu represents the ratio between inertial and viscous forces. The perturbation causes the shear layers to curl up into two vortices. We modify the initial condition using random numbers to

ux={U0r1,y≤1/2U0r2,y>1/2(12)u_x = \begin{cases} U_0 r_1, & y \le1/2 \\ U_0 r_2, & y > 1/2 \end{cases} \tag*{(12)}
uy={U0r3,x≤1/2U0r4,x>1/2(13)u_y = \begin{cases} U_0 r_3, & x \le1/2 \\ U_0 r_4, & x > 1/2 \end{cases} \tag*{(13)}

where r1,r2,r3,r4r_1,r_2,r_3,r_4 are uniformly distributed random numbers between [−1,1][-1,1]. This generates two opposite vortices that diffuse as time progresses and convect around the periodic domain depending on their initial locations.

The data set for the present study corresponds to Reynolds number 1000. We solve the Navier-Stokes equations for the velocity field using the entropic lattice Boltzmann method [20, 21] on a grid of 64×6464 \times64 points. The flow field is first allowed to evolve for 4tc4t_c (where tc=L/U0t_c = L/U_0 is the convection time) so that the initial sharp discontinuities vanish. Thereafter, time is reset to zero and the velocity (u\mathbf{u}) and vorticity (ωz\omega_z) are sampled from time t=0t = 0 to t=tct = t_c in steps of 0.02tc0.02t_c. Note that the vorticity field ωz(x,y)\omega_z(x,y) is calculated as the curl of the velocity using

ωz(x,y)=∇×u(x,y),(14)\omega_z(x,y) = \nabla\times\mathbf{u}(x,y), \tag*{(14)}

where ∇\nabla is the nabla operator and the velocity vector u\mathbf{u} has components ux,uyu_x, u_y for two-dimensional fluid flow. The dataset consists of fields from 10,000 such simulations. The initial condition of the velocity field [Eqns. (12),(13)] depend on uniformly distributed random numbers that vary from one sample to another. Figure 1 visualizes the two dimensional vorticity field of two such samples. The two oppositely rotating vortices are visible in the figure. In general, in an incompressible viscous flow the vortices stretch (absent for two dimensions) and turn under the influence of each other’s field and diffuse as the time advances.

Visualization of vorticity fields from two samples at different non-dimensional times

Figure 1. Visualization of vorticity field from two samples of a dataset of 10,000 simulations at Re=1000\mathrm{Re}=1000. The red and blue vortices are rotating clockwise and counter-clockwise respectively. The slices at different non-dimensional times t/tct/t_c show the motion of the vortices for unit convection time.

Methodology

In recent years, many studies have been conducted to build and evaluate deep learning models as alternatives to traditional PDE solvers for a wide variety of problems in transport phenomenon. They have been found particularly useful in replacing components of multiscale problems with identifiable computational bottlenecks or in modeling flows for which experimental data is available but constitutive relationships are unknown. One of the most popular deep learning approach for inverse problems is the physics-informed neural network (PINN) [22], which incorporates the governing PDEs in the loss function of the minimization problem during training. PINNs are successful in solving inverse problems for the transport coefficients they are trained on, however, they too sometimes suffer from high computational expense and lack of generalizability. These lacunae of PINNs are attributed to the fact that they learn mappings between finite dimensional Euclidean spaces.

Neural operators [11, 12, 13, 14], which have gained popularity recently, alleviate the limitations of PINNs. Particularly, the Fourier neural operators (FNOs) [12, 14] learn mappings between infinite-dimensional function spaces. A brief description of FNOs is provided in Appendix § 6.1. FNOs have become a popular alternative to the conventional neural networks for a wide range of physical applications like climate modeling [17], multiphase flows in porous media [23, 24], and wave propagation [25]. This popularity is due to their low computational cost, relatively small errors, and the support of features like zero-shot super-resolution for turbulent flows which are limited in other machine learning methods. Li et al. [12] proposed two types of FNOs for transient dynamics in two spatial dimensions:

  • The two dimensional FNO coupled with a recurrent neural network architecture for recurrent propagation in time, which maps the input of a few consecutive time steps to the next chronological time step.

  • The three dimensional FNO, which learns a mapping in two spatial and one temporal dimensions. This maps an input time window to an entire output time window in a single inference step.

In the present study, we choose the three dimensional FNO as we want to predict fields over a long time interval without accumulating errors from step by step temporal predictions. The Fourier series is truncated at a finite number of lower modes while filtering the higher modes. The FNO model used here has four Fourier layers where width represents the number of nodes in a layer. The code is available on Github [12] from the original authors of FNO and has also been implemented in NVIDIA’s Modulus package [26]. We design a simple study to understand the behavior and errors of surrogate models. As discussed previously, the data set contains velocity and vorticity fields from 10,000 simulations of doubly periodic shear layer corresponding to Reynolds number 1000. We calculate the errors for different number of samples, input-output training intervals, and time refinements. The L2L_2 errors reported in the following section are calculated at each time step in the prediction interval and averaged over N=500N=500 test samples. We note that the same test set of size 500 is used for all numerical experiments that follow. The fields are first normalized via the target data’s mean and standard deviation and then the error is calculated as

L2 Error=1N∑N[1nxny∑x∑y(ϕpred(x,y)−ϕtrue(x,y))2](15)L_2\ \mathrm{Error}=\frac{1}{N}\sum_N\left[\sqrt{\frac{1}{n_xn_y}\sum_x\sum_y\left(\phi^{\mathrm{pred}}(x,y)-\phi^{\mathrm{true}}(x,y)\right)^2}\right] \tag*{(15)}

where the field ϕ\phi is vorticity or velocity, nx,nyn_x,n_y are the number of grid points in xx and yy spatial directions, and NN is the number of test samples.

Numerical Experiments

The time stepping threshold of numerical schemes for solving partial differential equations are well understood from their stability analysis. However, an analogous threshold for data driven models can not be derived from similar mathematics. The idea behind this section is (a) to experiment with different time steps to understand their relation with the accumulated errors, and (b) to estimate the amount of data that needs to be generated from PR-DNS model which in itself is a computationally extensive task.

Required number of samples

As each simulation of PR-DNS takes substantial time and resources to run and generates a large amount of output data, we would like to assess the smallest number of samples required to train a sufficiently accurate surrogate model. To that end, we train three FNO models with the vorticity field using 1000, 4000, and 8000 samples, respectively. All three models are trained with 10 time steps (t=0t=0 to t=0.18tct=0.18t_c) as input and 40 time steps (t=0.20tct=0.20t_c to t=0.98tct=0.98t_c) as output. We calculate the errors on the same 500 test samples over the prediction time (t=0.20tct=0.20t_c to t=0.98tct=0.98t_c) and plot them in Figure 2.

Figure 2. Performance of models trained on 1000, 4000, and 8000 samples as input: L2L_2 error calculated on 500 test samples over the prediction time (t=0.20 tct = 0.20\,t_c to t=0.98 tct = 0.98\,t_c)

It is observed from Figure 2 that the errors of the model trained with 1000 samples grows faster with respect to time than the other two models. The errors for models trained with 4000 and 8000 samples do not have significant difference in terms of prediction errors. Thus, in the subsequent tests we will only evaluate models trained with 1000 and 4000 samples as this will save significant training data generation costs for the PR-DNS simulations.

Figure 3. Mean squared error for models trained with 4000 input samples and different hyperparameter settings. The order of hyperparameters from left to right is width, batch size, scheduler step, scheduler γ\gamma, learning rate, and number of modes respectively.

Choosing the hyperparameters

Similar to most machine learning models, the Fourier neural operator’s performance is dependent on the choice of the so-called hyperparameters. The hyperparameters here consist of the number of mode, neural net width, learning rate, batch size, scheduler step size, scheduler γ\gamma, etc. In this section, we calculate the training mean squared error (MSE) with 4000 training samples under a few batch sizes, widths, scheduler steps, and scheduler γ\gammas, learning rate, and number of modes. Figure 4 shows that the MSE with a width 10 is larger than those using other hyperparameter settings (due to its use of smaller neural nets) and with four modes (due to inadequate number of Fourier modes to represent a function). The training time for widths 20 and 40 are similar. As the width of 40 requires more floating point operations, we select 20 as the default width for the remaining investigations.

Figure 4. L2L_2 errors with different input and output time windows for models trained with 4000 samples. The numbers in the parentheses indicate the number of snapshots in the corresponding data set.

Varying input and output time windows

Each sample of doubly periodic shear layer consists of 51 time snapshots spreading uniformly over unit convection time in steps of 0.02 convection times. As the objective of the study is to accelerate Navier-Stokes solvers, a model with maximum output and minimum input windows is more beneficial computationally. In the meantime, a surrogate model with low prediction errors for longer output times is preferential.

The data set contains 51 snapshots sampled over unit convection time, which could be split into input and output windows for FNOs in many ways, however, we chose (a) 10 input, 40 output, (b) 20 input, 30 output, and (c) 10 inputs, 10 outputs to understand the trends in their respective errors. Thus, we test the error for three input-output time windows:

  • input t=0t = 0 to t=0.18 tct = 0.18\,t_c (10 snapshots); output t=0.20 tct = 0.20\,t_c to t=0.38 tct = 0.38\,t_c (10 snapshots)

  • input t=0t = 0 to t=0.18 tct = 0.18\,t_c (10 snapshots); output t=0.20 tct = 0.20\,t_c to t=0.98 tct = 0.98\,t_c (40 snapshots)

  • input t=0t = 0 to t=0.38 tct = 0.38\,t_c (20 snapshots); output t=0.40 tct = 0.40\,t_c to t=0.98 tct = 0.98\,t_c (30 snapshots).

This time refinement and input-output split with acceptable errors is expected to depend on the flow conditions too. For Reynolds number 1000 we find that L2L_2 errors are the lowest for 10 inputs and 10 outputs (see Figure 4). The plot also shows that the surrogate models accumulate errors for longer prediction times – a behaviour which is expected to restrict the longest possible prediction times for surrogate models. This experiment reveals that the errors grow if the input windows are much smaller than the output windows. Thus a model with fewer input snapshots and more output snapshots is expected to perform poorly in comparison to similar number of input output snapshots.

Next, we evaluate the errors for input-output time windows different from those for which the model was trained on. We train the model for t=0t = 0 to t=0.18 tct = 0.18\,t_c as input and t=0.20 tct = 0.20\,t_c to t=0.38 tct = 0.38\,t_c as output. Once the model is obtained, we change the input-output time windows to

  • input t=0.20 tct = 0.20\,t_c to t=0.38 tct = 0.38\,t_c, output t=0.40 tct = 0.40\,t_c to t=0.58 tct = 0.58\,t_c

  • input t=0.40 tct = 0.40\,t_c to t=0.58 tct = 0.58\,t_c, output t=0.40 tct = 0.40\,t_c to t=0.58 tct = 0.58\,t_c

  • input t=0.40 tct = 0.40\,t_c to t=0.58 tct = 0.58\,t_c, output t=0.60 tct = 0.60\,t_c to t=0.78 tct = 0.78\,t_c.

Thus, we are essentially computing errors for input-output windows moved ahead in time by 0.2 tct_c. In Figure 5, at the top we plot L2L_2 errors with the ground-truth as an input, whereas the bottom figure takes the preceding 0.2 tc0.2\,t_c interval’s output as its input. It can be seen that when the input is the ground truth the errors remain at relatively low levels. However while using previous prediction results as input, errors grow more rapidly. This is expected as inaccurate input amplifies output errors for both numerical simulations and ML models.

Figure 5. (Top) L2L_2 errors with the ground-truth as an input, (bottom) L2L_2 errors for the predictions with preceding 0.2 tc0.2\,t_c window as in input. The numbers in the parentheses indicate the number of snapshots in the corresponding data set.

Varying the time refinement

The maximum time step per iteration for classical PDE solvers is typically dependent on convergence and numerical stability. For instance, the Courant–Levy–Friedrichs condition dictates the time step of explicit time integration schemes. In contrast, the time steps of ML-based surrogate models is not bound to the time step of PDE solvers employed to generate data. Thus in this section we try to understand the maximum permissible time steps of FNO. We compare L2L_2 errors for two time refinements of 0.02 tc0.02\,t_c and 0.04 tc0.04\,t_c, with the input and output time window sizes both fixed to 10. For the time refinement of 0.02 tc0.02\,t_c, the input window contains 10 time steps from t=0t=0 to t=0.18 tct=0.18\,t_c and the output contains 10 time steps from t=0.20 tct=0.20\,t_c to t=0.38 tct=0.38\,t_c. For the time refinement of 0.04 tc0.04\,t_c, the input window contains 10 time steps from t=0t=0 to t=0.36 tct=0.36\,t_c and the output contains 10 time steps from t=0.40 tct=0.40\,t_c to t=0.76 tct=0.76\,t_c. It is obvious that the latter case is able to predict a longer time window thereby saving substantial computational cost of high-fidelity PR-DNS. However, it is important to ensure the errors remain within acceptable limits. Note that the computational cost of numerical simulation is the same for both the cases below (0.4 tc0.4\,t_c), however this study serves to answer which of the following options is better in terms of accuracy:

  • Run PR-DNS from t=0t=0 to t=0.18 tct=0.18\,t_c then use FNO to predict fields from t=0.20 tct=0.20\,t_c to t=0.38 tct=0.38\,t_c. This will be later integrated in a workflow that runs PR-DNS again from t=0.40 tct=0.40\,t_c to t=0.58 tct=0.58\,t_c then uses FNO to predict fields from t=0.60 tct=0.60\,t_c to t=0.78 tct=0.78\,t_c.

  • Run PR-DNS from t=0t=0 to t=0.38 tct=0.38\,t_c then use FNO to predict fields from t=0.40 tct=0.40\,t_c to t=0.78 tct=0.78\,t_c.

The computational cost of the first approach is slightly higher as one needs to perform two FNO predictions, however, it should be noted that the cost of prediction from FNO is only a small fraction of the entire workflow. From Figure 6 it is seen that the first option accumulates an L2L_2 error of 0.0180.018 till t=0.18 tct=0.18\,t_c whereas the second option accumulates an L2L_2 error of 0.0640.064 till t=0.78 tct=0.78\,t_c. If the first option is integrated in a workflow that runs PR-DNS again from t=0.40 tct=0.40\,t_c to t=0.58 tct=0.58\,t_c and performs an FNO prediction from t=0.60 tct=0.60\,t_c to t=0.78 tct=0.78\,t_c, the FNO prediction can be reasonably assumed to accumulate similar L2L_2 errors. Thus, the net error at t=0.78 tct=0.78\,t_c would be around 0.0360.036 which is smaller than the first approach’s 0.0640.064. Therefore, the second approach with smaller intervals but higher time refinement seems to be the better one.

Figure 6. Varying the time refinement of input-output windows for different sample sizes. The numbers in the parentheses indicate the number of snapshots in the corresponding data set.

Vorticity versus velocity formulation

The previous experiments are conducted with vorticity field as input and output data. This approach is advantageous for building surrogate models for incompressible two-dimensional flows because

  • One only has to deal with one variable i.e., the non-zero component of vorticity vector ω=(0,0,ωz)\omega= (0, 0, \omega_z).

  • The continuity equation is identically satisfied in the vorticity formulation as one can define a streamfunction ψ\psi such that ωz=−∇2ψ\omega_z = -\nabla^2\psi which automatically produces a divergence-free velocity field.

It should be noted that the full-fledged PR-DNS model solves the velocity formulation of the three-dimensional Navier-Stokes equations. Thus, it is important to assess the type and magnitude of errors when dealing with velocity data sets.

In this section, we calculate L2L_2 errors when the model training and testing is performed with velocity fields and compare them with the vorticity errors. It can be observed from Figure 7 that the L2L_2 error when using the velocity data set is lower than that of vorticity. Further investigation is required to find the reason for this behavior but it could be an artifact of availability of twice the volume of training data, i.e., two velocity fields ux,uyu_x, u_y are available as as opposed to a single field of ωz\omega_z in the vorticity formulation. However, on the flip side a prediction from velocity data set does not guarantee a divergence-free velocity field.

Figure 7. Comparison of L2L_2 errors on 500 test samples: vorticity data set versus velocity data set for 4000 training samples.

We visualize the vorticity fields obtained from various experiments in Figure 8. The true vorticity field at the top is the target ground truth. The middle plot is the vorticity field obtained from the model trained on vorticity data set. The bottom plot is the vorticity field estimated by processing the velocity data obtained from the model trained on the velocity data set. It can be seen that the vortices in the target field convect away from the top right corner without any appreciable structural difference. However, a loss in the braids of the vortices is observed in the middle plot in addition to a noisy vorticity field. The velocity data set performs better at preserving the braids but shows noise levels similar to the middle plot. We find that the errors in the continuity equation are of O(10−3)O(10^{-3}) (see Figure 9) when using velocity data set. This will be addressed in the future by incorporating the continuity equation in the loss function.

Comparison of true vorticity, vorticity from vorticity data, and vorticity from velocity data over time

Figure 8. Comparison of vorticity fields of a test sample from training on velocity versus vorticity data. The vorticity field for the velocity data set is found by processing the velocity data.

Deviation from divergence-free condition for fields obtained from training on velocity data set

Figure 9. Visualization of the deviation from divergence-free condition for fields obtained from training on velocity data set.

Conclusions and Future Directions

We have studied the performance of FNO for doubly periodic shear layer at a relatively mild Reynolds number of 1000. We have found that the lowest prediction errors are obtained when the input-output time intervals are smaller than 0.2 convection times. We have also seen that the velocity data set performs slightly better than the vorticity data, but, the velocity field obtained is not entirely divergence-free. Unlike traditional numerical methods at under-resolved grid refinements, FNOs do not give rise to any spurious nonphysical vortices. Neglecting the model training cost, the velocity prediction from FNO takes a fraction of the time compared to the numerical PDE solver. For example, the inference from FNO takes 0.21 seconds on Nvidia V100 GPU compared to 0.65 seconds for LBM solver on AMD EPYC 7402P CPU for a mesh of 64x64 over 0.2tc0.2t_c. Thus, by using FNO as the surrogate model for velocity fields we save the cost of the numerical PDE solver which varies from 70–90% (depending on grid size and number of cores used) of the total cost.

The data from our full-fledged PR-DNS model is three dimensional and corresponds to a Reynolds number of approximately 10,000. Thus, in the future we plan to extend FNOs to predict three dimensional velocity fields at a higher Reynolds number. This will require more data in terms of finer temporal and spatial resolutions as well as number of samples. There are other neural operators such as Clifford [27], Markov [28], and DeepONet [29] that could also be used if they exhibit better performance or yield lower errors. As this study focuses on only one component of the overall PR-DNS model (i.e., substitution of velocity solver with FNO), it remains to be seen how the other components of the model are impacted by it. However, we note that the errors in FNO’s prediction are similar to numerical diffusion of Navier-Stokes solvers at coarse grid resolution in that both manifest as an enhanced diffusion in the vorticity field. This will certainly impact the other variables behavior and will be investigated in future studies. Another outstanding question that needs to be answered by future studies is the identification of components of the PR-DNS model other than the Navier-Stokes equations that may benefit if modeled via machine learning methods. Although in this work we have evaluated the errors when replacing of the Navier-Stokes solver, our future investigations will assess judicious replacement of thermodynamical governing equations and/or particle propagation with ML-based surrogates as well. Finally, incorporating governing PDEs in the loss function will lead to smaller errors and is an avenue for further research.

Acknowledgments

The authors gratefully acknowledge financial support from the Laboratory Directed Research and Development program at Brookhaven National Laboratory. We also acknowledge administrative support from the Sustainable Research Pathways (SRP) Program for JY who was SRP student participant at Brookhaven National Lab.

Financial Disclosure

This work is supported by Laboratory Directed Research and Development program at Brookhaven National Laboratory, which is sponsored by the US Department of Energy, Office of Science, under Contract Number DE-SC0012704.

Conflict of Interest

The authors declare no potential conflict of interests.

Supporting Information

Additional supporting information may be found in the online version of the article at the publisher’s website.

Fourier Neural Operator (FNO) Learning

Informally, operator learning deals with the problem of learning mappings between (infinite dimensional) function spaces. In particular, a neural operator is a (suitably defined) neural network architecture designed to approximate some operator. Within the context of differential equations, such neural operators are sought as approximations, or surrogates, to the solution operator of differential equations [11, 12, 13, 14]. Following [14], we provide next a brief description of Fourier Neural Operators (FNOs) for partial differential equations (PDEs), as this is the class of neural operators we adopt in the present study. For a thorough treatment, the reader is referred to the original publications [12, 14].

Let A\mathcal{A} and U\mathcal{U} be two Banach spaces of functions defined, respectively, on bounded domains Ωda⊂Rda\Omega_{d_a} \subset\mathbb{R}^{d_a} and Ωdu⊂Rdu\Omega_{d_u} \subset\mathbb{R}^{d_u}. Let

G:A→U(16)\mathcal{G}:\mathcal{A}\to\mathcal{U} \tag*{(16)}

be a (possibly nonlinear) operator mapping functions a∈Aa\in\mathcal{A} to functions u∈Uu\in\mathcal{U} and

Gθ:A×Θ→U(17)\mathcal{G}_{\theta}:\mathcal{A}\times\Theta\to\mathcal{U} \tag*{(17)}

be a map parameterized by θ∈Rp\theta\in\mathbb{R}^{p} such that, for some θ∗∈Rp\theta^{*}\in\mathbb{R}^{p}, we have that

Gθ∗≈G(18)\mathcal{G}_{\theta^{*}}\approx\mathcal{G} \tag*{(18)}

is a suitable approximation and thus Gθ∗\mathcal{G}_{\theta^{*}} may serve as a useful surrogate to the operator G\mathcal{G}. In the case of PDEs, the space A\mathcal{A} comprises “input” functions, such as initial conditions, coefficients, or forcing terms, that the PDEs depend on, while the solution operator G\mathcal{G} maps these inputs to the solution u∈Uu\in\mathcal{U} satisfying the PDEs with the given inputs. Since, in practice, solving PDEs numerically is often a time-consuming task, fast and accurate surrogates are frequently needed.

In [14], the authors proposed a neural operator architecture (to be described shortly) and several classes of parameterizations, to serve as surrogates (A3) to operators of interest, such as solution operators of PDEs. They prove a universal approximation theorem showing that their proposed neural operator, parameterization classes of which include FNOs, can approximate any given nonlinear continuous operator. Furthermore, the neural operator proposed in [14] is discretization-invariant, meaning that it shares the same model parameters θ∗\theta_* among different discretizations of the underlying function spaces. This is advantageous since, upon training (i.e., learning the optimal model parameters θ∗\theta_*), the resulting neural operators can then be used with discretizations of the input space A\mathcal{A} and output space U\mathcal{U} which are possibly different than the discretizations used when training.

The architecture of a Fourier neural operator, FNO, is depicted in Figure A1. Here, a FNO Gθ:A→U\mathcal{G}_{\theta}:\mathcal{A}\to\mathcal{U} is defined as a composition

Architecture of a Fourier neural operator

Figure A1. Architecture of a Fourier neural operator (FNO). (Figure adapted from [14]. Permission to reuse material made available under a CC-BY 4.0 license, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided at https://jmlr.org/papers/v24/21-1524.html)

Gθ≡Q∘σL∘(WL+KL)∘σL−1∘(WL−1+KL−1)∘…∘σ1∘(W1+K1)∘P(19)\begin{aligned} \mathcal{G}_{\theta} &\equiv Q \circ\sigma_L \circ(W_L + K_L) \circ\sigma_{L-1} \circ(W_{L-1} + K_{L-1}) \\ &\quad\circ\ldots\circ\sigma_1 \circ(W_1 + K_1) \circ P \tag*{(19)} \end{aligned}

of linear operators {Wl+Kl}l=1L\{W_l + K_l\}_{l=1}^{L} and nonlinear activation functions {σl}l=1L\{\sigma_l\}_{l=1}^{L} for some given number LL of layers in the neural network, and where:

  1. First, the input a∈Aa \in\mathcal{A} is lifted to a higher dimensional representation

v0=P∘a(20)v_0 = P \circ a \tag*{(20)}

by a map PP, often parameterized by a shallow neural network PθPP_{\theta_P} with parameters θP\theta_P.

  1. The above lifting operation is followed by a number LL of iterative updates v0↦v1↦…↦vLv_0 \mapsto v_1 \mapsto\ldots\mapsto v_L, with

vl=σl∘(Wl+Kl)∘vl−1,l=1,…,L,(21)v_l = \sigma_l \circ(W_l + K_l) \circ v_{l-1}, \qquad l=1,\ldots,L, \tag*{(21)}

where, for each l=1,…,Ll=1,\ldots,L, WlW_l is a linear operator parametrized by a neural network WθWW_{\theta_W} with parameters θW\theta_W and

Kl≡F−1∘R∘F,(22)K_l \equiv\mathcal{F}^{-1} \circ R \circ\mathcal{F}, \tag*{(22)}

where F\mathcal{F} denotes the Fourier transform to frequency space and F−1\mathcal{F}^{-1} its inverse (transforming to physical space) and RR is an operator filtering frequencies, which is parameterized by a neural network RθRR_{\theta_R} with parameters θR\theta_R.

  1. Finally, the output is

u=Q∘vL,(23)u = Q \circ v_L, \tag*{(23)}

where QQ is a map that projects vLv_L to u∈Uu \in\mathcal{U}, and is parameterized by a neural network QθQQ_{\theta_Q} with parameters θQ\theta_Q.

The parameters of the FNO Gθ\mathcal{G}_{\theta} from (A4) are thus given by θ=(θP,θW,θR,θQ)\theta=(\theta_P,\theta_W,\theta_R,\theta_Q).

References

  1. [1]Zheng Gao, Yangang Liu, Xiaolin Li, Chunsong Lu, J. Geophys. Res. Atmos. 123(4), 2194 (2018).
  2. [2]Lulin Xue, Sudarsan Bera, Sisi Chen, Harish Choudhary, Shivasi Dixit, Wojciech W. Grabowski, Sandeep Jayakumar, Steven Krueger, Gayatri Kulkarni, Sonia Lasher-Trapp, Holly Mallinson, Thara Prabhakaran, Shin ichiro Shima, Bull. Am. Meteorol. Soc. 103(5), E1413 (2022), https://doi.org/10.1175/BAMS-D-22-0018.1.
  3. [3]Benjamin Peherstorfer, Karen Willcox, Max Gunzburger, SIAM Review 60(3), 550 (2018), https://doi.org/10.1137/16M1082469.
  4. [4]Nikos Spyropoulos, Dimitris Vlastos, George Papadakis, John M. Prospathopoulos, Vasilis A. Rizotis, J. Phys.: Conf. 2265(3) (2022), https://dx.doi.org/10.1088/1742-6596/2265/3/032008.
  5. [5]Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, Stephan Hoyer, Proc. Natl. Acad. Sci. 118(21), e2101784118 (2021).
  6. [6]Karthik Duraisamy, Gianluca Iaccarino, Heng Xiao, Annu. Rev. Fluid Mech. 51, 357 (2019).
  7. [7]J Nathan Kutz, Steven L Brunton, Bingni W Brunton, Joshua L Proctor, Dynamic mode decomposition: data-driven modeling of complex systems, (SIAM2016).
  8. [8]Steven L Brunton, Joshua L Proctor, J Nathan Kutz, Proc. Natl. Acad. Sci. 113(15), 3932 (2016).
  9. [9]Jean-Michel Tuncy, Mihir Durve, Andrea Montessori, Sauro Succi, arXiv preprint arXiv:2305.06222 (2023).
  10. [10]Tianle Wang, Bigeng Wang, Elisha Siddiqui M̄atekole, Mohammad Atif, in Smoky Mountains Computational Sciences and Engineering Conference, pp. 345–365.
  11. [11]Tianping Chen, Hong Chen, IEEE Transactions on Neural Networks 6(4), 911 (1995).
  12. [12]Zongyi Li, Nikola Kovachki, Kamyar Azizadehnesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, arXiv:2010.08895 (2020).arxiv.org/abs/2010.08895
  13. [13]Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, George Em Karniadakis, Nat Mach Intell 3, 218 (2021), https://doi.org/10.1038/s42256-021-00302-5.
  14. [14]Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizadehnesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, Journal of Machine Learning Research 24(89), 1 (2023), https://www.jmlr.org/papers/volume24/21-1524/21-1524.pdf.
  15. [15]Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, George Em Karniadakis, Comput. Methods Appl. Mech. Eng. 393, 114778 (2022), https://doi.org/10.1016/j.cma.2022.114778.
  16. [16]Peishi Jiang, Nis Meinert, Helga Jordão, Constantin Weisser, Simon Holgate, Alexander Lavin, Björn Lütjens, Dava Newman, Haruko Wainwright, Catherine Walker, Patrick Barnard, Digital Twin Earth - Coasts: Developing a fast and physics-informed surrogate model for coastal floods via neural operators (2021), arXiv:2110.07100.arxiv.org/abs/2110.07100
  17. [17]Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizadehnesheli, Pedram Hassan-zadeh, Karthik Kashinath, Animashree Anandkumar, FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators (2022).
  18. [18]David L Brown, J. Comput. Phys. 122(1), 165 (1995).
  19. [19]Mohammad Atif, Praveen Kumar Kolluru, Santosh Ansumali, Phys. Rev. E 106, 055307 (2022).
  20. [20]Mohammad Atif, Praveen Kumar Kolluru, Chakradhar Thantanapally, Santosh Ansumali, Phys. Rev. Lett. 119, 240602 (2017).
  21. [21]Seyed Ali Hosseini, Mohammad Atif, Santosh Ansumali, Iliya V Karlin, Comput. Fluids , 105884 (2023).
  22. [22]Maziar Raissi, Paris Perdikaris, George E Karniadakis, J. Comput. Phys. 378, 686 (2019).
  23. [23]Thomas J Grady II, Rishi Khan, Mathias Louboutin, Ziyi Yin, Philipp A Witte, Ranveer Chandra, Russell J Hewett, Felix J Herrmann, arXiv:2204.01205 (2022).DOI
  24. [24]Zhongyi Yang, Min Zhu, Dongzhuo Li, Qiuzi Li, Yanhua O Yuan, Lu Lu, arXiv:2303.04778 (2023).
  25. [25]Steven Guan, Ko-Tsung Hsu, Parag V Chitnis, arXiv:2108.09374 (2021).DOI
  26. [26]NVIDIA Modulus . https://docs.nvidia.com/deeplearning/modulus/index.html.
  27. [27]Johannes Brandstetter, Rianne van den Berg, Max Welling, Jayesh K Gupta, arXiv preprint arXiv:2209.04934 (2022).
  28. [28]Zongyi Li, Nikola Kovachki, Kamyar Azizadehnesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, arXiv:2106.06898v2 (2021).arxiv.org/abs/2106.06898
  29. [29]Vivek Oommen, Khemraj Shukla, Somdatta Goswami, Rémi Dingreville, George Em Karniadakis, Npj Comput. Mater. 8(1), 190 (2022).

Paper details

vlnc:W_7vndrnds
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents