betaFeedback

WorksMathematics

Energy-Conserving Neural Network for Turbulence Closure Modeling

Abstract

In turbulence modeling, we are concerned with finding closure models that represent the effect of the subgrid scales on the resolved scales. Recent approaches gravitate towards machine learning techniques to construct such models. However, the stability of machine-learned closure models and their abidance by physical structure (e.g. symmetries, conservation laws) are still open problems. To tackle both issues, we take the `discretize first, filter next' approach. In this approach we apply a spatial averaging filter to existing fine-grid discretizations. The main novelty is that we introduce an additional set of equations which dynamically model the energy of the subgrid scales. Having an estimate of the energy of the subgrid scales, we can use the concept of energy conservation to derive stability. The subgrid energy containing variables are determined via a data-driven technique. The closure model is used to model the interaction between the filtered quantities and the subgrid energy. Therefore the total energy should be conserved. Abiding by this conservation law yields guaranteed stability of the system. In this work, we propose a novel skew-symmetric convolutional neural network architecture that satisfies this law. The result is that stability is guaranteed, independent of the weights and biases of the network. Importantly, as our framework allows for energy exchange between resolved and subgrid scales it can model backscatter. To model dissipative systems (e.g. viscous flows), the framework is extended with a diffusive component. The introduced neural network architecture is constructed such that it also satisfies momentum conservation. We apply the new methodology to both the viscous Burgers' equation and the Korteweg-De Vries equation in 1D. The novel architecture displays superior stability properties when compared to a vanilla convolutional neural network.

Introduction

Direct numerical simulations (DNSs) of turbulent flows are often infeasible due to the high computational requirements. Especially for applications in design and uncertainty quantification this rapidly becomes computationally infeasible, as typically many simulations are required [1, 2]. To tackle this issue several different approaches have been proposed, such as reduced order models [3], Reynolds-averaged Navier-Stokes (RANS) [4], and Large Eddy Simulation (LES) [5]. These approaches differ in how much of the physics is modeled. Here we will focus on the LES approach.

In LES, the large-scale physics is modeled directly by a coarse-grid discretization. The coarse grid is accounted for by applying a filter to the original equations. However, as the filter does not commute with the nonlinear terms in the equations, a commutator error arises. This prevents one from obtaining an accurate solution without knowledge of the subgrid-scale (SGS) content. This commutator error is referred to as the closure term. Modeling this term is the main concern of the LES community. A major difficulty in this process is dealing with energy moving from the small scales to the large scales (backscatter) [6, 7]. This is because the SGS energy is unknown during the time of the simulation. This makes accounting for backscatter without leading to numerical instabilities difficult [8]. Classical physics-based closure models are therefore often represented by a dissipative model, e.g. of eddy-viscosity type [9]. This ensures a net decrease in energy. Another option is that the closure model is clipped such that backscatter is removed [10]. Even though the assumption of a global net decrease in energy is valid [9], explicit modeling of backscatter is still important. This is because locally the effect of backscatter can be of great significance [11, 12]. Closure models that explicitly model the global SGS energy at a given point in time, to allow for backscatter without sacrificing stability, also exist [13]. Recently, machine learning approaches, or more specifically neural networks (NNs), have come forward as viable closure models. They have been shown to outperform the classical approaches for different use cases [14, 15, 16, 17, 18]. However, stability remains an important issue, along with abidance by physical structure such as mass, momentum, and energy conservation [16, 19, 20, 21].

In [19] homogeneous isotropic turbulence for the compressible Navier-Stokes equations was treated. A convolutional neural network (CNN) was trained to reproduce the closure term from high-resolution flow data. Although a priori cross-correlation analysis on the training data showed promising results, stable models could only be achieved by projecting onto an eddy-viscosity basis. In [20] a gated recurrent NN was applied to the same test case. This network displayed even higher cross-correlation with the closure term, but still yielded unstable models. Even after training on data with added artificial noise the model remained unstable [21]. In [16] incompressible turbulent channel flow was treated. Here NNs with varying levels of locality were used to construct a closure model. They showed that increasing the view of the NN improves a priori performance. However, a posteriori analysis showed that this increased input space also led to instabilities. Even after introducing backscatter clipping these larger models were still outperformed by the highly localized NN models. Two promising approaches to improving the stability of NN closure models are ‘trajectory fitting’ [14, 15, 22, 23, 24] and reinforcement learning [25, 26]. Both of these approaches have in common that instead of fitting the NN to the exact closure term (which is what we will refer to as ‘derivative fitting’), one optimizes directly with respect to how well the solution is reproduced. This has been shown to yield more accurate and stable closure models [14, 15, 24]. The difference between these two methods is that trajectory fitting uses exact gradients, such that gradient-based optimizers can be applied to optimize the NN weights [27]. Reinforcement learning does not require these gradients. This makes it suitable for non-differentiable processes such as chess and self-driving cars [28].

These approaches have all been applied to the Navier-Stokes equations. However, in this paper we consider a 1D1\mathrm{D} simplification, namely Burgers’ equation. Several studies have been carried out which apply machine learning to this equation. In [29] physics-informed neural networks (PINNs) were successfully applied to Burgers’ equation. In PINNs the PDE is encoded into the loss function of the neural network. The advantage of PINNs is that they allow us to approximate the solution without explicitly discretizing space and time. In the context of closure modeling, several studies have been carried out. For example, in [30] and [31] a neural network was fitted to predict the closure term for Burgers’ equation with forcing. In [31] the trained neural network was successfully applied to unseen viscosity values. This was achieved by limited retraining on new data. This technique is known as transfer learning. Furthermore, several studies have been carried out which show the benefits of trajectory fitting [24, 32, 33]. In [32] trajectory fitting is combined with a Fourier neural operator to predict a spatially dependent Smagorinsky coefficient. This combination outperformed the other considered approaches. It also guarantees stability by being strictly dissipative. Furthermore, Fourier neural operators have the advantage that they are grid independent [34].

However, none of the discussed approaches leads to a provably stable NN closure model, while still allowing for backscatter. In addition, they do not guarantee abidance by the underlying energy conservation law. The latter is something that to our knowledge does not yet exist in the case of LES closure models. To resolve these shortcomings, we present a new NN closure model that satisfies both momentum and kinetic energy conservation and is therefore stable by design. As stated earlier, the difficulty of this task mainly lies in the fact that: (i) The kinetic energy conservation law includes terms which depend on the SGS content which is too expensive to simulate directly. (ii) Consequently, the kinetic energy of the large scales is not a conserved quantity (in the limit of vanishing viscosity). To tackle these issues we take the ‘discretize first, filter next’ approach [23, 24]. This means that we start from a high-resolution discretization with NN degrees of freedom, to which we apply a discrete filter. This filter projects the solution onto a coarse computational grid of dimension II, with I≪NI \ll N. Given the discrete filter, the exact closure term can be obtained by computing the commutator error. The main advantage of this approach is that the closure term now also accounts for the discretization error. Based on the filter’s properties we then derive an energy conservation law that can be split into two components: one that depends solely on the resolved scales (resolved energy) and another that solely depends on the SGS content (SGS energy) [13]. Like in existing works the closure model is represented by a convolutional neural network (CNN) [35]. The main novelty comes from the addition of a set of SGS variables. These SGS variables represent the SGS energy, projected onto the coarse grid. The key insight is that the resulting system of equations should still conserve energy in the inviscid limit. We then choose our CNN architecture such that it is consistent with this limit. In this way we still allow for backscatter without sacrificing stability.

The paper is structured in the following way: In section 2 we discuss Burgers’ and Korteweg-de Vries equation and their energy and momentum conservation properties. We introduce the discrete filter, the resulting closure problem, and derive a new energy conservation law. This law describes the exchange between the resolved and the SGS energy. In section 3 we introduce our novel machine learning framework for modeling the closure term. This approach satisfies the derived energy conservation law using the set of SGS variables to represent the SGS energy. In addition, we show how to satisfy momentum conservation. In section 4 we study the convergence and stability of our closure modeling framework and compare this to a vanilla CNN. We also analyze its structure-preserving properties in terms of momentum and energy conservation and its ability to extrapolate in space and time. In section 5 we present a short discussion on the applicability of our framework to the Navier-Stokes equations. In section 6 we conclude our work.

Governing equations, discrete filtering, and closure problem

Before constructing a machine learning closure, we formulate a description of the closure problem on the discrete level. For this purpose we introduce the filter and reconstruction operator which we apply to the discrete solution. In this way we also account for the discretization error. In addition, we discuss the effects of filtering on the physical structure of the system.

Spatial discretization

We consider an initial value problem of the following form:

∂u∂t=f(u),(1)\frac{\partial u}{\partial t}=f(u), \tag*{(1)}
u(x,0)=u0(x),(2)u(\mathbf{x},0)=u_0(\mathbf{x}), \tag*{(2)}

which describes the evolution of some quantity u(x,t)u(\mathbf{x},t) in space x∈Ω\mathbf{x} \in\Omega and time tt on the spatial domain Ω⊆Rd\Omega\subseteq\mathbb{R}^{d}, given initial state u0u_0. The dynamics of the system is governed by the right-hand side (RHS) f(u)f(u), which typically involves partial derivatives of uu with respect to x\mathbf{x}. After spatial discretization (method of lines), we obtain the vector u(t)∈RN\mathbf{u}(t) \in\mathbb{R}^{N}. The elements uiu_i of this vector approximate the value of uu at each of the NN grid points xi∈Ω\mathbf{x}_i \in\Omega for i=1,…,Ni=1,\ldots,N, such that ui≈u(xi)u_i \approx u(\mathbf{x}_i). The discrete analogue of the IVP is then

dudt=fh(u),(3)\frac{\mathrm{d}\mathbf{u}}{\mathrm{d}t}=f_h(\mathbf{u}), \tag*{(3)}
u(0)=u0,(4)\mathbf{u}(0)=\mathbf{u}_0, \tag*{(4)}

where fhf_h represents a spatial discretization of ff. It is assumed that all the physics described by equation (1) is captured in the discrete solution u\mathbf{u}. This means that whenever the physics involves a wide range of spatial scales, a very large number of degrees of freedom NN is needed to adequately resolve all the scales. This results in a large amount of computational resources required to solve these equations. This is what we aim to alleviate.

Burgers’ and Korteweg-de Vries equation, and physical structure

We are interested in the modeling and simulation of turbulent flows. For this purpose, we first consider Burgers’ equation, a 1D simplification of the Navier-Stokes equations. Burgers’ equation describes the evolution of the velocity u(x,t)u(x,t) according to partial differential equation (PDE)

∂u∂t=−12∂u2∂x+∂∂x(ν∂u∂x).(5)\frac{\partial u}{\partial t}=-\frac{1}{2}\frac{\partial u^2}{\partial x}+\frac{\partial}{\partial x}\left(\nu\frac{\partial u}{\partial x}\right). \tag*{(5)}

The first term on the RHS represents nonlinear convection and the second term diffusion, weighted by the positive viscosity field ν(x)≥0\nu(x) \ge0. This equation expresses similar behavior to 3D turbulence in the fact that smaller scales are created by the nonlinear convective term, which then dissipate through diffusion [36]. We will be interested in two properties of the Burgers’ equation, which we collectively call ‘structure’.

Firstly, momentum PP is conserved on periodic domains:

dPdt=ddt∫ΩudΩ⏟=:P=∫Ω(−12∂u2∂x+∂∂x(ν∂u∂x))dΩ=0.(6)\frac{\mathrm{d}P}{\mathrm{d}t} = \frac{\mathrm{d}}{\mathrm{d}t}\underbrace{\int_{\Omega} u\mathrm{d}\Omega}_{=:P} = \int_{\Omega} \left(-\frac{1}{2}\frac{\partial u^2}{\partial x} + \frac{\partial}{\partial x}\left(\nu\frac{\partial u}{\partial x}\right)\right)\mathrm{d}\Omega= 0. \tag*{(6)}

Secondly, on periodic domains (kinetic) energy EE is conserved in the absence of viscosity:

dEdt=ddt12∫Ωu2dΩ⏟=:E=∫Ω(−u2∂u2∂x+u∂∂x(ν∂u∂x))dΩ=−∫Ων(∂u∂x)2dΩ≤0,(7)\frac{\mathrm{d}E}{\mathrm{d}t} = \frac{\mathrm{d}}{\mathrm{d}t}\underbrace{\frac{1}{2}\int_{\Omega} u^2\mathrm{d}\Omega}_{=:E} = \int_{\Omega}\left(-\frac{u}{2}\frac{\partial u^2}{\partial x} + u\frac{\partial}{\partial x}\left(\nu\frac{\partial u}{\partial x}\right)\right)\mathrm{d}\Omega= -\int_{\Omega}\nu\left(\frac{\partial u}{\partial x}\right)^2\mathrm{d}\Omega\le0, \tag*{(7)}

where we used integration by parts. As we solely deal with viscous flows, i.e. ν(x)>0\nu(x) > 0, we disregard energy loss due to the presence of shocks [37]. Note that these conservation laws only hold in the absence of forcing.

These properties can be preserved in a discrete setting by employing a structure-preserving scheme [37]. On a uniform grid the convective term is discretized with the following skew-symmetric scheme:

(C(u)u)i=−13h(ui+12−ui−12)−13hui(ui+1−ui−1),(8)(C(\mathbf{u})\mathbf{u})_i = -\frac{1}{3h}(u_{i+1}^2-u_{i-1}^2)-\frac{1}{3h}u_i(u_{i+1}-u_{i-1}), \tag*{(8)}

where hh is the grid spacing. The skew-symmetry entails that uTC(u)u=0\mathbf{u}^{T}C(\mathbf{u})\mathbf{u}=0. This is used later to derive energy conservation. Furthermore, the diffusion operator is discretized as

(−QTdiag⁡(ν)Qu)i=1h2(νi(ui+1−ui)+νi−1(ui−1−ui)),(9)(-\mathbf{Q}^{T}\operatorname{diag}(\nu)\mathbf{Q}\mathbf{u})_i = \frac{1}{h^2}\left(\nu_i(u_{i+1}-u_i)+\nu_{i-1}(u_{i-1}-u_i)\right), \tag*{(9)}

where Q\mathbf{Q} is a simple forward difference approximation of the first derivative and νi=ν(xi)\nu_i=\nu(x_i) [3, 38]. In this paper we deal with constant viscosity ν(x)=ν\nu(x)=\nu such that we obtain the following system of ordinary differential equations (ODEs):

dudt=C(u)u+νDu.(10)\frac{\mathrm{d}\mathbf{u}}{\mathrm{d}t}=C(\mathbf{u})\mathbf{u}+\nu\mathbf{D}\mathbf{u}. \tag*{(10)}

Here D=−QTQ\mathbf{D}=-\mathbf{Q}^{T}\mathbf{Q} corresponds to a simple central difference approximation of the second derivative. For the time discretization we employ an explicit RK4 scheme [39].

This discretization conserves the discrete momentum Ph=h1TuP_h=h\mathbf{1}^{T}\mathbf{u} in the periodic case:

dPhdt=h1Tdudt=0,(11)\frac{\mathrm{d}P_h}{\mathrm{d}t}=h\mathbf{1}^{T}\frac{\mathrm{d}\mathbf{u}}{\mathrm{d}t}=0, \tag*{(11)}

where 1\mathbf{1} is a column vector with all entries equal to one. Note that this equation discretely represents the integral in (6). Furthermore, due to the skew-symmetry of the convection operator the evolution of the discrete kinetic energy Eh=h2uTuE_h=\frac{h}{2}\mathbf{u}^{T}\mathbf{u} is given by:

Burgers’ equation:

dEhdt=huTdudt=hνuTDu=−hν∥Qu∥22≤0.(12)\frac{\mathrm{d}E_h}{\mathrm{d}t}=h\mathbf{u}^{T}\frac{\mathrm{d}\mathbf{u}}{\mathrm{d}t}=h\nu\mathbf{u}^{T}\mathbf{D}\mathbf{u}=-h\nu\lVert\mathbf{Q}\mathbf{u}\rVert_2^2\le0. \tag*{(12)}

This is the discrete equivalent of (7). In both the continuous and discrete formulation we used the product rule to obtain the derivative. The norm ∥.∥2\lVert.\rVert_2 represents the conventional 2-norm. From (12) we conclude that this discretization ensures net kinetic energy dissipation, and conservation in the inviscid limit. From this point forward we will refer to the kinetic energy simply as energy.

In addition to Burgers’ equation we will consider the Korteweg-de Vries (KdV) equation:

∂u∂t=−ϵ2∂u2∂x−μ∂3u∂x3,(13)\frac{\partial u}{\partial t}=-\frac{\epsilon}{2}\frac{\partial u^2}{\partial x}-\mu\frac{\partial^3u}{\partial x^3}, \tag*{(13)}

where ε\varepsilon and μ\mu are scalar parameters. The KdV equation conserves momentum and energy irrespective of the values of ε\varepsilon and μ\mu. We discretize the nonlinear term in the same way as for Burgers’ equation, using the skew-symmetric scheme. The third-order spatial derivative is approximated by a skew-symmetric central difference stencil: (−ui−2+2ui−1−2ui+1+ui+2)/(2h3)(-u_{i-2} + 2u_{i-1} - 2u_{i+1} + u_{i+2})/(2h^3), see [40]. The resulting discretization is not only momentum conserving, but also energy conserving:

KdV equation:dEhdt=0.(14)\text{KdV equation:}\qquad\frac{\mathrm{d}E_h}{\mathrm{d}t}=0. \tag*{(14)}

Discrete filtering

In order to alleviate the high computational expenses for large NN we apply a spatial averaging filter to the fine-grid solution u\mathbf{u}. This results in the coarse-grid approximation uˉ∈RI\bar{\mathbf{u}} \in\mathbb{R}^{I}. The coarse grid follows from dividing Ω\Omega into II non-overlapping cells Ωi\Omega_i with cell centers Xi\mathbf{X}_i. The coarse grid is refined into the fine grid by splitting each Ωi\Omega_i into JJ subcells ωij\omega_{ij} with cell centers xij\mathbf{x}_{ij}. The subdivision is intuitively pictured in Figure 1, for a uniform 1D grid. Furthermore, we define the mass matrices ω∈RN×N\boldsymbol{\omega} \in\mathbb{R}^{N \times N} and Ω∈RI×I\boldsymbol{\Omega} \in\mathbb{R}^{I \times I} which contain the volumes of the fine and coarse cells on the main diagonal, respectively.

Figure 1. Subdivision of the spatial grid. The dots represent cell centers xij\mathbf{x}_{ij} and Xi\mathbf{X}_i for N=9N=9 and I=J=3I=J=3.

To reduce the degrees of freedom of the system we apply a spatial averaging filter to u\mathbf{u}. This filter simply computes uˉ\bar{\mathbf{u}} as a weighted average of u\mathbf{u}, within each coarse cell. We represent this by the following matrix vector product:

uˉ=Wu,(15)\bar{\mathbf{u}}=\mathbf{W}\mathbf{u}, \tag*{(15)}

where W∈RI×N\mathbf{W} \in\mathbb{R}^{I \times N} is the filter. The filter is defined as

W=Ω−1Oω(16)\mathbf{W}=\boldsymbol{\Omega}^{-1}\mathbf{O}\boldsymbol{\omega} \tag*{(16)}

with overlap matrix O∈RI×N\mathbf{O} \in\mathbb{R}^{I \times N}:

O:=[1⋯1⋱⋱⋱1⋯1].(17)\mathbf{O}:= \begin{bmatrix} 1 & \cdots& 1 \\ & \ddots& \ddots& \ddots\\ & & 1 & \cdots& 1 \end{bmatrix}. \tag*{(17)}

This matrix contains ones at index (i,j)(i,j) if ωij\omega_{ij} lies in Ωi\Omega_i. Mathematically, uˉ\bar{\mathbf{u}} can be regarded as a representation of u\mathbf{u} in a reduced basis. To project back onto the original basis we use a reconstruction operator R\mathbf{R}:

R:=OT(18)\mathbf{R}:=\mathbf{O}^{T} \tag*{(18)}

which is a right inverse of W\mathbf{W}, i.e.

WR=I.(19)\mathbf{W}\mathbf{R}=\mathbf{I}. \tag*{(19)}

The matrix R\mathbf{R} approximates the reconstruction by a piece-wise constant function [41]. This is intuitively pictured in Figure 2. Filtering a reconstructed solution Ruˉ\mathbf{R}\bar{\mathbf{u}} leaves uˉ\bar{\mathbf{u}} unchanged, i.e.

Fine-grid, reconstructed, and subgrid-scale contents, and energy before and after filtering

Figure 2. (Left) Fine-grid u\mathbf{u}, reconstructed Ru‾\mathbf{R}\overline{\mathbf{u}}, and SGS content u′\mathbf{u}' for u=sin⁡(x)u = \sin(x). Here N=1000N = 1000, I=20I = 20, and J=50J = 50. The SGS content in the fourth coarse cell μ4\boldsymbol{\mu}_4 is also indicated. (Right) Energy during a simulation of KdV equation with periodic BCs before and after filtering.

uˉ=(WR)⏟=IpWu(20)\bar{\mathbf{u}}=\underbrace{(\mathbf{W}\mathbf{R})}_{=\mathbf{I}}^{p}\mathbf{W}\mathbf{u} \tag*{(20)}

for p∈N0p \in\mathbb{N}_0. We will refer to this property as the ‘projection’ property, as it is similar to how repeated application of a projection operator leaves a vector unchanged.

By subtracting Ru‾\mathbf{R}\overline{\mathbf{u}} from u\mathbf{u} we obtain the subgrid-scale (SGS) content u′∈RN\mathbf{u}' \in\mathbb{R}^{N}:

u′:=u−Ru‾.(21)\mathbf{u}' := \mathbf{u} - \mathbf{R}\overline{\mathbf{u}}. \tag*{(21)}

In theory, one could define a more accurate R\mathbf{R}, e.g. through polynomial reconstruction or data-driven approaches [23], and obtain a smaller u′\mathbf{u}'. However, this particular choice of R\mathbf{R} is made such that the energy is invariant under reconstruction, as will be shown in equation (27). This is an important property for our methodology to work. Furthermore, we will refer to the SGS content in a single coarse cell Ωi\Omega_i as μi∈RJ\boldsymbol{\mu}_i \in\mathbb{R}^{J}, see Figure 2. Applying the filter to u′\mathbf{u}' yields zero:

Wu′=Wu−WR⏟=Iu‾=u‾−u‾=0Ω.(22)\mathbf{W}\mathbf{u}' = \mathbf{W}\mathbf{u} - \underbrace{\mathbf{W}\mathbf{R}}_{=\mathbf{I}}\overline{\mathbf{u}} = \overline{\mathbf{u}} - \overline{\mathbf{u}} = \mathbf{0}_{\Omega}. \tag*{(22)}

where 0Ω\mathbf{0}_{\Omega} is a vector with all entries equal to zero, defined on the coarse grid. This can be seen as the discrete equivalent of a property of a Reynolds operator [5]. To illustrate, we display each of the introduced quantities for a 1D sinusoidal wave in Figure 2.

Discrete closure problem

Next, we consider the time evolution of u‾\overline{\mathbf{u}}. Since we employ a spatial filter which does not depend on time, filtering and time-differentiation commute:

Wdudt=d(Wu)dt.\mathbf{W}\frac{\mathrm{d}\mathbf{u}}{\mathrm{d}t} = \frac{\mathrm{d}(\mathbf{W}\mathbf{u})}{\mathrm{d}t}.

The closure problem arises because this is not true for the spatial discretization, i.e.

Wfh(u)≠fH(Wu)(23)\mathbf{W}f_h(\mathbf{u}) \ne f_H(\mathbf{W}\mathbf{u}) \tag*{(23)}

where fHf_H represents the same discretization scheme as fhf_h, but on the coarse grid. The closure problem is that the equations for u‾\overline{\mathbf{u}} are ‘unclosed’. This means that knowing the fine-grid solution u\mathbf{u} is required to evolve u‾\overline{\mathbf{u}} in time. In this way we do not achieve any computational speedup.

To resolve this we write the filtered system in closure model form:

du‾dt=fH(u‾)+(Wfh(u)−fH(u‾))⏟=:c(u),(24)\frac{\mathrm{d}\overline{\mathbf{u}}}{\mathrm{d}t} = f_H(\overline{\mathbf{u}}) + \underbrace{\left(\mathbf{W}f_h(\mathbf{u}) - f_H(\overline{\mathbf{u}})\right)}_{=:c(\mathbf{u})}, \tag*{(24)}

where c(u)∈RI\mathbf{c}(\mathbf{u}) \in\mathbb{R}^{I} is the closure term. Note that this equation is still exact. c(u)\mathbf{c}(\mathbf{u}) is essentially the discrete equivalent of the commutator error in LES [5]. One advantage of having first discretized the problem is that c(u)\mathbf{c}(\mathbf{u}) also includes the discretization error, with respect to a fine-grid simulation. The aim in closure modeling is generally to approximate c(u)\mathbf{c}(\mathbf{u}) by a closure model c~(uˉ)\tilde{\mathbf{c}}(\bar{\mathbf{u}}). In § 3 we choose to represent c~\tilde{\mathbf{c}} by a neural network.

Inner products and energy decomposition

To describe the total energy that is present in the system, we define the following inner products and norms:

(a,b)ξ:=aTξb(25)(\mathbf{a},\mathbf{b})_{\xi} := \mathbf{a}^{T}\boldsymbol{\xi}\mathbf{b} \tag*{(25)}
∥a∥ξ2:=(a,a)ξ(26)\lVert\mathbf{a}\rVert_{\xi}^{2} := (\mathbf{a},\mathbf{a})_{\xi} \tag*{(26)}

for ξ∈{ω,Ω}\xi\in\{\omega,\Omega\}, and vectors a\mathbf{a} and b\mathbf{b}. With this notation we can represent the inner product on both the fine and coarse grid. For ξ=I\xi= \mathbf{I} we obtain the conventional inner product and 2-norm, denoted as (a,b)=aTb(\mathbf{a},\mathbf{b}) = \mathbf{a}^{T}\mathbf{b} and ∥a∥2\lVert\mathbf{a}\rVert^{2}. Besides the projection property (20) an additional characteristic of the filter/reconstruction pair is that the inner product is conserved under reconstruction (see Appendix § 10):

(Raˉ,Rbˉ)ω=(aˉ,bˉ)Ω.(27)(\mathbf{R}\bar{\mathbf{a}},\mathbf{R}\bar{\mathbf{b}})_{\omega} = (\bar{\mathbf{a}},\bar{\mathbf{b}})_{\Omega}. \tag*{(27)}

Using this, the total energy EhE_h in the system can be decomposed as

Eh:=12∥u∥ω2=12∥Ruˉ+u′∥ω2=12∥Ruˉ∥ω2+(Ruˉ,u′)ω+12∥u′∥ω2=12∥uˉ∥Ω2+12∥u′∥ω2(28)\begin{aligned} E_h := \frac{1}{2}\lVert\mathbf{u}\rVert_{\omega}^{2} = \frac{1}{2}\lVert\mathbf{R}\bar{\mathbf{u}}+\mathbf{u}'\rVert_{\omega}^{2} \\ &= \frac{1}{2}\lVert\mathbf{R}\bar{\mathbf{u}}\rVert_{\omega}^{2}+(\mathbf{R}\bar{\mathbf{u}},\mathbf{u}')_{\omega}+\frac{1}{2}\lVert\mathbf{u}'\rVert_{\omega}^{2} = \frac{1}{2}\lVert\bar{\mathbf{u}}\rVert_{\Omega}^{2}+\frac{1}{2}\lVert\mathbf{u}'\rVert_{\omega}^{2} \tag*{(28)} \end{aligned}

where we replaced u\mathbf{u} by the decomposition in (21). Furthermore, we used the fact that Ruˉ\mathbf{R}\bar{\mathbf{u}} is orthogonal to u′\mathbf{u}' to simplify the expression, see Appendix § 10. The final expression shows that our choice of W\mathbf{W} and R\mathbf{R} is such that the total energy of the system can be split into two parts. These constitute of the resolved energy Eˉh\bar{E}_h, which exclusively depends on uˉ\bar{\mathbf{u}}, and the SGS energy Eh′E'_h, which exclusively depends on u′\mathbf{u}'. The energy conservation law can also be decomposed into a resolved and SGS part:

dEhdt=dEˉhdt+dEh′dt=(uˉ,duˉdt)Ω+(u′,du′dt)ω=0,(29)\frac{\mathrm{d}E_h}{\mathrm{d}t} = \frac{\mathrm{d}\bar{E}_h}{\mathrm{d}t}+\frac{\mathrm{d}E'_h}{\mathrm{d}t} = \left(\bar{\mathbf{u}},\frac{\mathrm{d}\bar{\mathbf{u}}}{\mathrm{d}t}\right)_{\Omega}+\left(\mathbf{u}',\frac{\mathrm{d}\mathbf{u}'}{\mathrm{d}t}\right)_{\omega}=0, \tag*{(29)}

where we used the product rule to arrive at this relation. For Burgers’ equation with ν>0\nu> 0, the last equality sign changes to ≤\leq. This means that even for dissipative systems the resolved energy can still increase (so-called ‘backscatter’), as long as the total energy is decreasing. For the KdV equation (13), which is strictly energy conserving, this decomposition can be seen in Figure 2. Here one can clearly see the continuous exchange of energy between Eˉh\bar{E}_h and Eh′E'_h, while the sum of the two remains constant.

Momentum conservation

Next to the energy, we investigate the effect of filtering on the momentum. The total discrete momentum is given by

Ph=(1ω,u)ω,(30)P_h = (\mathbf{1}_{\omega},\mathbf{u})_{\omega}, \tag*{(30)}

where 1ω\mathbf{1}_{\omega} is a vector with all entries equal to one, defined on the fine grid. From this definition we can show, see Appendix § 10, that the discrete momentum is invariant upon filtering, i.e.

(1ω,u)ω=(1Ω,uˉ)Ω.(31)(\mathbf{1}_{\omega},\mathbf{u})_{\omega} = (\mathbf{1}_{\Omega},\bar{\mathbf{u}})_{\Omega}. \tag*{(31)}

This means the closure term does not add momentum into the system, i.e.

(1Ω,c(u))Ω=0.(32)(\mathbf{1}_{\Omega},\mathbf{c}(\mathbf{u}))_{\Omega} = 0. \tag*{(32)}

Structure-preserving closure modeling framework

In this section, the derived discrete energy and momentum balances will be used to construct a novel structure-preserving closure model. We will also discuss how to fit the parameters of the model.

The framework

Many existing closure approaches aim at approximating c(u)\mathbf{c}(\mathbf{u}) by a closure model c~(u‾;Θ)\widetilde{\mathbf{c}}(\overline{\mathbf{u}};\boldsymbol{\Theta}). Here Θ\boldsymbol{\Theta} are parameters to be determined such that the approximation is accurate. In this work, we propose a novel formulation, in which we extend the system of equations for u‾\overline{\mathbf{u}} with a set of II auxiliary SGS variables s∈RI\mathbf{s} \in\mathbb{R}^{I}. These SGS variables locally approximate the SGS energy, but projected onto the coarse grid. This will be detailed later. The extended system of equations has the form

ddt[u‾s]≈GΘ(u‾,s):=[fH(u‾)0Ω]+Ω2−1(K−KT)[u‾s]−Ω2−1QTQ[u‾s],(33)\frac{\mathrm{d}}{\mathrm{d}t} \begin{bmatrix} \overline{\mathbf{u}}\\ \mathbf{s} \end{bmatrix} \approx\mathbf{G}_{\Theta}(\overline{\mathbf{u}},\mathbf{s}) := \begin{bmatrix} \mathbf{f}_{H}(\overline{\mathbf{u}})\\ \mathbf{0}_{\Omega} \end{bmatrix} +\Omega_{2}^{-1}(\mathcal{K}-\mathcal{K}^{T}) \begin{bmatrix} \overline{\mathbf{u}}\\ \mathbf{s} \end{bmatrix} -\Omega_{2}^{-1}\mathcal{Q}^{T}\mathcal{Q} \begin{bmatrix} \overline{\mathbf{u}}\\ \mathbf{s} \end{bmatrix}, \tag*{(33)}

where K=K(u‾,s,Θ)∈R2I×2I\mathcal{K}=\mathcal{K}(\overline{\mathbf{u}},\mathbf{s},\boldsymbol{\Theta})\in\mathbb{R}^{2I\times2I} and Q=Q(u‾,s,Θ)∈R2I×2I\mathcal{Q}=\mathcal{Q}(\overline{\mathbf{u}},\mathbf{s},\boldsymbol{\Theta})\in\mathbb{R}^{2I\times2I} depend on the solution in a parameterized fashion. Next to the introduction of s\mathbf{s}, the second main novelty in this work is to formulate the closure model in terms of a skew-symmetric and a dissipative term. The skew-symmetric term is introduced to allow for exchange of energy between u‾\overline{\mathbf{u}} and s\mathbf{s}. The dissipative term is introduced to provide additional dissipation, as this is required (see Appendix § 11). The operators K\mathcal{K} and Q\mathcal{Q} will be modeled in terms of neural networks (NNs) with trainable parameters (contained in Θ\boldsymbol{\Theta}). So even though the notation in (33) suggests linearity of the closure model, the dependence of K\mathcal{K} and Q\mathcal{Q} on u‾\overline{\mathbf{u}} and s\mathbf{s} makes the model nonlinear. The construction of the introduced operators will be detailed in sections 3.3. As our energy definition includes Ω\Omega we include the inverse of the concatenated mass matrix Ω2−1\Omega_{2}^{-1} in our system of equations (33). This ensures energy conservation/dissipation regardless of the grid topology. This mass matrix is defined as

Ω2=[ΩΩ].(34)\Omega_{2}= \begin{bmatrix} \Omega& \\ & \Omega \end{bmatrix}. \tag*{(34)}

Given the extended system of equations, the total energy (28) is approximated as

Eh≈Es:=12∥a∥Ω22=12∥u‾∥Ω2+12∥s∥Ω2,(35)E_{h}\approx E_{s}:=\frac{1}{2}\lVert\mathbf{a}\rVert_{\Omega_{2}}^{2} =\frac{1}{2}\lVert\overline{\mathbf{u}}\rVert_{\Omega}^{2} +\frac{1}{2}\lVert\mathbf{s}\rVert_{\Omega}^{2}, \tag*{(35)}

where the second term approximates the SGS energy. Furthermore, we concatenate u‾\overline{\mathbf{u}} and s\mathbf{s} into a single state vector a∈R2I\mathbf{a}\in\mathbb{R}^{2I}:

a:=[u‾s].(36)\mathbf{a}:= \begin{bmatrix} \overline{\mathbf{u}}\\ \mathbf{s} \end{bmatrix}. \tag*{(36)}

The evolution equation for the approximated total energy is given by

dEsdt=(a,dadt)Ω2=(u‾,fH(u‾))Ω−∥Qa∥22,(37)\frac{\mathrm{d}E_{s}}{\mathrm{d}t} = \left(\mathbf{a},\frac{\mathrm{d}\mathbf{a}}{\mathrm{d}t}\right)_{\Omega_{2}} = \left(\overline{\mathbf{u}},\mathbf{f}_{H}(\overline{\mathbf{u}})\right)_{\Omega} -\lVert\mathcal{Q}\mathbf{a}\rVert_{2}^{2}, \tag*{(37)}

as the skew-symmetric term involving K−KT\mathcal{K}-\mathcal{K}^{T} cancels. This is a property of skew-symmetric matrices [38]. Consequently, this formulation guarantees stability provided that fH\mathbf{f}_{H} is structure-preserving.

Our key insight is that by explicitly including an approximation of the SGS energy we are able to satisfy the energy conservation balance, equation (29). The energy balance serves not only as an important constraint for the closure model (represented by a NN), but also guarantees stability of our closure model. This is because the energy is a norm of the solution which is bounded in time. This framework thus allows for the modeling of backscatter without sacrificing stability.

SGS variables

Next, let us consider appropriate expressions for s\mathbf{s}. The exact SGS energy on the coarse grid is given by:

W(u′)2(38)\mathbf{W}(\mathbf{u}')^2 \tag*{(38)}

where (⋅)2(\cdot)^2 is to be interpreted element-wise. This would yield s=±W(u′)2\mathbf{s} = \pm\sqrt{\mathbf{W}(\mathbf{u}')^2}, where ⋅\sqrt{\cdot} is also to be interpreted element-wise. The square root is taken to comply with the energy definition in (35). However, this definition for s\mathbf{s} resulted in poor performance during testing. We argue this was caused by the strict positivity (or negativity). Inevitable small errors in the model predictions caused some of the elements of s\mathbf{s} to switch sign. As the model was trained on positive s\mathbf{s}, the simulation quickly diverged from the true trajectory. Attempts at resolving this issue were not successful.

Instead we propose the use of a local linear compression. This formulation naturally allows for both positive and negative values of sis_i. This compression is written as (assuming a uniform grid):

si=tTμi,i=1,…,I,(39)s_i = \mathbf{t}^{T}\boldsymbol{\mu}_i,\qquad i = 1,\ldots,I, \tag*{(39)}

where we recall that μi∈RJ\boldsymbol{\mu}_i \in\mathbb{R}^{J} represents the SGS content in a single coarse cell Ωi\Omega_i. Furthermore, t∈RJ\mathbf{t} \in\mathbb{R}^{J} are the compression parameters. We aim to choose t\mathbf{t} such that we obtain s2≈W(u′)2\mathbf{s}^2 \approx\mathbf{W}(\mathbf{u}')^2. The optimal values of t\mathbf{t} are obtained using a singular value decomposition of the SGS content. This is outlined in Appendix C. From (39) we construct an operator Ts∈RI×N\mathbf{T}_s \in\mathbb{R}^{I \times N} which transform the SGS content into s\mathbf{s}:

s=Tsu′.(40)\mathbf{s} = \mathbf{T}_s\mathbf{u}'. \tag*{(40)}

Combining the compression with the filter, see (15), we define the operator T∈R2I×N\mathbf{T} \in\mathbb{R}^{2I \times N} which transforms u\mathbf{u} into the state vector a\mathbf{a}:

a=[WTs(I−RW)]⏟=:Tu.(41)\mathbf{a} = \underbrace{ \begin{bmatrix} \mathbf{W} \\ \mathbf{T}_s(\mathbf{I}-\mathbf{R}\mathbf{W}) \end{bmatrix}}_{=:\mathbf{T}} \mathbf{u}. \tag*{(41)}

Due to the linearity of the transformation we simply obtain

dadt=Tdudt,(42)\frac{\mathrm{d}\mathbf{a}}{\mathrm{d}t} = \mathbf{T}\frac{\mathrm{d}\mathbf{u}}{\mathrm{d}t}, \tag*{(42)}

where T\mathbf{T} is the Jacobian of the transformation. In the linear case this Jacobian does not depend on u\mathbf{u} which significantly simplifies computing reference data for dadt\frac{\mathrm{d}\mathbf{a}}{\mathrm{d}t}. In addition, it follows that if the true RHS includes a forcing term F∈RN\mathbf{F} \in\mathbb{R}^{N} we simply account for this by adding TF\mathbf{T}\mathbf{F} to the RHS of (33).

To illustrate how the compression works in practice we consider a snapshot from a simulation of Burgers’ equation (ν=0.01\nu= 0.01) with periodic BCs, see Figure 3. We observe that s\mathbf{s} serves as an energy storage for the SGS content, which is mainly present near shocks. If we look at the SGS energy trajectory we find that its behavior is captured both qualitatively and quantitatively by the SGS compression. Although we still miss some of the energy, the oscillations due to the traveling shock are nicely captured. From this we argue that a linear compression suffices. For more complex systems autoencoders might offer an alternative [42].

Figure 3. (Left) Learned SGS compression applied to Burgers’ equation for N=1000N = 1000, with I=20I = 20 and J=50J = 50. By filtering and applying the SGS compression the degrees of freedom of this system are effectively reduced from N=1000N = 1000 to 2I=402I = 40. (Right) True SGS energy and compressed SGS energy during this simulation of Burgers’ equation.

Construction of the operators

Similarly to the structure-preserving discretization for Burgers’ equation, presented in section 2.2, we want our closure model to locally advect momentum and energy through the domain. In this way we do not violate the conservation laws. It is therefore that we inspire our machine learning closure model on this structure-preserving discretization. In this section we outline how to construct K\mathcal{K} and Q\mathcal{Q} from the output of a convolutional neural network (CNN) [35].

Diffusive operator

Let us start by considering the diffusion operator in (9), namely −QTdiag⁡(ν)Q-\mathbf{Q}^{T}\operatorname{diag}(\boldsymbol{\nu})\mathbf{Q}. This operator locally diffuses energy and momentum through space at a rate which is determined by a positive viscosity field. In the periodic case the momentum is conserved, as 1TQT=0\mathbf{1}^{T}\mathbf{Q}^{T}=0. and the energy is dissipated, as −uTQTdiag⁡(ν)Qu=−∥diag⁡(ν)Qu∥22≤0-\mathbf{u}^{T}\mathbf{Q}^{T}\operatorname{diag}(\boldsymbol{\nu})\mathbf{Q}\mathbf{u}=-\lVert\sqrt{\operatorname{diag}(\boldsymbol{\nu})}\mathbf{Q}\mathbf{u}\rVert_{2}^{2}\leq0.

Drawing inspiration from this operator we introduce the following form for the diffusive term in our closure model:

Q(a;Θ)=q(a;Θ)B1(Θ)(43)\mathcal{Q}(\mathbf{a};\boldsymbol{\Theta})=\mathbf{q}(\mathbf{a};\boldsymbol{\Theta})\mathbf{B}_{1}(\boldsymbol{\Theta}) \tag*{(43)}

such that QTQ=B1Tq2B1\mathcal{Q}^{T}\mathcal{Q}=\mathbf{B}_{1}^{T}\mathbf{q}^{2}\mathbf{B}_{1} resembles the discussed diffusion operator. Here q(a;Θ)=diag⁡(q1,q2)∈R2I×2I\mathbf{q}(\mathbf{a};\boldsymbol{\Theta})=\operatorname{diag}(\mathbf{q}_{1},\mathbf{q}_{2})\in\mathbb{R}^{2I\times2I} is constructed from two output channels q1\mathbf{q}_{1} and q2\mathbf{q}_{2} of a CNN which takes u‾\overline{\mathbf{u}}, s\mathbf{s}, and fH(u‾)f_{H}(\overline{\mathbf{u}}) as inputs. fH(u‾)f_{H}(\overline{\mathbf{u}}) was added as input channel because it significantly improved the performance of the neural network. This is supported by [24]. Looking at the introduced form, q2\mathbf{q}^{2} can be thought of as a set of learned non-uniform and nonlinear viscosity fields. The square ensures positivity of these fields. Furthermore, the introduced matrix B1(Θ)∈R2I×2I\mathbf{B}_{1}(\boldsymbol{\Theta})\in\mathbb{R}^{2I\times2I} is a linear operator which encodes a set of parameterized convolutions. It will be used to satisfy momentum conservation, similarly to Q\mathbf{Q}. This will be discussed later in § 3.3.3.

Advective operator

For our skew-symmetric operator K−KT\mathcal{K}-\mathcal{K}^{T} we take a similar approach and introduce the following form:

K(a;Θ)=B2T(Θ)k(a;Θ)B3(Θ)(44)\mathcal{K}(\mathbf{a};\boldsymbol{\Theta})=\mathbf{B}_{2}^{T}(\boldsymbol{\Theta})\mathbf{k}(\mathbf{a};\boldsymbol{\Theta})\mathbf{B}_{3}(\boldsymbol{\Theta}) \tag*{(44)}

such that K−KT=B2TkB3−B3TkB2\mathcal{K}-\mathcal{K}^{T}=\mathbf{B}_{2}^{T}\mathbf{k}\mathbf{B}_{3}-\mathbf{B}_{3}^{T}\mathbf{k}\mathbf{B}_{2}. Similarly to the diffusive operator we use B2(Θ),B3(Θ)∈R2I×2I\mathbf{B}_{2}(\boldsymbol{\Theta}),\mathbf{B}_{3}(\boldsymbol{\Theta})\in\mathbb{R}^{2I\times2I} to satisfy momentum conservation. As is the case for q\mathbf{q}, the fields k(a;Θ)=diag⁡(k1,k2)\mathbf{k}(\mathbf{a};\boldsymbol{\Theta})=\operatorname{diag}(\mathbf{k}_{1},\mathbf{k}_{2}) are constructed from an additional set of two outputs channels of the CNN, i.e.

\begin{bmatrix}\mathbf{q}_{1} & \mathbf{q}_{2} & \mathbf{k}_{1} & \mathbf{k}_{2}\end{bmatrix}=\operatorname{CNN}(\overline{\mathbf{u}},\mathbf{s},f(\overline{\mathbf{u}});\boldsymbol{\Theta}). \tag*{(45)}\end{bmatrix}

This means the CNN has in total four output channels to construct both q\mathbf{q} and k\mathbf{k}. Furthermore, k\mathbf{k} can be thought of as a set of learned velocity fields which advect momentum and energy through the domain, in addition to exchanging energy between u‾\overline{\mathbf{u}} and s\mathbf{s}.

Momentum conservation

The entire framework (33) is summarized as

GΘ(a)=[fH(u‾)+Ω2−1(B2TkB3−B3TkB2)a−Ω2−1B1Tq2B1a](46)\mathcal{G}_{\Theta}(\mathbf{a}) = \begin{bmatrix} f_H(\overline{\mathbf{u}}) + \Omega_2^{-1}(\mathcal{B}_2^{\mathsf{T}}\mathbf{k}\mathcal{B}_3 - \mathcal{B}_3^{\mathsf{T}}\mathbf{k}\mathcal{B}_2)\mathbf{a} - \Omega_2^{-1}\mathcal{B}_1^{\mathsf{T}}\mathbf{q}^2\mathcal{B}_1\mathbf{a} \end{bmatrix} \tag*{(46)}

From this we find that momentum conservation, see (32), places an additional constraint on the operators:

([1Ω0Ω],GΘ(a))Ω2=[1Ω0Ω]T(B2TkB3−B3TkB2)a−[1Ω0Ω]TB1Tq2B1a=0,(47)\left(\begin{bmatrix}1_{\Omega}\\0_{\Omega}\end{bmatrix},\mathcal{G}_{\Theta}(\mathbf{a})\right)_{\Omega_2} = \begin{bmatrix}1_{\Omega}\\0_{\Omega}\end{bmatrix}^{\mathsf{T}} (\mathcal{B}_2^{\mathsf{T}}\mathbf{k}\mathcal{B}_3-\mathcal{B}_3^{\mathsf{T}}\mathbf{k}\mathcal{B}_2)\mathbf{a} - \begin{bmatrix}1_{\Omega}\\0_{\Omega}\end{bmatrix}^{\mathsf{T}} \mathcal{B}_1^{\mathsf{T}}\mathbf{q}^2\mathcal{B}_1\mathbf{a}=0, \tag*{(47)}

assuming that fHf_H is momentum conserving. This constraint is to be satisfied for periodic BCs, and we choose to compose Bi\mathcal{B}_i of convolutions (or stencils) expressed as linear operators:

Bi=[Bi11Bi12Bi21Bi22].(48)\mathcal{B}_i = \begin{bmatrix} \mathcal{B}_i^{11} & \mathcal{B}_i^{12}\\ \mathcal{B}_i^{21} & \mathcal{B}_i^{22} \end{bmatrix}. \tag*{(48)}

The operator Bi\mathcal{B}_i can therefore be thought of as the connection between two layers in a CNN, with each layer containing two channels [35]. In 1D each of the submatrices is characterized by 2B+12B+1 parameters, where B>0B>0 is the width of the convolution. In this way, each convolution takes into account BB neighboring grid cells from each side. Momentum conservation is ensured by constraining these submatrices in a clever way. From the constraint (47) and the definition of Bi\mathcal{B}_i, in (48), we find that momentum conservation is satisfied if the sum of the convolution weights for Bij1\mathcal{B}_i^{j1} is zero ∀i,j\forall i,j, as is the case for Q\mathbf{Q}. To see how this works we consider a general convolution operator B\mathbf{B}, characterized by parameters b−B,…,bB∈Rb_{-B},\ldots,b_B\in\mathbb{R}. Applying this operator to a discrete field f\mathbf{f}, while constraining the sum of the weights to zero, is achieved as follows:

(Bf)i=∑j=−BBb‾jfi+j,(49)(\mathbf{B}\mathbf{f})_i=\sum_{j=-B}^{B}\overline{b}_j f_{i+j}, \tag*{(49)}
b‾j=bj−∑k=−BBbk2B+1,(50)\overline{b}_j=b_j-\sum_{k=-B}^{B}\frac{b_k}{2B+1}, \tag*{(50)}

such that ∑j=−BBb‾j=0\sum_{j=-B}^{B}\overline{b}_j=0 indeed holds. Applying this procedure to Bij1\mathcal{B}_i^{j1}, ∀i,j\forall i,j, ensures (47) is satisfied up to machine precision, independent of the parameter values. The remaining convolutions are left unconstrained, i.e. we simply take b‾j=bj\overline{b}_j=b_j.

Properties & further discussion

The key insight is that we are free to choose any set of parameters Θ\Theta without violating the prescribed structure of the system. Furthermore, as K\mathcal{K} and Q\mathbf{Q} are based solely on convolutions in the CNN and the B\mathcal{B} matrices, they effectively correspond to nonlinear local stencils. The entire framework is therefore translation equivariant. For periodic boundary conditions (BCs) we apply circular padding to both the CNN inputs and the state vector [35]. For non-periodic BCs we refer to Appendix § 13. The parameters Θ\Theta include the weights of the CNN, as well as the parameters characterizing the B\mathcal{B} matrices. As both the CNN and the convolution operations in B\mathcal{B} are sparse, our model remains computationally efficient.

In Figure 4 the framework is applied to Burgers’ equation, where we compare it to the direct numerical simulation (DNS). It is once again interesting to see that ss is largest at the shocks, indicating the presence of significant SGS content. Comparing the magnitude of the different terms in (33), see Figure 5, we observe that the skew-symmetric term, that is responsible for redistributing the energy, is most important. In fact it is more important than the coarse-grid discretization. In other words, our closure model has learned dynamics that are highly significant to correctly predict the evolution of the filtered system. This means that even though the closure term is large, we can still accurately model it.

Four heatmaps comparing SP and DNS solutions for $u$ and $s$, with axes $t$ and $x$ and colorbars

Figure 4. A simulation of Burgers’ equation with periodic BCs using our trained structure-preserving closure model for I=20I = 20 and J=50J = 50 (left), along with the DNS solution for N=1000N = 1000 (right).

Figure 5. Magnitude of each of the different terms present in (33) corresponding to the simulation in Figure 4 with I=20I = 20, J=50J = 50, and N=1000N = 1000.

Finding the optimal parameter values

The optimal parameter values of the network can be obtained numerically by minimizing

L(Xu;Θ):=1p∑u∈Xu∥GΘ(Tu)−Tfh(u)∥22(51)\mathcal{L}(\mathbf{X}_u;\boldsymbol{\Theta}) := \frac{1}{p}\sum_{\mathbf{u}\in\mathbf{X}_u}\left\|\mathcal{G}_{\boldsymbol{\Theta}}(\mathbf{T}\mathbf{u})-\mathbf{T}f_h(\mathbf{u})\right\|_2^2 \tag*{(51)}

with respect to Θ\boldsymbol{\Theta} for the training set Xu\mathbf{X}_u containing pp DNS snapshots. We will refer to this approach as ‘derivative fitting’, as we minimize the residual between the predicted and the true RHS.

An alternative is to optimize Θ\boldsymbol{\Theta} such that the solution itself is accurately reproduced. To achieve this we minimize

Ln(Xu;Θ):=1pn∑u∈Xu∑i=1n∥SˉΘi(Tu)−TSi(Δt‾/Δt)(u)∥22.(52)\mathcal{L}_n(\mathbf{X}_u;\boldsymbol{\Theta}) := \frac{1}{pn}\sum_{\mathbf{u}\in\mathbf{X}_u}\sum_{i=1}^{n}\left\|\bar{\mathbf{S}}_{\boldsymbol{\Theta}}^i(\mathbf{T}\mathbf{u})-\mathbf{T}\mathbf{S}^{i(\overline{\Delta t}/\Delta t)}(\mathbf{u})\right\|_2^2. \tag*{(52)}

Here SˉΘi(Tu)\bar{\mathbf{S}}_{\boldsymbol{\Theta}}^i(\mathbf{T}\mathbf{u}) represents the output of the solver after successive application of an explicit time integration scheme for ii time steps, with step size Δt‾\overline{\Delta t}, starting from initial condition Tu\mathbf{T}\mathbf{u}, using the introduced closure model in (33). The DNS counterpart is indicated by Si(Δt‾/Δt)(u)\mathbf{S}^{i(\overline{\Delta t}/\Delta t)}(\mathbf{u}), with step size Δt\Delta t, starting from initial condition u\mathbf{u}. Note the appearance of the ratio Δt‾/Δt\overline{\Delta t}/\Delta t. This is because the coarse grid allows for larger time steps [43]. We will refer to this method as ‘trajectory fitting’. This approach has been shown to yield more accurate and stable closure models [14, 15, 22, 23, 24]. In this paper we propose a hybrid of the two approaches, as trajectory fitting is more computationally more expensive. This hybrid approach will be detailed later.

Results

Test cases

To test our closure modeling framework we consider the previously introduced Burgers’ equation with ν=0.01\nu= 0.01 on the spatial domain Ω=[0,2π]\Omega= [0, 2\pi] for two test cases: (i) periodic BCs without forcing and (ii) inflow/outflow (I/O) BCs with time-independent forcing. The implementation of BCs is discussed in Appendix § 13. We also consider a third test case: (iii) the KdV equation with ϵ=6\epsilon= 6 and μ=1\mu= 1 on the spatial domain Ω=[0,32]\Omega= [0, 32] for periodic BCs. Parameter values for Burgers’ and KdV are taken from [30]. Reference DNSs are carried out on a uniform grid of N=1000N = 1000 for Burgers’ and N=600N = 600 for KdV up to time T=10T = 10. The data that is generated from these reference simulations is split into a training (70%) and a validation set (30%). The simulation conditions (initial conditions, BCs, and forcing) for training are generated randomly, as described in Appendix § 14. The closure models will be tested on unseen simulation conditions, sampled from the same distributions as the training data. In addition to this, the construction of the training and validation set, the training procedure, and the hyperparameter tuning procedure are also described in Appendix § 14.

Considered closure models

For the analysis, we will compare our structure-preserving framework (SP) to a vanilla CNN. The vanilla CNN output is multiplied by a forward difference operator and the result is used as a closure model, i.e.

c~(u;Θ)=Q‾CNN⁡(u‾,fh(u‾);Θ),(53)\tilde{\mathbf{c}}(\mathbf{u};\boldsymbol{\Theta})=\overline{\mathbf{Q}}\operatorname{CNN}(\overline{\mathbf{u}},f_h(\overline{\mathbf{u}});\boldsymbol{\Theta}), \tag*{(53)}

where Q‾\overline{\mathbf{Q}} is a forward difference discretization of the first derivative on the coarse grid. This is done to satisfy momentum conservation [24, 32]. In addition, we consider a constant Smagorinsky model (SM) for use case (i) and (ii). This model contains a single scalar parameter CsC_s. This parameter is optimized in the same way as the NN parameters. We chose this model as it was the best performing non-machine learning closure in [32], outperforming the dynamic Smagorinsky model. After discretization, it takes the following form:

c~(u;Cs)=−Q‾Tdiag⁡(νt(Cs))Q‾u‾,(54)\tilde{\mathbf{c}}(\mathbf{u};C_s)=-\overline{\mathbf{Q}}^{T}\operatorname{diag}(\nu_t(C_s))\overline{\mathbf{Q}}\overline{\mathbf{u}}, \tag*{(54)}
νt(Cs)=(hCs)2∣Q‾u‾∣.(55)\nu_t(C_s)=(hC_s)^2\lvert\overline{\mathbf{Q}}\overline{\mathbf{u}}\rvert. \tag*{(55)}

such that νt\nu_t represents a parameterized and solution dependent viscosity. Here we have taken the filter width equal to the grid-spacing hh. Moreover, we also consider SP0 which initializes the simulation with s(t=0)=0Ω\mathbf{s}(t = 0) = \mathbf{0}_{\Omega} instead of the true s\mathbf{s}. This emulates the situation in which the true initial condition is unknown. Finally, we consider the no closure (NC) case, which corresponds to a coarse-grid solution of the PDEs. To march the solution forward in time we employ an explicit RK4 scheme [39] with time step size Δt=0.01\Delta t = 0.01 (4×4\times larger than the DNS) for Burgers’ and Δt=5×10−3\Delta t = 5\times10^{-3} (50×50\times larger than the DNS) for KdV.

To make a fair comparison we compare closure models with the same number of degrees of freedom (DOF). For SP we have DOF =2I= 2I, as we obtain an additional set of II degrees of freedom corresponding to the addition of the SGS variables. For the remaining closure models we simply have DOF =I= I.

Training the closure models

As use cases (i) and (ii) both correspond to Burgers’ equation we train the corresponding closure models on a dataset containing both simulation conditions. In this way we end up with a single closure model that works for both (i) and (ii). The SP closure models contain in total 2780 parameters (consisting of two hidden layers with each 20 channels and a kernel size of 5 for the underlying CNN) for Burgers’ equation and 5352 (consisting of two hidden layers with each 30 channels and a kernel size of 5) for KdV. The purely CNN-based closure models consist of 3261 parameters (two hidden layers with each 20 channels and a kernel size of 7). These settings are based on the hyperparameter tuning procedure in Appendix § 14. For KdV we omit the dissipative component in (33), as it is a conservative system. In between hidden layers we employ the ReLU activation function. We employ a linear activation function at the final layer. For Burgers’ we choose B=1B = 1 for the construction of the B\mathcal{B} matrices, matching the width of the coarse discretization. For KdV we do the same and therefore take B=2B = 2.

As stated earlier, the model parameters are optimized by first derivative fitting and then trajectory fitting. During testing we observed that solely derivative fitting resulted in instabilities and poor performance for the vanilla CNN, especially in the KdV case. In contrast, our SP method is guaranteed to be stable, and simply derivative fitting already resulted in reasonable performance. To maximize the potential of each closure model we decided to include trajectory fitting in the training procedure. The full procedure is outlined in Appendix § 14. We implemented our closure models in the Julia programming language [44] using the Flux.jl package [45, 46]. The code can be found at https://github.com/tobyvg/ECNCM_1D.

Closure model performance

We examine the performance of the trained closure models based on how well the filtered DNS solution is reproduced. For our comparison we will make extensive use of the normalized root-mean-squared error (NRMSE) metric. This metric is defined as

NRMSE⁡ uˉ(t)=1∣Ω∣∥uˉ(t)−uˉDNS(t)∥Ω2.(56)\operatorname{NRMSE}\ \bar{\mathbf{u}}(t) = \sqrt{\frac{1}{|\Omega|}\left\lVert\bar{\mathbf{u}}(t) - \bar{\mathbf{u}}^{\mathrm{DNS}}(t)\right\rVert_{\Omega}^{2}}. \tag*{(56)}

It is used to compare the approximated solution uˉ\bar{\mathbf{u}} at time tt, living on the coarse grid, to the filtered DNS result uˉDNS\bar{\mathbf{u}}^{\mathrm{DNS}}. We will refer to this metric as the solution error. In addition, we define the integrated-NRMSE (I-NRMSE) as

I-NRMSE⁡ uˉ=1T∑iΔt NRMSE⁡ uˉ(iΔt),0≤iΔt≤T,(57)\operatorname{I-NRMSE}\ \bar{\mathbf{u}} = \frac{1}{T}\sum_i \Delta t\ \operatorname{NRMSE}\ \bar{\mathbf{u}}(i\Delta t), \qquad0 \le i\Delta t \le T, \tag*{(57)}

such that the sum represents integrating the solution error in time. We will refer to this metric as the integrated solution error.

Convergence

As we refine the resolution of the coarse grid, and with this increase the number of DOF, we expect convergence of both the compression error Ls\mathcal{L}_s, see Appendix § 12, and the solution error. We consider DOF ∈{20,30,40,50,60,70,80,90,100}\in\{20, 30, 40, 50, 60, 70, 80, 90, 100\}. For each DOF value we train a different set of closure models. If NN is not divisible by II we first project the DNS result onto a grid with a resolution that is divisible by II before applying the filter. In total 36 closure models are trained: two (SP and CNN) for each combination of the 9 considered coarse-grid resolutions and equation (Burgers’ and KdV).

The SGS compression error evaluated over the validation set is shown in Figure 6. We observe monotonic convergence of the compression error as we refine the grid. We expect the compression error to further converge to zero, as we keep on refining. The faster convergence for the KdV equation is likely caused by the lower fine-grid resolution, as compared to Burgers’ equation.

Figure 6. Convergence of the SGS compression error LsL_s when refining the coarse grid, evaluated on the validation set for Burgers’ equation (N=1000N = 1000) and KdV equation (N=600N = 600). Both the effective DOF and the corresponding compression factor JJ are depicted on the x-axis.

Next, we consider the integrated solution error, see Figure 7. The presented plots represent an average taken over 20 simulations with unseen simulation conditions. We consider different numbers of DOF which enables us to evaluate the convergence. For test cases (i) and (ii) we observe almost monotonic convergence of the solution error for NC, SM, and SP/SP0. Furthermore, SP improves upon NC with roughly one order of magnitude, surpassing SM. SP0 only performs slightly worse than SP, but still converges relatively smoothly. On the other hand, the solution error for the CNN behaves quite erratically: sometimes more accurate than SP, and sometimes less accurate than NC (case (i), DOF = 90).

Figure 7. (Top) Integrated solution error evaluated at T=10T = 10 averaged over 20 simulations for the different use cases (i)–(iii) and an increasing number of DOF. Only stable simulations are considered for the depicted averages. Absence of a scatter point indicates none of the simulations were stable. (Bottom) Similar to the top plots, but depicting the average simulation time as a fraction of the DNS time. NC = no closure, CNN = convolutional neural network closure, SP = structure-preserving closure, SP0 = structure-preserving closure with s(t=0)=0Ωs(t = 0) = 0_{\Omega}, and SM = constant Smagorinsky model.

For test case (iii) (KdV) we find that for most numbers of DOF the CNN outperforms SP, while not resulting in stable closure models for DOF ∈{90,100}\in\{90, 100\}. Furthermore, for the lower numbers of DOF we observe slightly better performance for SP. From this we conclude that the compression error (see Figure 6) is likely not the limiting factor of the closure model performance. Looking at the difference between SP and SP0 it seems that initializing with the true s seems to lead to a larger difference, as compared to Burgers’ equation. As SGS energy does not dissipate in the KdV equation, but flows back into the resolved scales, the dependence on the true s is likely higher than for Burgers’.

Overall, the conclusion is that our proposed SP closure model leads to more robust simulations than the CNN, while still improving upon NC with roughly an order of magnitude.

Computation time

Looking at the relative computation times in Figure 7 we find that both SP and the CNN are at least 2×2\times faster than the DNS, for Burgers’ equation. Furthermore, the computation time for the CNN is slightly shorter, but seems to increase at a larger rate when increasing the DOF. On our laptop CPU the computation time of a single DNS took no longer than 5 seconds. For such small simulations we expect the computation time to be largely determined by computational overhead. We therefore expect the relative speedup to increase for larger systems. In fact, the training time amounts to roughly 20 minutes for each neural network. This is rather long compared to a DNS, however for more complex/larger systems we expect the significance of the training time to decrease, relative to the DNS time. Especially for 2D/3D problems. where the curse of dimensionality sets in, significant speedups can be expected [15, 18]. Also, once a model is trained it can be applied to unseen simulation conditions, without retraining.

For the KdV equation, the relative speedup is roughly 20×20\times. This is likely caused by the small time step size required for the DNS (Δt=10−4\Delta t = 10^{-4}). Larger time step sizes for the DNS consistently resulted in unstable simulations within a few time steps. In addition, we find that computing the true ss for the initial condition does not really affect the computation time.

Consistency of the training procedure

It is important to note that the closure models trained in the previous section possess a degree of randomness. This is caused by the (random) initialization of the network weights and the random selection of the mini-batches. This likely caused the irregular convergence behavior shown in the previous section. In order to evaluate this effect, we train 10 separate replica models for DOF=60\mathrm{DOF} = 60, which only differ in the random seed. The trained models are evaluated in terms of stability (number of unstable simulations) and integrated solution error. A simulation is considered unstable when it produces NaN values for uˉ\bar{u}. In total 20 simulations per closure model are carried out using the same simulation conditions as in the convergence study. The results are depicted in Figure 8. With regard to stability we observe that all trained SP closure models produced exclusively stable simulations. This is in accordance with the earlier derived stability condition (37). For the non-periodic test case (ii) we also observe a clear stability advantage, as compared to the CNN.

Scatter plots of integrated solution error and unstable simulations for Burgers’, Burgers’ (I/O), and KdV

Figure 8. Integrated solution error evaluated at T=10T = 10 averaged over 20 simulations and % of unstable simulations for each closure model in the trained ensemble of 10 closure models (DOF = 60). Use cases (i)-(iii) are considered. For (ii) two CNN closure models produced 100% unstable simulations and are therefore omitted from the graph. CNN = convolutional neural network closure and SP = structure-preserving closure.

Regarding this integrated solution error, we observe that the SP closure models all perform very consistently (errors are almost overlapping). The CNNs sometimes outperform SP, but also show very large outliers. This confirms our conclusion of the previous section that our SP closure models are much more robust than the CNNs, which can be ‘hit or miss’ depending on the randomness in the training procedure. However, we still find that SP is often outperformed by the CNN for test case (iii).

Structure preservation

As we formulated a structure-preserving closure model it is important to evaluate if the structure is indeed preserved. For this we consider a single simulation of the KdV equation with periodic BCs, for DOF = 40. This is an interesting test case, as the energy should be exactly conserved. In Figure 9 we look at both the change in momentum and energy during the simulation. For momentum conservation we

Figure 9. Change in momentum ΔPh(t)=Ph(t)−Ph(0)\Delta P_h(t) = P_h(t) - P_h(0) (left) and total energy ΔEh/s(t)=Eh/s(t)−Eh/s(0)\Delta E_{h/s}(t) = E_{h/s}(t) - E_{h/s}(0) (right) for a simulation of the KdV equation with periodic BCs starting from an unseen initial condition. The presented results correspond to DOF = 40. DNS = direct numerical simulation, CNN = convolutional neural network closure and SP = structure-preserving closure.

also include the CNN, as this satisfies momentum equation through multiplication by a discrete derivative operator, see (53). Regarding momentum conservation we find that both SP and the CNN indeed conserve momentum up to machine precision (single-precision). For energy conservation we observe the same for SP. From this we conclude that SP indeed preserves the relevant structure.

Furthermore, we also consider a single simulation of Burgers’ equation with periodic BCs, see Figure 10. Here we find that the total energy is always decreasing for SP. However, we find the resolved energy to oscillate due to backscatter, which matches the filtered DNS result. From this we conclude that, although the energies do not match the DNS result exactly, SP indeed allows for backscatter to be modeled correctly.

Resolved and total energy for a simulation of Burgers’ equation

Figure 10. Resolved and total energy for a simulation of Burgers’ equation with periodic BCs starting from an unseen initial condition. The presented results correspond to DOF = 40. DNS = direct numerical simulation and SP = structure-preserving closure.

Burgers’ equation & energy spectra

To find out what happens when a CNN becomes unstable we consider Burgers’ equation with periodic BCs, for DOF = 90. For this CNN we observed a rather large error spike in the convergence study (Figure 7). To analyze this error spike we consider a single simulation starting from an unseen initial condition. The results are depicted in Figure 11. Here we observe a large buildup of numerical noise for the CNN, as the simulation progresses. In addition, we observe that SP nicely suppresses the wiggles produced by NC around the shock [37]. SM also manages to do this, although to a lesser extent.

Solutions of Burgers’ equation with periodic BCs at different points in time

Figure 11. Solutions of Burgers’ equation with periodic BCs at different points in time, starting from an unseen initial condition. The solutions are produced by the different closure models corresponding to DOF = 90. DNS = direct numerical simulation, NC = no closure, CNN = convolutional neural network closure, SP = structure-preserving closure, and SM = constant Smagorinsky model.

Next, we look at the energy trajectories and energy spectra corresponding to this simulation, see Figure 12. The energy spectra are given as a function of the wavenumber kk. These spectra are computed by carrying out a discrete Fourier transform of uˉ\bar{u} and computing the energy for every kk [47]. Furthermore, the energy spectra are only depicted up to the wavenumber resolved by the closure models. We find that for the CNN the energy starts to diverge at around t=2t = 2. This worsens as the simulation progresses and a large increase in energy is observed. Looking at the energy spectrum this corresponds to a buildup of energy in the small scales (large kk) and the numerical noise observed in the solution. This can cause the simulation to blow up. Furthermore, we find that SM is too dissipative. This results mostly in a lack of energy in the small scales. In addition, NC is not dissipative enough, resulting in a slight buildup of energy in the small scales. Finally, SP seems to capture the energy balance nicely, looking at both the trajectory and the spectrum. SP, and to a lesser extent NC, show the expected k−2k^{-2} slope in the energy spectrum [48].

Figure 12. (Top) Resolved energies corresponding to the simulation in Figure 11 using the different closures for DOF = 90. For SP we depicted the total energy. The SP energy trajectory mostly overlaps with the DNS. (Bottom) The average energy spectra of this simulation, evaluated on the interval 3≤t≤73 \le t \le7. The left plot corresponds to I=DOFI = \mathrm{DOF} and the right plot to I=DOF/2I = \mathrm{DOF}/2. Furthermore, we also show the spectra of the unfiltered DNS. These slightly differ from the filtered DNS in the small wavenumbers, as our filter is not a spectral filter [47]. DNS = direct numerical simulation, NC = no closure, CNN = convolutional neural network closure, SP = structure-preserving closure, and SM = constant Smagorinsky model.

Extrapolation in parameter space

Next, we are interested in how well such closure models are capable of extrapolating in parameter space. In particular, we consider different viscosity values ν\nu for the Burgers’ equation. For this purpose we provide ν\nu as an additional input to the neural network. For the training data we consider ν∈[10−2,10−1.75,10−1.5,10−1.25,10−1]\nu\in[10^{-2}, 10^{-1.75}, 10^{-1.5}, 10^{-1.25}, 10^{-1}]. To generate the training data we randomly generate three different initial conditions and carry out a DNS for each considered ν\nu. To accommodate smaller values of ν\nu used in the extrapolation experiment (up to ν=10−3\nu= 10^{-3}) we refine the grid to N=5000N = 5000. Furthermore, we use Δt=10−5\Delta t = 10^{-5} for the time integration and simulate up to T=5T = 5. The smaller time step size is used to accommodate for the finer grid. Every 1000th time step we save the solution and use this as training data for the closure models. For both SP and the CNN we train three closure models which only differ in the random seed used for the training procedure and initial parameter values. The networks underlying the methodologies are kept small, namely only a single hidden layer with 20 channels. This is done, as smaller networks show higher potential for generalization [49]. The remaining hyperparameters and the training procedure are kept the same, see Appendix § 14. For our closure models we reduce the DOF of the system from 5000 to 50 (a 100 fold reduction). We also use a larger time step size of Δt=0.01\Delta t = 0.01 (a 1000 fold increase). We evaluate the performance of each of the closure models for three simulations starting from different initial conditions. For each initial condition we consider 9 different values of ν\nu in the range [10−310^{-3}, 10−110^{-1}]. The integrated solution error, see (57), is once again used as the performance metric. The results are depicted in Figure 13. For each model we find that the error increases for smaller values of ν\nu. This is to be expected as

Figure 13. Integrated solution error evaluated at T=5T = 5 averaged over the three trained closure models and three simulations for each viscosity value. The black vertical separates the training range (right) from the extrapolation range (left). NC = no closure, CNN = convolutional neural network closure, SP = structure-preserving closure, and SM = constant Smagorinsky model.

smaller values of ν\nu lead to stronger spatial gradients (shocks) which are harder to deal with in a numerical setting [37, 38]. When comparing the performance of the different closures we find that our proposed SP scheme consistently outperforms the other closure models. This is the case for both inter- and extrapolation. Regarding the CNN we observe a drop in performance as the viscosity decreases, even within the training range. It seems that the CNN is not able to adapt to the range of values of ν\nu, as previously the CNN was able to perform quite well for DOF = 50 and a single viscosity value, see Figure 7. Furthermore, we find that SM outperforms NC across the considered range. This means that the obtained Smagorinsky constant generalizes well. This is to be expected as it is a simple model containing only a single parameter [9, 49].

Extrapolation in space and time

As a final test case we evaluate how well the closure models are capable of extrapolating in space and time. We consider the KdV equation on an extended spatial domain Ω=[0,96]\Omega= [0, 96] and run the simulation until T=50T = 50. This corresponds to a 3×3\times and 5×5\times increase, respectively, with respect to the training data. We choose the KdV equation for this experiment as it is non-dissipative. It therefore leads to more interesting long-term behavior. As closure models, we use the ones trained during the convergence study that correspond to the grid-spacing of the grids employed in this test case.

Let us start by looking at snapshots from different points in the simulation, see Figure 14. At the edge of the training region t=10t = 10 we find that the solutions are still roughly aligned, except for NC which already contains a lot of numerical noise. At t=25t = 25 we find that both SP and the CNN start to diverge from the DNS. In addition, we observe a buildup of numerical noise for the CNN. This worsens at t=50t = 50, while SP seems to remain free of numerical noise.

Solutions of the KdV equation with periodic BCs at different points in time

Figure 14. Solutions of the KdV equation with periodic BCs at different point in time, starting from an unseen initial condition. The solutions are produced by the different closure models corresponding to DOF = 40. In this case, the spatial and temporal domain are increased 3× and 5× with respect to the training data. DNS = direct numerical simulation, NC = no closure, CNN = convolutional neural network closure, and SP = structure-preserving closure.

To make a more thorough analysis we consider the trajectories of the resolved energy. This is presented in Figure 15. We find that for SP the resolved energy (in blue) stays in close proximity to the filtered DNS (in magenta). This is in contrast to the CNN (in red) which starts to diverge from the DNS (in black) around t=5t = 5. The resolved energy for the CNN also exceeds the maximum allowed total energy EhE_h (in orange) at different points in the simulation, which is nonphysical. We conclude that adding the SGS variables and conserving the total energy helps with capturing the delicate energy balance between resolved and SGS energy that characterizes the DNS. It is also interesting to note that NC (in green) conserves the resolved energy, as the coarse discretization conserves the discrete energy.

Figure 15. Trajectory of the resolved energy Eˉh\bar{E}_h for the simulation presented in Figure 14 for each of the different models corresponding to DOF = 40. The DNS resolved energy is depicted for both II = DOF (to compare with the CNN) and II = DOF/2 (to compare with SP). DNS = direct numerical simulation, NC = no closure, CNN = convolutional neural network closure, and SP = structure-preserving closure.

To make a more quantitative analysis of this phenomenon we investigate the trajectory of the solution error and the Gaussian kernel density estimate (KDE) [50] of the resolved energy distributions. The latter analysis is carried out to analyze whether the closure models capture the correct energy balance between the resolved and SGS energy. The results for DOF ∈{40,60,80}\in\{40, 60, 80\} are depicted in Figure 16. Looking at the solution error trajectories we find that at the earlier stages of the simulation the CNN outperforms SP. However, SP slowly catches up with the CNN past the training region. With regards to the resolved energy distribution we find that for each of the considered numbers of DOF SP is capable reproducing the DNS distribution. On the other had, the CNN closure models struggle to capture this distribution. For DOF = 40 a significant part of the distribution even exceeds the total energy present in the DNS, i.e. there occurs a nonphysical influx of energy.

Figure 16. Solution error trajectory (top) and KDEs estimating the distribution of Eˉh\bar{E}_h (bottom) for the trained closure models corresponding to different numbers of DOF. These quantities are computed for a simulation of the KdV equation with the same initial condition on the extended spatial and temporal domain. In the top row the vertical black line indicates the maximum time present in the training data, while in the bottom row it indicates the total energy of the DNS (which should not be exceeded). The DNS resolved energy is again depicted for both I=DOFI = \mathrm{DOF} (to compare with the CNN) and I=DOF/2I = \mathrm{DOF}/2 (to compare with SP). NC = no closure, CNN = convolutional neural network closure, and SP = structure-preserving closure.

From this we conclude that both SP and the CNN closure models are capable of extrapolating beyond the training data. However, only SP is capable of correctly capturing the energy balance between the resolved and unresolved scales. This allows it to more accurately capture the statistics of the DNS result.

Discussion on the applicability to 2D/3D Navier-Stokes

A number of challenges arises when considering to apply the proposed methodology to the 2D/3D Navier-Stokes equations. Firstly, the discrete nature of the presented closure modeling framework, making it dependent on the compression factor JJ between the coarse and the fine-grid resolution. This is a result of taking the ‘discrete first’ approach. While this allows the method to be highly specialized to the discretization, it can limit the applicability of the closure model to grid pairs associated with JJ. One possibility is to train a single closure model with training data from multiple compression factors. Another possibility is to consider a continuous formulation of the closure problem, along with a continuous expression for the SGS variables. The finite element framework might be a useful starting point, as it approximates the solution using a continuous, but still highly local, basis [51]. In addition, it easily allows for non-Cartesian grids. The convolutional layers in the closure model could then be replaced by graph convolutions [52], which work for unstructured grids. Another useful alternative is to consider the Fourier neural operator, which promises to be discretization invariant [34]. Furthermore, one could do a sparse regression on the closure model to identify physical terms [53], and train the model for different JJ to understand the effect of discretization error. This would effectively separate the two sources of error. Note that our framework can also be applied to nonuniform Cartesian grids, as long as the compression factor JJ is constant, and an energy-conserving discretization is available. The size of the computational domain can even be increased by an arbitrary factor as compared to the training domain, without retraining the closure model. The latter was shown in section 4.8. For even more irregular grids graph neural networks could be used to increase the flexibility of the framework [52].

A second challenge is the linear compression for ss, which might be a limiting factor for 2D/3D flows. As an alternative, non-linear compression methods like autoencoders could be employed [42]. Furthermore, the natural extension of the compressed SGS energy ss to 2D and 3D is probably the sub-grid scale stress tensor. This means that our framework needs to be extended with multiple sub-grid scale equations, e.g. s1s_1, s2s_2, . . . Finally, we could draw inspiration from the scale-adaptive simulation method which also uses the SGS energy in its equations [54].

Conclusion

In this paper we proposed a novel way of constructing machine learning-based closure models in a structure-preserving fashion. We started by applying a spatial averaging filter to a fine-grid solution and writing the resulting system in closure model form. We showed that by applying this filter we effectively remove part of the energy. Next, we introduced a linear compression of the subgrid-scales (SGSs) into a set of SGS variables, defined on the coarse grid. These serve as a means of reintroducing the removed energy back into the system. This allows us to use the concept of kinetic energy conservation in closure modeling. In turn we introduced an extended system of equations which models the evolution of the filtered solution as well as the evolution of the SGS variables. For this extended system we proposed a structure-preserving closure modeling framework which allows for energy exchange between the filtered solution and the SGS variables, in addition to dissipation. This framework serves to constrain the underlying convolutional neural network (CNN) such that no additional energy enters the system. In this way we achieve stability by abiding by the underlying energy conservation law. The advantage is that the framework still allows for backscatter through the energy present in the SGS variables. In addition, momentum conservation is also satisfied. Finally, the framework was applied to both Burgers’ and Korteweg-de Vries (KdV) equation.

A convergence study showed that the learned SGS variables are able to accurately match the original SGS energy content, with accuracy consistently improving when refining the coarse-grid resolution.

Given the SGS compression, our proposed structure-preserving framework (SP) was compared to a vanilla CNN. Overall, our SP method performed roughly on par with the CNN in terms of accuracy, albeit the CNN outperformed SP slightly for the KdV equation. However, the results for the CNN were typically inconsistent, not showing clear convergence of the error upon increasing the grid resolution. In addition, the CNN suffered from stability issues. On the other hand, our SP method produced stable results in all cases, while also consistently improving upon the ‘no closure model’ result. To be more specific, it did so by roughly an order of magnitude in terms of reproducing the reference solution.

This conclusion was further strengthened by training an ensemble of closure models. This was done to investigate the consistency of the closure model performance with respect to the randomness inherent to the training procedure. We observed that the trained vanilla CNNs differed significantly in performance and stability, whereas the different SP models performed very similarly to each other. The SP closures also displayed no stability issues. Our SP framework has therefore shown to be more robust and successfully resolves the stability issues which plague conventional CNNs.

Our numerical experiments confirmed the structure-preserving properties of our method: exact momentum conservation, energy conservation up to a time discretization error, and a strictly decreasing energy in the presence of dissipation. We also showed that our method succeeds in accurately modeling backscatter in both Burgers’ and the KdV equation.

Regarding extrapolation in parameter space, out method outperformed both the CNN and the Smagorinsky model on viscosity values in- and outside the training range. From this we conclude our methodology leads to more accurate results than the conventional methods for this type of application. Furthermore, when extrapolating in space and time, the advantage of including the SGS variables and embedding structure-preserving properties became even more apparent: Our method is much better at capturing the delicate energy balance between the resolved and SGS energy. This in turn yielded better long-term error behavior. Based on these results we conclude that including the SGS variables, as well as adherence to the physical structure, has the important advantages of stability and long-term accuracy. In addition, it also leads to more consistent performance. This work therefore serves as an important starting point for building physical constraints into machine learning-based turbulence closure models. More generally, our framework is potentially applicable to a wide range of systems that possess multiscale behavior, while also possessing a secondary conservation law, for example incompressible pipe flow [55]. Currently our efforts are mainly directed towards the incompressible Navier-Stokes equations. For instance, Kolmogorov flow would be a good starting point [56].

CRediT authorship contribution

T. van Gastelen: Conceptualization, Methodology, Software, Writing - original draft. W. Edeling: Writing - review & editing. B. Sanderse: Conceptualization, Methodology, Writing - review & editing, Funding acquisition.

Data availability

The code used to generate the training data and the implementation of the neural networks can be found at https://github.com/tobyvg/ECNCM_1D.

Acknowledgements

This publication is part of the project “Unraveling Neural Networks with Structure-Preserving Computing” (with project number OCENW.GROOT.2019.044 of the research programme NWO XL which is financed by the Dutch Research Council (NWO)). Part of this publication is funded by Eindhoven University of Technology. Finally, we thank the reviewers for their feedback, enhancing the quality of the article.

Filter properties

Here we derive important properties of the spatial averaging filter. We first show that equation (27) holds:

(Ra‾,Rb‾)ω=a‾TRTωRb‾=a‾TΩWω−1ωRb‾=a‾TΩWRb‾⏟=I=(a‾,b‾)Ω,(58)(\overline{\mathbf{R}a},\overline{\mathbf{R}b})_{\omega}=\overline{a}^{T}\mathbf{R}^{T}\omega\mathbf{R}\overline{b}=\overline{a}^{T}\Omega\mathbf{W}\omega^{-1}\omega\mathbf{R}\overline{b}=\overline{a}^{T}\Omega\mathbf{W}\underbrace{\mathbf{R}\overline{b}}_{=\mathbf{I}}=(\overline{a},\overline{b})_{\Omega}, \tag*{(58)}

where we used that fact that

ΩWω−1=RT.(59)\Omega\mathbf{W}\omega^{-1}=\mathbf{R}^{T}. \tag*{(59)}

Next, we proof that Ru‾\mathbf{R}\overline{u} is orthogonal to u′\mathbf{u}':

(Ru‾,u′)ω=(Ru‾,u−Ru‾)ω=(Ru‾,(I−RW)u)ω=u‾TRTω(I−RW)u=u‾TΩW(I−RW)u=(u‾,(W−WRW⏟=W)u)Ω=(u‾,(W−W)u)Ω=0.(60)\begin{aligned} (\mathbf{R}\overline{u},\mathbf{u}')_{\omega}=(\mathbf{R}\overline{u},\mathbf{u}-\mathbf{R}\overline{u})_{\omega}=(\mathbf{R}\overline{u},(\mathbf{I}-\mathbf{R}\mathbf{W})\mathbf{u})_{\omega}=\overline{u}^{T}\mathbf{R}^{T}\omega(\mathbf{I}-\mathbf{R}\mathbf{W})\mathbf{u} \\ &=\overline{u}^{T}\Omega\mathbf{W}(\mathbf{I}-\mathbf{R}\mathbf{W})\mathbf{u}=(\overline{u},(\mathbf{W}-\underbrace{\mathbf{W}\mathbf{R}\mathbf{W}}_{=\mathbf{W}})\mathbf{u})_{\Omega}=(\overline{u},(\mathbf{W}-\mathbf{W})\mathbf{u})_{\Omega}=0. \tag*{(60)} \end{aligned}

Finally, we show that equation (31) holds:

(1ω,u)ω=1ωTωu=1ΩTRTωu=1ΩTΩWω−1ωu=1ΩTΩWu=(1Ω,Wu)Ω=(1Ω,u‾)Ω,(61)(\mathbf{1}_{\omega},\mathbf{u})_{\omega}=\mathbf{1}_{\omega}^{T}\omega\mathbf{u}=\mathbf{1}_{\Omega}^{T}\mathbf{R}^{T}\omega\mathbf{u}=\mathbf{1}_{\Omega}^{T}\Omega\mathbf{W}\omega^{-1}\omega\mathbf{u}=\mathbf{1}_{\Omega}^{T}\Omega\mathbf{W}\mathbf{u}=(\mathbf{1}_{\Omega},\mathbf{W}\mathbf{u})_{\Omega}=(\mathbf{1}_{\Omega},\overline{u})_{\Omega}, \tag*{(61)}

where we used the fact that 1ω=R1Ω\mathbf{1}_{\omega}=\mathbf{R}\mathbf{1}_{\Omega}.

Comparing coarse and fine-grid dissipation

Here we compare the rate of dissipation induced by the 1D diffusion operator discretized on the coarse grid, D‾∈RI×I\overline{\mathbf{D}}\in\mathbb{R}^{I\times I} (for grid-spacing HH), with the dissipation induced by the same operator but discretized on the fine grid, D∈RN×N\mathbf{D}\in\mathbb{R}^{N\times N} (for grid-spacing h=HJh=\frac{H}{J}). The difference in dissipated energy between these two quantities ΔD\Delta_D is given by:

ΔD=(u,Du)ω−(u‾,D‾u‾)Ω=uTωDu−(Wu)TΩD‾Wu=uTHJDu−uTHWTD‾Wu=JHuT(H2J2D−H2JWTD‾W)u=JHuT(h2D−1JWT(H2D‾)W)⏟=:DΔu,(62)\begin{aligned} \Delta_D=(\mathbf{u},\mathbf{D}\mathbf{u})_{\omega}-(\overline{\mathbf{u}},\overline{\mathbf{D}}\overline{\mathbf{u}})_{\Omega}=\mathbf{u}^{T}\omega\mathbf{D}\mathbf{u}-(\mathbf{W}\mathbf{u})^{T}\Omega\overline{\mathbf{D}}\mathbf{W}\mathbf{u}=\mathbf{u}^{T}\frac{H}{J}\mathbf{D}\mathbf{u}-\mathbf{u}^{T}H\mathbf{W}^{T}\overline{\mathbf{D}}\mathbf{W}\mathbf{u} \\ &=\frac{J}{H}\mathbf{u}^{T}\left(\frac{H^{2}}{J^{2}}\mathbf{D}-\frac{H^{2}}{J}\mathbf{W}^{T}\overline{\mathbf{D}}\mathbf{W}\right)\mathbf{u}=\frac{J}{H}\mathbf{u}^{T}\underbrace{\left(h^{2}\mathbf{D}-\frac{1}{J}\mathbf{W}^{T}(H^{2}\overline{\mathbf{D}})\mathbf{W}\right)}_{=:\mathbf{D}^{\Delta}}\mathbf{u}, \tag*{(62)} \end{aligned}

for periodic BCs. Here we have written DΔ\mathbf{D}^{\Delta} such that it is independent of the grid-spacing, but only depends on the ratio J=HhJ=\frac{H}{h}. Note that DΔ\mathbf{D}^{\Delta} is a symmetric matrix and as such its eigenvalues λNΔ≤…≤λ1Δ\lambda_{N}^{\Delta}\leq\ldots\leq\lambda_{1}^{\Delta} are real and its eigenvectors can be chosen to form an orthogonal basis. Furthermore, λ1Δ=0\lambda_{1}^{\Delta}=0 for periodic boundary conditions. We can bound ΔD\Delta_D by noting that

ΔD≤max⁡iJHλiΔ∥u∥22.(63)\Delta_D\leq\max_{i}\frac{J}{H}\lambda_{i}^{\Delta}\lVert\mathbf{u}\rVert_{2}^{2}. \tag*{(63)}

If the eigenvalues are all strictly nonpositive, it follows that the difference in dissipation ΔD\Delta_D is always less than or equal to zero. In other words, D\mathbf{D} extracts more (or equal) energy from the reference system as D‾\overline{\mathbf{D}} does from the filtered system. To prove that the eigenvalues of DΔ\mathbf{D}^{\Delta} are indeed nonpositive turns out to be a difficult problem which we circumvent with a numerical ‘proof’. In Figure B.17 we display the largest non-zero eigenvalue λ2Δ\lambda_{2}^{\Delta} for different values of II and JJ, indicating that λiΔ≤0\lambda_{i}^{\Delta}\leq0 for realistic values of II and JJ.

Figure B.17. Largest non-zero eigenvalue λ2Δ\lambda_2^\Delta of DΔ\mathbf{D}_\Delta for different values of II and JJ.

SGS compression

In this section we outline how we obtain the SGS compression parameter values t∈RJ\mathbf{t} \in\mathbb{R}^{J} such that s≈W(u′)2\mathbf{s} \approx\mathbf{W}(\mathbf{u}')^2. This can be achieved by using a singular value decomposition (SVD). The SVD minimizes the following loss function for t^∈RJ\hat{\mathbf{t}} \in\mathbb{R}^{J} (here we assume a uniform grid):

Ls(Xμ;Θ)=1pI∑μ∈Xμ∣J−1tTμ−μTttTμ∣=1pIJ∑μ∈Xμ∣μTμ−μTt^t^Tμ∣=1pIJ∑μ∈Xμ∣μTμ+μTt^t^Tt^t^Tμ−2μTt^t^Tμ∣=1pIJ∑μ∈Xμ(t^t^Tμ−μ)T(t^t^Tμ−μ)(64)\begin{aligned} \mathcal{L}_{s}(\mathbf{X}_{\mu};\boldsymbol{\Theta}) = \frac{1}{pI}\sum_{\mu\in\mathcal{X}_{\mu}}\left|J^{-1}\mathbf{t}^{T}\mu-\mu^{T}\mathbf{t}\mathbf{t}^{T}\mu\right| = \frac{1}{pIJ}\sum_{\mu\in\mathcal{X}_{\mu}}\left|\mu^{T}\mu-\mu^{T}\hat{\mathbf{t}}\hat{\mathbf{t}}^{T}\mu\right| \\ &= \frac{1}{pIJ}\sum_{\mu\in\mathcal{X}_{\mu}}\left|\mu^{T}\mu+\mu^{T}\hat{\mathbf{t}}\hat{\mathbf{t}}^{T}\hat{\mathbf{t}}\hat{\mathbf{t}}^{T}\mu-2\mu^{T}\hat{\mathbf{t}}\hat{\mathbf{t}}^{T}\mu\right| = \frac{1}{pIJ}\sum_{\mu\in\mathcal{X}_{\mu}}\left(\hat{\mathbf{t}}\hat{\mathbf{t}}^{T}\mu-\mu\right)^{T}\left(\hat{\mathbf{t}}\hat{\mathbf{t}}^{T}\mu-\mu\right) \tag*{(64)} \end{aligned}

subject to t^Tt^=1\hat{\mathbf{t}}^{T}\hat{\mathbf{t}}=1,

where μ\mu refers to a column vector of snapshot matrix Xμ∈RJ×Ip\mathbf{X}_{\mu}\in\mathbb{R}^{J\times Ip} and t=J−1/2t^\mathbf{t}=J^{-1/2}\hat{\mathbf{t}}. The last expression for Ls\mathcal{L}_{s} is a projection error. Such an error is typically minimized in a reduced order modeling setting to obtain a proper basis, see [3]. For this one typically uses an SVD. Conveniently, minimizing this error is equivalent to minimizing our required energy error (first expression). In this expression the prefactor J−1J^{-1} accounts for W\mathbf{W} in the SGS energy (38). The snapshot matrix Xμ\mathbf{X}_{\mu} is constructed from the training data set Xu∈RN×p\mathbf{X}_{u}\in\mathbb{R}^{N\times p} which contains pp DNS snapshots as the columns. From this matrix we compute the SGS content: Xu′=(I−RW)Xu\mathbf{X}_{u}'=(\mathbf{I}-\mathbf{R}\mathbf{W})\mathbf{X}_{u}. Finally, this matrix is reshaped into Xμ\mathbf{X}_{\mu} such that each column corresponds to the SGS content in a coarse cell, i.e. u′→[μi,…,μI]\mathbf{u}'\to[\mu_i,\ldots,\mu_I] for each column in Xu′\mathbf{X}_{u}'. The SVD of Xμ\mathbf{X}_{\mu} is given by

Xμ=UμΣμVμT,(65)\mathbf{X}_{\mu}=\mathbf{U}_{\mu}\mathbf{\Sigma}_{\mu}\mathbf{V}_{\mu}^{T}, \tag*{(65)}

where Uμ∈RJ×J\mathbf{U}_{\mu}\in\mathbb{R}^{J\times J} and Vμ∈RIp×Ip\mathbf{V}_{\mu}\in\mathbb{R}^{Ip\times Ip} are unitary matrices containing the left and right-singular vectors, respectively, and Σμ∈RJ×Ip\mathbf{\Sigma}_{\mu}\in\mathbb{R}^{J\times Ip} contains the singular values on the diagonal. The values for t^\hat{\mathbf{t}} that minimize (C.1) correspond to the first column of Uμ\mathbf{U}_{\mu}.

Non-periodic boundary conditions

To extend our method to different types of BCs we resort to what the machine learning community refers to as padding [35] and the scientific computing community refers to as the ghost-cell method [57]. We will treat both inflow and outflow BCs, on uniform 1D grids, as this is relevant for Burgers’ equation.

Implementation for the fine grid

The ghost-cell method enhances the discretization with ghost cells beyond the domain boundary ∂Ω\partial\Omega (with domain Ω=[a,b]\Omega= [a,b]), as displayed in Figure D.18. Here we present the implementation for the fine grid.

The inflow (Dirichlet) BC is given by u(x=a,t)=α(t)u(x = a,t) = \alpha(t). Based on this, we compute ghost value u0u_0 as

u(x=a,t)=α(t)=u1+u02→u0=2α(t)−u1.(66)u(x = a,t) = \alpha(t) = \frac{u_1 + u_0}{2} \quad\rightarrow\quad u_0 = 2\alpha(t) - u_1. \tag*{(66)}

This corresponds to the first ghost cell outside the left boundary, see Figure D.18. For the outflow BC we use a symmetric BC at the right boundary, given by ∂u∂x∣x=b,t=0\left.\frac{\partial u}{\partial x}\right|_{x=b,t} = 0. This is implemented by taking uN+1=uNu_{N+1} = u_N, where uN+1u_{N+1} corresponds to the first ghost cell outside the right boundary, see Figure D.18.

Figure D.18. 1D grid enhanced with ghost cells beyond the domain boundaries, indicated by the hollow circles.

Implementation for the filtered system

Our structure-preserving closure modeling framework is effectively a nonlinear stencil, due to the presence of a convolutional neural network. It therefore takes information from kk neighboring grid cells on each side. This means we require kk ghost cells on either side of the domain boundary ∂Ω\partial\Omega. To find appropriate choices for the ghost values uˉi\bar{\mathbf{u}}_i and si\mathbf{s}_i (i=−k+1,…,0,I+1,…,I+ki = -k + 1,\ldots,0,I + 1,\ldots,I + k) we consider the fine-grid solution u\mathbf{u} and appropriately extend this past the domain boundary, see Figure D.19.

Inflow BC

For the left inflow BC we extend (D.1) to

u−i+1=2α(t)−ui,i=1,2,…(67)u_{-i+1} = 2\alpha(t) - u_i,\qquad i = 1,2,\ldots \tag*{(67)}

We can rewrite this as a function of uˉi\bar{u}_i and SGS content μi\mu_i:

(uˉ−i+1+μ−i+1,J−j)=2α(t)−(uˉi+μi,1+j),1≤i≤k,0≤j≤J−1.(68)(\bar{u}_{-i+1} + \mu_{-i+1,J-j}) = 2\alpha(t) - (\bar{u}_i + \mu_{i,1+j}),\qquad1 \leq i \leq k,\quad0 \leq j \leq J - 1. \tag*{(68)}

This can be split into a filtered part:

uˉ−i+1=2α(t)−uˉi,1≤i≤k,(69)\bar{u}_{-i+1} = 2\alpha(t) - \bar{u}_i,\qquad1 \leq i \leq k, \tag*{(69)}

which yields the ghost values for uˉ\bar{u} past the left boundary, and a SGS part:

μ−i+1,J−j=−μi,1+j,1≤i≤k,0≤j≤J−1.(70)\mu_{-i+1,J-j} = -\mu_{i,1+j},\qquad1 \leq i \leq k,\quad0 \leq j \leq J - 1. \tag*{(70)}

The latter can be simplified as

μ−i+1=−Pμi,1≤i≤k,(71)\mu_{-i+1} = -P\mu_i,\qquad1 \leq i \leq k, \tag*{(71)}

where P∈RJ×J\mathbf{P} \in\mathbb{R}^{J \times J} is the permutation matrix that represents the reflection across the boundary.

Outflow BC

For the symmetric outflow BC we extend the fine-grid solution past the domain as

uN+i=uN−i+1,i=1,2,…(72)u_{N+i} = u_{N-i+1},\qquad i = 1,2,\ldots \tag*{(72)}

In terms of uˉi\bar{u}_i and μi\mu_i this becomes

uˉI+i+μI+i,1+j=uˉI−i+1+μI−i+1,J−j,1≤i≤k,0≤j≤J−1,(73)\bar{u}_{I+i} + \mu_{I+i,1+j} = \bar{u}_{I-i+1} + \mu_{I-i+1,J-j},\qquad1 \leq i \leq k,\quad0 \leq j \leq J - 1, \tag*{(73)}

which can again be split into an equation for the ghost values for uˉ\bar{u}:

uˉI+i=uˉI−i+1,1≤i≤k,(74)\bar{u}_{I+i} = \bar{u}_{I-i+1},\qquad1 \leq i \leq k, \tag*{(74)}

and a SGS part

μI+i=PμI−i+1,1≤i≤k.(75)\mu_{I+i} = P\mu_{I-i+1},\qquad1 \leq i \leq k. \tag*{(75)}

Figure D.19. An example solution uu with N=1000N = 1000 filtered onto a coarse grid with I=5I = 5 extended past ∂Ω\partial\Omega, according to (D.2) (α=710\alpha= \frac{7}{10}) for the left boundary and (D.7) for the right boundary. ∂Ω\partial\Omega is indicated by the dashed vertical lines. uˉ\bar{u} is extended past ∂Ω\partial\Omega according to (D.4) and (D.9) and u′u' is extended according to (D.6) and (D.10).

BCs for the SGS variables

After extending the solution using the ghost-cell method we aim to find the value for ss beyond the boundary. In order to find these ghost values we make use of the fact that we can interpret the operation tTt\mathbf{t}^{T}\mathbf{t} as the back and forth projection from a reduced basis to the physical SGS basis and back. This means that

μi≈Jtsi\mu_i \approx\sqrt{\mathbf{J}\mathbf{t}}s_i

and

si=JtTtsi.s_i = \mathbf{J}\mathbf{t}^{T}\mathbf{t}s_i.

The idea is to project sis_i back onto physical SGS space, apply the P\mathbf{P} operator, and then project back to find the appropriate ghost values. Using this idea we obtain the following relations for the left and right boundary, respectively:

μ−i+1=−Pμi→s−i+1=−JtTPtsi,1≤i≤k,(76)\mu_{-i+1} = -\mathbf{P}\mu_i \quad\rightarrow\quad s_{-i+1} = -\mathbf{J}\mathbf{t}^{T}\mathbf{P}\mathbf{t}s_i,\qquad1 \leq i \leq k, \tag*{(76)}
μI+i=PμI−i+1→sI+i=JtTPtsI−i+1,1≤i≤k.(77)\mu_{I+i} = \mathbf{P}\mu_{I-i+1} \quad\rightarrow\quad s_{I+i} = \mathbf{J}\mathbf{t}^{T}\mathbf{P}\mathbf{t}s_{I-i+1},\qquad1 \leq i \leq k. \tag*{(77)}

Note that tTPt\mathbf{t}^{T}\mathbf{P}\mathbf{t} is a scalar which only needs to be computed once.

A simulation for this inflow/outflow (I/O) BC implementation is shown in Figure D.20. Here we simulate Burgers’ equation with I/O BCs, and some additional forcing, see Appendix § 14. We depict the solution produced by our structure-preserving closure modeling framework and compare it to the DNS.

Training procedure

In order to train our machine learning-based closure models we first require DNSs to generate reference data. This reference data serves as a target for our machine learning models to reproduce. In this section we describe how we randomly generate simulation conditions (initial conditions, BCs, and forcing) for closure model training and testing. In addition, we describe the training procedure and the chosen hyperparameter values, obtained from the hyperparameter tuning procedure.

A simulation of Burgers’ equation with I/O BCs and forcing using our trained structure-preserving closure model for DOF = 40 (left), along with the DNS solution for N = 1000 (right)

Figure D.20. A simulation of Burgers’ equation with I/O BCs and forcing using our trained structure-preserving closure model for DOF = 40 (left), along with the DNS solution for N=1000N = 1000 (right).

Generating training data

To generate initial conditions, forcing, and unsteady Dirichlet BCs we make use of the following parameterized Fourier decomposition (with parameters α1,α2,α3∈R\alpha_1, \alpha_2, \alpha_3 \in\mathbb{R}):

ξ(y;α1,α2,α3)=α1+α2M∑i=1MCi1sin⁡(i2πα3y)+Ci2cos⁡(i2πα3y),(78)\xi(y; \alpha_1, \alpha_2, \alpha_3) = \alpha_1 + \frac{\alpha_2}{\sqrt{M}} \sum_{i=1}^{M} C_{i1} \sin\left(i\frac{2\pi}{\alpha_3}y\right) + C_{i2} \cos\left(i\frac{2\pi}{\alpha_3}y\right), \tag*{(78)}

where MM is uniformly sampled from {2,3,…,8}\{2, 3, \ldots, 8\} and Cij∼pC_{ij} \sim p from

p(y)={1,for 12≤∣y∣≤1,0,elsewhere.(79)p(y) = \begin{cases} 1, & \text{for } \frac{1}{2} \leq|y| \leq1,\\ 0, & \text{elsewhere.} \end{cases} \tag*{(79)}

In the case of Burgers’ equation we carry out 100 reference simulations on a uniform grid with N=1000N = 1000 on the domain Ω=[0,2π]\Omega= [0, 2\pi] for ν=0.01\nu= 0.01. To march the solution forward in time we employ an RK4 scheme with a time step size of Δt=2.5×10−3\Delta t = 2.5 \times10^{-3} and simulate up to T=10T = 10 [39]. 50 simulations are carried out using periodic BCs and 50 with I/O BCs. For the periodic case the initial condition is given by u(x,t=0)=ξ(x;2,1,∣Ω∣)u(x, t = 0) = \xi(x; 2, 1, |\Omega|). For the I/O case the inflow condition is given by u(0,t)=ξ(t,2,1,2π)u(0, t) = \xi(t, 2, 1, 2\pi) and the outflow condition by a symmetric BC on the right side of the domain. The implementation of the BCs is described in Appendix § 13. The initial condition is given by a constant valued function, equal to the inflow condition at t=0t = 0. In addition, we also add a steady forcing term F(x)=ξ(x;0,12,∣Ω∣)F(x) = \xi(x; 0, \frac{1}{2}, |\Omega|) to the RHS of (5) for the I/O case.

With regards to the KdV equation we employ a uniform grid with N=600N = 600 on the domain Ω=[0,32]\Omega= [0, 32] for ϵ=6\epsilon= 6 and μ=1\mu= 1. The solution is marched forward in time using an RK4 scheme with a time step size of Δt=10−4\Delta t = 10^{-4}, up to T=10T = 10. In this case we only consider periodic BCs, with the initial condition given by u(x,0)=ξ(x;0,35,∣Ω∣)u(x, 0) = \xi(x; 0, \frac{3}{5}, |\Omega|), and perform 100 reference simulations.

For both Burgers’ and KdV reference data is saved at each time interval of 5×10−35 \times10^{-3}. We randomly sample 10% of the data from these datasets to generate the two datasets used for training (one for each equation). Both of these are split into a training (70%) and validation set (30%). For testing purposes the unseen simulation conditions are generated in a similar manner, but with different randomly sampled MM and CijC_{ij}.

Hyperparameters and tuning

The chosen hyperparameters for our SP closure and the vanilla CNN are displayed in Table E.1. The weights and biases are initiated using the Glorot normal initialization algorithm [58]. They are optimized using the Adam optimization algorithm with parameters α\alpha (learning-rate), β1\beta_1 (decay rate for the first momentum estimates), β2\beta_2 (decay rate for the second momentum estimates), ϵ\epsilon (small constant to combat numerical instability) [27]. Hyperparameters are selected based on how well the trained closure models reproduce the RHS for the solution snapshots present in the validation set, corresponding to DOF=60\mathrm{DOF}=60. For this purpose, models are trained without trajectory fitting. For the hyperparameter optimization we opt to vary the number of hidden layers, for which we consider {0,1,2}\{0,1,2\}, and the number of channels per hidden layer, for which we consider {10,20,30}\{10,20,30\}. The performance of each of the trained closure models is shown in figure E.21. The best performing combination of hyperparameters (displayed in Table E.1), for

Figure E.21. NRMSE for reproducing the RHS for each of the considered hyperparameter configurations for Burgers’ (left) and KdV (right) averaged over the validation set for DOF = 60.

each equation, is selected to train the final closure models. This time trajectory fitting is included.

hyperparameterCNNSP
α\alpha10−310^{-3}10−310^{-3}
β1\beta_10.90.9
β2\beta_20.9990.999
ϵ\epsilon10−810^{-8}10−810^{-8}
mini-batch size2020
# iterations derivative fitting100100
# iterations trajectory fitting2020
trajectory fitting Δt\Delta t (Burgers’)0.010.01
trajectory fitting Δt\Delta t (KdV)5×10−35 \times10^{-3}5×10−35 \times10^{-3}
trajectory fitting # time steps (Burgers’)55
trajectory fitting # time steps (KdV)2020
nonlinear activation function (underlying) CNNReLUReLU
final activation function (underlying) CNNlinearlinear
kernel size75
stride11
# hidden layers22
# channels per hidden layer Burgers’2020
# channels per hidden layer KdV2030
total # parameters Burgers’32612780
total # parameters KdV32615352
BB (Burgers’)-1
BB (KdV)-2

Table E.1. Hyperparameters for the trained closure models

References

  1. [1]R. C. Smith, Uncertainty quantification: theory, implementation, and applications, Vol. 12, Siam, 2013.DOI
  2. [2]D. Sasaki, S. Obayashi, K. Nakahashi, Navier-stokes optimization of supersonic wings with four objectives using evolutionary algorithm, Journal of Aircraft 39 (4) (2002) 621–629.
  3. [3]B. Sanderse, Non-linearly stable decoupled-order models for incompressible flow with energy-conserving finite volume methods, Journal of Computational Physics 421 (2020) 109736. doi:10.1016/j.jcp.2020.109736.DOI
  4. [4]G. Alfonsi, Reynolds-averaged navier-stokes equations for turbulence modeling, Applied Mechanics Reviews - APPL MECH REV 62 (07 2009). doi:10.1115/1.3124648.DOI
  5. [5]P. Sagaut, C. Meneveau, Large Eddy Simulation for Incompressible Flows: An Introduction, Scientific Computation, Springer, 2006. URL https://books.google.nl/books?id=0DYiH6RNy0QC
  6. [6]J. J. O’Neill, X.-M. Cai, R. Kinnersley, A generalised stochastic backscatter model: large-eddy simulation of the neutral surface layer, Quarterly Journal of the Royal Meteorological Society 141 (692) (2015) 2617–2629. arXiv:https://rmets.onlinelibrary.wiley.com/doi/pdf/10.1002/qj.2548, doi:https://doi.org/10.1002/qj.2548. URL https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.2548
  7. [7]D. Carati, S. Ghosal, P. Moin, On the representation of backscatter in dynamic localization models, Physics of Fluids 7 (3) (1995) 606 – 616, cited by: 115. doi:10.1063/1.868585. URL https://www.scopus.com/inward/record.uri?eid=2-s2.0-0000176178&doi=10.1063%2f1.868585&partnerID=40&md5=2bf2184a75c303b29897eb25f5dc357bDOI
  8. [8]D. K. Lilly, A proposed modification of the germano subgrid-scale closure method, Physics of Fluids A: Fluid Dynamics 4 (3) (1992) 633–635. arXiv:https://doi.org/10.1063/1.858280, doi:10.1063/1.858280. URL https://doi.org/10.1063/1.858280
  9. [9]J. Smagorinsky, General circulation experiments with the primitive equations: I. the basic experiment, Monthly weather review 91 (3) (1963) 99–164.doi.org/10.1175/1520-0493(1963)091<0099:gcewtp>2.3.co;2
  10. [10]A. Prakash, K. E. Jansen, J. A. Evans, Optimal clipping of structural subgrid stress closures for large eddy simulation (2022). doi:10.48550/ARXIV.2201.09122. URL https://arxiv.org/abs/2201.09122
  11. [11]U. Piomelli, W. H. Cabot, P. Moin, S. Lee, Subgrid-scale backscatter in turbulent and transitional flows, Physics of Fluids A 3 (7) (1991) 1766 – 1771, cited by: 315. doi:10.1063/1.857956. URL https://www.scopus.com/inward/record.uri?eid=2-s2.0-0001165360&doi=10.1063%2f1.857956&partnerID=40&md5=d73e57f3428cb7323a4ef7f4f6cfacd
  12. [12]U. Piomelli, T. A. Zang, C. G. Speziale, M. Y. Hussaini, On the large-eddy simulation of transitional wall-bounded flows, Physics of Fluids A 2 (2) (1990) 257 – 265, cited by: 96; All Open Access, Green Open Access. doi:10.1063/1.857774. URL https://www.scopus.com/inward/record.uri?eid=2-s2.0-0001126541&doi=10.1063%2f1.857774&partnerID=40&md5=b fefe4458776d8d80cb1eed212c5c924DOI
  13. [13]S. Ghosal, T. S. Lund, P. Moin, K. Akselvoll, A dynamic localization model for large-eddy simulation of turbulent flows, Journal of Fluid Mechanics 286 (1995) 229–255. doi:10.1017/S0022112095000711.DOI
  14. [14]H. Frezat, J. L. Sommer, R. Fablet, G. Balarac, R. L’huensat, A posteriori learning of quasi-geostrophic turbulence parametrization: an experiment on integration steps (2021). doi:10.48550/ARXIV.2111.06841. URL https://arxiv.org/abs/2111.06841
  15. [15]B. List, L.-W. Chen, N. Thuerey, Learned turbulence modelling with differentiable fluid solvers (2022). doi:10.48550/ARXIV.2202.06988. URL https://arxiv.org/abs/2202.06988
  16. [16]J. Park, H. Choi, Toward neural-network-based large eddy simulation: application to turbulent channel flow, Journal of Fluid Mechanics 914 (2021) A16. doi:10.1017/jfm.2020.931.DOI
  17. [17]Y. Guan, A. Chattopadhyay, A. Subel, P. Hassanzadeh, Stable a posteriori les of 2d turbulence using convolutional neural networks: Backscattering analysis and generalization to higher re via transfer learning, Journal of Computational Physics 458 (2022) 111090. doi:https://doi.org/10.1016/j.jcp.2022.111090. URL https://www.sciencedirect.com/science/article/pii/S0021999122001528
  18. [18]D. Kochkov, J. A. Smith, A. Alieva, Q. Wang, M. P. Brenner, S. Hoyer, Machine learning&#x2013;accelerated computational fluid dynamics, Proceedings of the National Academy of Sciences 118 (21) (2021) e2101784118. arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.2101784118, doi:10.1073/pnas.2101784118. URL https://www.pnas.org/doi/abs/10.1073/pnas.2101784118
  19. [19]A. Beck, D. Flad, C.-D. Munz, Deep neural networks for data-driven les closure models, Journal of Computational Physics 398 (2019) 108910. doi:https://doi.org/10.1016/j.jcp.2019.108910. URL https://www.sciencedirect.com/science/article/pii/S0021999119306151
  20. [20]M. Kurz, A. Beck, A machine learning framework for les closure terms, ETNA - Electronic Transactions on Numerical Analysis 56 (2022) 117–137. doi:10.1553/etna_vol56s117.DOI
  21. [21]M. Kurz, A. Beck, Investigating model-data inconsistency in data-informed turbulence closure terms, in: 14th WCCM-ECCOMAS Congress 2020, 2021. doi:10.23967/wccm-eccomas.2020.115.
  22. [22]J. F. MacArt, J. Sirignano, J. B. Freund, Embedded training of neural-network subgrid-scale turbulence models, Phys. Rev. Fluids 6 (2021) 050502. doi:10.1103/PhysRevFluids.6.050502. URL https://link.aps.org/doi/10.1103/PhysRevFluids.6.050502DOI
  23. [23]S. D. Agdestin, B. Sanderse, Learning filtered discretization operators: non-intrusive versus intrusive approaches (2022). doi:10.48550/ARXIV.2208.09363. URL https://arxiv.org/abs/2208.09363
  24. [24]H. Melchers, D. Crommelin, B. Koren, V. Menkovski, B. Sanderse, Comparison of neural closure models for discretised pdes, ArXiv preprint arXiv:2210.14675 (2022).DOI
  25. [25]M. Kurz, P. Offenhäuser, A. Beck, Deep reinforcement learning for turbulence modeling in large eddy simulations (2022). doi:10.48550/ARXIV.2206.11038. URL https://arxiv.org/abs/2206.11038
  26. [26]H. J. Bae, P. Koumoutsakos, Scientific multi-agent reinforcement learning for wall-models of turbulent flows, Nature Communications 13 (1) (2022) 1443. doi:10.1038/s41467-022-28957-7. URL https://doi.org/10.1038/s41467-022-28957-7
  27. [27]D. P. Kingma, J. Ba, Adam: A method for stochastic optimization (2014). doi:10.48550/ARXIV.1412.6980. URL https://arxiv.org/abs/1412.6980
  28. [28]Y. Li, Deep reinforcement learning: An overview, CoRR abs/1701.07274 (2017). arXiv:1701.07274. URL http://arxiv.org/abs/1701.07274
  29. [29]V. Shankar, V. Puri, R. Balakrishnan, R. Maulik, V. Viswanathan, Differentiable physics-enabled closure modeling for burgers’ turbulence, Machine Learning: Science and Technology 4 (1) (2023) 015017. doi:10.1088/2632-2153/acb19c. URL https://dx.doi.org/10.1088/2632-2153/acb19c
  30. [30]Y. Bar-Sinai, S. Hoyer, J. Hickey, M. P. Brenner, Learning data-driven discretizations for partial differential equations, Proceedings of the National Academy of Sciences 116 (31) (2019) 15344–15349. arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1814058116, doi:10.1073/pnas.1814058116. URL https://www.pnas.org/doi/abs/10.1073/pnas.1814058116
  31. [31]A. Subel, A. Chattopadhyay, Y. Guan, P. Hassanzadeh, Data-driven subgrid-scale modeling of forced Burgers turbulence using deep learning with generalization to higher Reynolds numbers via transfer learning, Physics of Fluids 33 (3) (2021) 031702. arXiv:https://pubs.aip.org/aip/pof/article-pdf/doi/10.1063/5.0040286/14127880/031702/1_online.pdf, doi:10.1063/5.0040286. URL https://doi.org/10.1063/5.0040286
  32. [32]V. Shankar, V. Puri, R. Balakrishnan, R. Maulik, V. Viswanathan, Differentiable physics-enabled closure modeling for burgers’ turbulence, Machine Learning: Science and Technology 4 (1) (2023) 015017. doi:10.1088/2632-2153/acb19c. URL https://dx.doi.org/10.1088/2632-2153/acb19c
  33. [33]S. Kang, E. M. Constantescu, Learning subgrid-scale models with neural ordinary differential equations (2023). arXiv:2212.09967.arxiv.org/abs/2212.09967
  34. [34]Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, A. Anandkumar, Fourier neural operator for parametric partial differential equations, CoRR abs/2010.08895 (2020). arXiv:2010.08895. URL https://arxiv.org/abs/2010.08895
  35. [35]O. O’Shea, R. Nash, An introduction to convolutional neural networks (2015). doi:10.48550/ARXIV.1511.08458. URL https://arxiv.org/abs/1511.08458
  36. [36]M. D. Love, Subgrid modelling studies with burgers’ equation, Journal of Fluid Mechanics 100 (1) (1980) 87–110. doi:10.1017/S0022112080001024.DOI
  37. [37]A. Jameson, The construction of discretely conservative finite volume schemes that also globally conserve energy or entropy, J. Sci. Comput. 34 (2008) 152–187. doi:10.1007/s10915-007-9171-7.
  38. [38]B. Sanderse, Energy-conserving discretization methods for the incompressible navier-stokes equations : application to the simulation of wind-turbine wakes, Phd thesis 2 (research not tu/e / graduation tu/e), Centrum voor Wiskunde en Informatica (2013). doi:10.6100/IR750543.DOI
  39. [39]J. Butcher, Runge–Kutta methods, Scholarpedia 2 (9) (2007) 3147, revision #91735. doi:10.4249/scholarpedia.3147.DOI
  40. [40]J.-L. Yan, L.-H. Zheng, A class of momentum-preserving finite difference schemes for the korteweg-de vries equation, Computational Mathematics and Mathematical Physics 59 (10) (2019) 1582–1596. doi:10.1134/S0965542519100154. URL https://doi.org/10.1134/S0965542519100154
  41. [41]F. Trias, O. Lehmkuhl, A. C. Pérez-Segarra, R. Verstappen, Symmetry-preserving discretization of navier–stokes equations on collocated unstructured grids, Journal of Computational Physics 258 (2014) 246–267. doi:https://doi.org/10.1016/j.jcp.2013.10.031. URL https://www.sciencedirect.com/science/article/pii/S0021999113007079
  42. [42]D. Bank, N. Koenigstein, R. Giryes, Autoencoders, Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook (2023) 353–374.
  43. [43]C. A. De Moura, C. S. Kubrusly, The courant–friedrichs–lewy (cfl) condition, AMC 10 (12) (2013).
  44. [44]J. Bezanson, A. Edelman, S. Karpinski, V. B. Shah, Julia: A fresh approach to numerical computing, SIAM review 59 (1) (2017) 65–98. URL https://doi.org/10.1137/141000671
  45. [45]M. Innes, E. Saba, K. Fischer, D. Gandhi, M. C. Rudilosso, N. M. Joy, T. Karmali, A. Pal, V. Shah, Fashionable modelling with flux, CoRR abs/1811.01457 (2018). arXiv:1811.01457. URL https://arxiv.org/abs/1811.01457
  46. [46]M. Innes, Flux: Elegant machine learning with julia, Journal of Open Source Software (2018). doi:10.21105/joss.00602.DOI
  47. [47]W. Edeling, D. Crommelin, Reducing data-driven dynamical subgrid scale models by physical constraints, Computers & Fluids 201 (2020) 104470. doi:https://doi.org/10.1016/j.compfluid.2020.104470. URL https://www.sciencedirect.com/science/article/pii/S0045793020300438
  48. [48]S. Moradi, B. Teaca, J. Anderson, Role of phase synchronisation in turbulence, AIP Advances 7 (11) (2017) 115213. arXiv:https://pubs.aip.org/aip/adv/article-pdf/doi/10.1063/1.5003871/13011179/115213\_1\_online.pdf, doi:10.1063/1.5003871. URL https://doi.org/10.1063/1.5003871
  49. [49]V. Shankar, V. Puri, R. Balakrishnan, R. Maulik, V. Viswanathan, Differentiable physics-enabled closure modeling for burgers’ turbulence (2022). arXiv:2209.11614.arxiv.org/abs/2209.11614
  50. [50]Weglarczyk, Stanislaw, Kernel density estimation and its application, ITM Web Conf. 23 (2018) 00037. doi:10.1051/itmconf/20182300037. URL https://doi.org/10.1051/itmconf/20182300037
  51. [51]V. Girault, P.-A. Raviart, Finite element methods for Navier-Stokes equations: theory and algorithms, Vol. 5, Springer Science & Business Media, 2012.
  52. [52]F. D. A. Belbute-Peres, T. Economon, Z. Kolter, Combining differentiable pde solvers and graph neural networks for fluid flow prediction, in: international conference on machine learning, PMLR, 2020, pp. 2402–2411.
  53. [53]S. L. Brunton, J. L. Proctor, J. N. Kutz, Sparse identification of nonlinear dynamics with control (sindy)**slb acknowl- edges support from the u.s. air force center of excellence on nature inspired flight technologies and ideas (fa9550-14-1- 0398). jlp thanks bill and melinda gates for their active support of the institute of disease modeling and their sponsorship through the global good fund. jnk acknowledges support from the u.s. air force office of scientific research (fa9550-09- 0174)., IFAC-PapersOnLine 49 (18) (2016) 710–715, 10th IFAC Symposium on Nonlinear Control Systems NOLCOS 2016. doi:https://doi.org/10.1016/j.ifacol.2016.10.249. URL https://www.sciencedirect.com/science/article/pii/S2405896316318298
  54. [54]F. Menter, Y. Egorov, The scale-adaptive simulation method for unsteady turbulent flow predictions. part 1: theory and model description, Flow, turbulence and combustion 85 (1) (2010) 113–138.DOI
  55. [55]J. Buist, B. Sanderse, S. Dubinkina, R. Henkes, C. Oosterlee, Energy-conserving formulation of the two-fluid model for incompressible two-phase flow in channels and pipes, arXiv preprint arXiv:2104.07728 (2021).DOI
  56. [56]G. J. Chandler, R. R. Kerswell, Invariant recurrent solutions embedded in a turbulent two-dimensional kolmogorov flow, Journal of Fluid Mechanics 722 (2013) 554–595. doi:10.1017/jfm.2013.122.DOI
  57. [57]A. Dadone, B. Grossman, Ghost-cell method for analysis of inviscid three-dimensional flows on cartesian-grids, Computers & Fluids 36 (10) (2007) 1513–1528, special Issue Dedicated to Professor Michele Napolitano on the Occasion of his 60th Birthday. doi:https://doi.org/10.1016/j.compfluid.2007.03.013. URL https://www.sciencedirect.com/science/article/pii/S0045793007000394
  58. [58]X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Y. W. Teh, M. Titterington (Eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Vol. 9 of Proceedings of Machine Learning Research, PMLR, Chia Laguna Resort, Sardinia, Italy, 2010, pp. 249–256. URL https://proceedings.mlr.press/v9/glorot10a.html

Paper details

vlnc:W_kk9tenr8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents