betaFeedback

WorksMathematics

Symbolic Regression of Data-Driven Reduced Order Model Closures for Under-Resolved, Convection-Dominated Flows

Abstract

Data-driven closures correct the standard reduced order models (ROMs) to increase their accuracy in under-resolved, convection-dominated flows. There are two types of data-driven ROM closures in current use: (i) structural, with simple ansatzes (e.g., linear or quadratic); and (ii) machine learning-based, with neural network ansatzes. We propose a novel symbolic regression (SR) data-driven ROM closure strategy, which combines the advantages of current approaches and eliminates their drawbacks. As a result, the new data-driven SR closures yield ROMs that are interpretable, parsimonious, accurate, generalizable, and robust. To compare the data-driven SR-ROM closures with the structural and machine learning-based ROM closures, we consider the data-driven variational multiscale ROM framework and two under-resolved, convection-dominated test problems: the flow past a cylinder and the lid-driven cavity flow at Reynolds numbers Re = 10000, 15000, and 20000. This numerical investigation shows that the new data-driven SR-ROM closures yield more accurate and robust ROMs than the structural and machine learning ROM closures.

Introduction

High-performance computing and modern numerical algorithms have made high-fidelity fluid-thermal analysis tractable in geometries of ever increasing complexity. Despite continued advances in these areas, direct numerical (DNS), large eddy simulation (LES), and even unsteady Reynolds-averaged Navier–Stokes (URANS) simulations of turbulent thermal transport remain too costly for routine analysis and design of thermal–hydraulic systems, where hundreds of cases must be considered.

Reduced order models (ROMs) offer a promising alternative by leveraging expensive high-fidelity simulations (referred to as full order models or FOMs) to first extract a low-dimensional basis that captures the principal features of the underlying flow fields, and then construct computational models whose dimensions are orders of magnitude lower than the FOM dimension. In the numerical simulation of fluid flows, Galerkin ROMs (G-ROMs), which use data-driven basis functions in a Galerkin framework, have provided efficient and accurate approximations of laminar flows, such as the two-dimensional flow past a circular cylinder at low Reynolds numbers [1, 2]. However, turbulent flows are notoriously hard for the standard G-ROM. Indeed, to capture the complex dynamics, a large number [3] of ROM basis functions is required, which yields high-dimensional ROMs that cannot be used in realistic applications. Thus, computationally efficient, low-dimensional ROMs are used instead. Unfortunately, these ROMs are inaccurate since the ROM basis functions that were not used to build the G-ROM have an important role in dissipating the energy from the system [4]. Indeed, without enough dissipation, the low-dimensional G-ROM generally yields spurious numerical

Corresponding author: [email protected] (Alessandro Lucantonio) oscillations. Thus, closures and stabilization strategies are required for the low-dimensional G-ROMs to be stable and accurate [4, 5, 6, 7, 8, 9].

The ROM closures are ROM corrections in which the standard G-ROM is supplemented with a ROM closure model, τROM\tau^{\mathrm{ROM}}, which models the effect of the unresolved scales (i.e., the effect of the ROM basis functions that were not used to build the G-ROM). There are several types of ROM closure modeling strategies, which are carefully reviewed in [4].

Arguably the most popular class of ROM closures is data-driven ROM closures [4, 10], in which FOM data are used to train the ROM closure model. There are several data-driven approaches for modeling τROM\tau^{\mathrm{ROM}}, which are surveyed in [4]. To construct the data-driven ROM closure model, τROM\tau^{\mathrm{ROM}}, we first postulate a model form for τROM\tau^{\mathrm{ROM}}. Currently, there are two strategies for the ROM closure model form:

  1. A principled, structural strategy, in which the ROM closure model ansatz has a simple mathematical structure (e.g., a linear model or a quadratic model [11, 12]).

  1. A nonintrusive, machine learning (ML) strategy, in which the ROM closure model ansatz is based on neural networks [4, 10, 13, 14].

In the final step of the construction of the data-driven ROM closure model, we leverage the available data to determine the unknown parameters in the ROM closure model ansatz. This can be achieved, e.g., by solving a least squares problem (for the structural strategy) or by minimizing a loss function (for the machine learning strategy).

Each strategy has pros and cons:

  1. The structural strategy yields simple, interpretable ROM closure models, which can be easily combined with the ROM operators in the standard G-ROM. However, it is not clear that the simple building blocks (e.g., the linear and quadratic terms) can accurately represent the complex dynamics modeled by the ROM closure terms.

  1. The machine learning strategy, on the other hand, yields models that can, in principle, approximate the complex underlying dynamics. However, this strategy yields ROM closure models that contain parameters whose meaning is generally unclear.

Finding data-driven ROM closure models that are both accurate and physically interpretable is a major challenge in the development of ROMs for realistic turbulent flows. In this paper, we propose a novel data-driven ROM closure modeling strategy that centers around symbolic regression (SR). The novel SR-ROM closure modeling strategy combines the advantages of the current approaches and eliminates their drawbacks: The SR-ROM closure models are interpretable, parsimonious (i.e., contain much fewer parameters than, e.g., the machine learning strategy), and are rich enough to approximate the complex ROM closure dynamics. Given the limited number of parameters that characterize the new SR-ROM closures, they are less prone to overfitting and, thus, generalize better than the current data-driven ROM closures. Furthermore, the new SR-ROM closure models have the following additional advantages: They are accurate, computationally efficient, and, because they are constructed by using a sound mathematical strategy, they are amenable to analysis. We emphasize that, although SR has been successfully used in data-driven modeling, to our knowledge, the SR-ROM closure model proposed in this paper is the first instance in which SR is used in ROM closure modeling.

To illustrate our new SR-ROM closure modeling strategy, we need to choose a data-driven ROM closure modeling approach. To this end, we choose the data-driven variational multiscale ROM (d2-VMS-ROM) [11, 12], which is a successful data-driven ROM closure modeling approach utilized in various applications (from flow past a cylinder to flow past a backward-facing step to the quasi-geostrophic equations[11, 15]). The d2-VMS-ROM is a data-driven ROM closure that leverages the intrinsic hierarchical nature of the ROM basis and the VMS framework [16] to construct data-driven ROM closures. Specifically, the d2-VMS-ROM strategy proceeds by decomposing the ROM space Xr:=span⁡{φ1,…,φR}X^{r}:=\operatorname{span}\{\varphi_{1},\ldots,\varphi_{R}\} into the low-dimensional space of large, resolved scales, Xr:=span⁡{φ1,…,φr}X^{r}:=\operatorname{span}\{\varphi_{1},\ldots,\varphi_{r}\}, and the large-dimensional space of small, unresolved scales, X′:=span⁡{φr+1,…,φR}X^{\prime}:=\operatorname{span}\{\varphi_{r+1},\ldots,\varphi_{R}\}. To ensure computational efficiency, the d2-VMS-ROM is constructed in the low-dimensional space XrX^{r}. However, to ensure numerical accuracy, the d2-VMS-ROM is supplemented with a ROM closure model, τROM\tau^{\mathrm{ROM}}, which models the effect of the unresolved scales in X′X^{\prime}. We note that one distinguishing feature of the d2-VMS-ROM is that it is built by displaying the explicit formula for the ROM closure term τROM\tau^{\mathrm{ROM}}, which is achieved by leveraging the VMS formulation. Once we determine the precise form of the ROM closure term, we use one of the two data-driven modeling strategies outlined above (i.e., either a structural or an ML approach) to determine the unknown parameters of the closure model.

In this paper, we consider several d2-VMS-ROM formulations, which are based on linear regression, quadratic manifold formulation, and ML. Furthermore, we also consider a d2-VMS-ROM formulation based on the novel SR-ROM closure model. To assess the performance of the new SR-ROM closure, we perform an investigation of all four formulations in the numerical simulation of two convection-dominated test problems: flow past a cylinder and lid-driven cavity flow.

The rest of the paper is organized as follows: We begin by introducing the FOM in § 2. The ROM methodology is presented in § 3.1. § 3.2 provides the relevant background information and notation for the VMS approach. We also introduce the novel SR-ROM method along with various ML techniques used to model the closure term for the VMS-ROM, which we refer to as ML-VMS-ROMs. In § 4, the SR-ROM is compared with the current ML-VMS-ROMs in the numerical simulation of two test problems in the under-resolved, convection-dominated regime: the 2D flow past a cylinder and the 2D lid-driven cavity at Reynolds numbers Re=10000\mathrm{Re}=10000, 1500015000, and 2000020000. Concluding remarks are presented in § 5.

Full Order Model (FOM)

The governing equations are the incompressible Navier-Stokes equations (NSE):

∂u∂t+(u⋅∇)u=−∇p+1ReΔu,∇⋅u=0.(1)\frac{\partial\boldsymbol{u}}{\partial t}+(\boldsymbol{u}\cdot\nabla)\boldsymbol{u}=-\nabla p+\frac{1}{\mathrm{Re}}\Delta\boldsymbol{u},\qquad\nabla\cdot\boldsymbol{u}=0. \tag*{(1)}

Here, u\boldsymbol{u} represents the velocity subject to homogeneous Dirichlet boundary conditions, pp denotes the pressure, and Re\mathrm{Re} is the Reynolds number.

We employ the spectral element method for the spatial discretization of (1). The PqP_q–Pq−2P_{q-2} velocity-pressure coupling [17] is considered, where the velocity u\boldsymbol{u} is represented as a tensor-product Lagrange polynomial of degree qq in the reference element Ω^:=[−1,1]2\widehat{\Omega}:=[-1,1]^2, while the pressure pp is of degree q−2q-2. This yields a velocity space XNX^\mathcal{N} for approximating the velocity, and a pressure space YN‾Y^{\overline{\mathcal{N}}} for approximating the pressure, where the finite dimensions N\mathcal{N} and N‾\overline{\mathcal{N}} are the global numbers of spectral element degrees of freedom in the corresponding spectral element spaces.

The semi-discretization of (1) reads: Find (u,p)∈(XN,YN‾)(\boldsymbol{u},p)\in(X^\mathcal{N},Y^{\overline{\mathcal{N}}}) such that, for all (v,q)∈(XN,YN‾)(\boldsymbol{v},q)\in(X^\mathcal{N},Y^{\overline{\mathcal{N}}}),

(∂u∂t,v)+((u⋅∇)u,v)=(p,∇⋅v)−1Re(∇u,∇v),(2)\left(\frac{\partial\boldsymbol{u}}{\partial t},\boldsymbol{v}\right)+\left((\boldsymbol{u}\cdot\nabla)\boldsymbol{u},\boldsymbol{v}\right)=(p,\nabla\cdot\boldsymbol{v})-\frac{1}{\mathrm{Re}}(\nabla\boldsymbol{u},\nabla\boldsymbol{v}), \tag*{(2)}
−(∇⋅u,q)=0,(3)-(\nabla\cdot\boldsymbol{u},q)=0, \tag*{(3)}

where (⋅,⋅)(\cdot,\cdot) denotes the L2L^2 inner product. The resulting spatially-discretized ordinary differential equations are solved using a semi-implicit BDFkk/EXTkk scheme [18]. We refer to [19] for the detailed derivation of the FOM and the treatment with the nonhomogeneous Dirichlet boundary conditions.

Reduced Order Models (ROMs)

We present the Galerkin reduced order model (G-ROM) in § 3.1. In § 3.2, we first provide background information and establish notation for the VMS-ROM framework. Then, we leverage several ML strategies to construct VMS-ROM closure models. The resulting ROMs, which we collectively call ML-VMS-ROMs, are of four types: the linear regression VMS-ROM (§ 3.2.1); the data-driven VMS-ROM (§ 3.2.2); the novel symbolic regression VMS-ROM (§ 3.2.3); and the neural network VMS-ROM (§ 3.2.4).

Galerkin ROM (G-ROM)

To construct the G-ROM, we begin by collecting a set of velocity snapshots {uk:=u(x,tk)−φ0}k=1K\{\boldsymbol{u}^k:=\boldsymbol{u}(\boldsymbol{x},t^k)-\varphi_0\}_{k=1}^{K}, which correspond to the FOM solutions at time instances tkt^k, with the subtraction of the zeroth mode, φ0\varphi_0. We then employ the standard proper orthogonal decomposition (POD) procedure [20, 21] to construct the reduced basis functions. The Gramian matrix is formed using the L2L^2 inner product, and the first rr POD basis functions {φi}i=1r\{\varphi_i\}_{i=1}^{r} are constructed from the first rr eigenmodes of the Gramian.

The G-ROM is constructed by inserting the ROM basis expansion

ur(x)=φ0(x)+∑j=1rur,jφj(x)(4)\boldsymbol{u}_r(\boldsymbol{x})=\varphi_0(\boldsymbol{x})+\sum_{j=1}^{r}u_{r,j}\varphi_j(\boldsymbol{x}) \tag*{(4)}

into (1): Find uru_r such that, for all v∈X1v \in X_1,

(∂ur∂t,vi)+1Re⁡(∇ur,∇vi)+((ur⋅∇)ur,vi)=0,(5)\left(\frac{\partial u_r}{\partial t},v_i\right)+\frac{1}{\operatorname{Re}}(\nabla u_r,\nabla v_i)+\left((u_r\cdot\nabla)u_r,v_i\right)=0, \tag*{(5)}

where X1:=span⁡{ϕi}i=1rX_1:=\operatorname{span}\{\phi_i\}_{i=1}^r is the reduced space.

Remark 3.1. In this paper, the zeroth mode ϕ0\phi_0 is set to be the initial condition of the FOM simulation.

Remark 3.2. We note that, in the case of fixed geometries, the divergence and pressure terms drop out of (5) since the ROM basis is weakly divergence-free. For ROMs that include the pressure approximation, see, e.g., [1, 2, 22, 23, 24].

From (5), a system of differential equations in the coefficients of the POD bases, ur,ju_{r,j}, is derived:

Bdurdt=−C(u‾r)u‾r−1Re⁡Au‾r,(6)B\frac{d u_r}{d t}=-C(\overline{u}_r)\overline{u}_r-\frac{1}{\operatorname{Re}}A\overline{u}_r, \tag*{(6)}

where ur∈Rru_r\in\mathbb{R}^r is the vector of POD coefficients {ur,j}j=1r\{u_{r,j}\}_{j=1}^r, and u‾r∈Rr+1\overline{u}_r\in\mathbb{R}^{r+1} is the augmented vector that includes the zeroth mode’s coefficient. AA, BB, and CC represent the reduced stiffness, mass, and advection operators, respectively, with entries

Aij=∫Ω∇ϕi:∇ϕj dV,Bij=∫Ωϕi⋅ϕj dV,Cikj=∫Ωϕi⋅(ϕk⋅∇)ϕj dV.(7)A_{ij}=\int_{\Omega}\nabla\phi_i:\nabla\phi_j\,dV,\quad B_{ij}=\int_{\Omega}\phi_i\cdot\phi_j\,dV,\quad C_{ikj}=\int_{\Omega}\phi_i\cdot(\phi_k\cdot\nabla)\phi_j\,dV. \tag*{(7)}

For temporal discretization of (6), a semi-implicit scheme with kkth-order backward differencing (BDFkk) and kkth-order extrapolation (EXTkk) is considered. The fully discretized reduced system at time tlt^l is then derived:

(β0ΔtB+1Re⁡A)url+1=−∑i=1kαi[C(url−i)url−i+(C1+C2)u‾rl−i−c0]−B∑i=1kβiΔturl−i−1Re⁡a0,(8)\left(\frac{\beta_0}{\Delta t}B+\frac{1}{\operatorname{Re}}A\right)u_r^{l+1}=-\sum_{i=1}^{k}\alpha_i\left[C(u_r^{l-i})u_r^{l-i}+(C_1+C_2)\overline{u}_r^{l-i}-c_0\right]-B\sum_{i=1}^{k}\frac{\beta_i}{\Delta t}u_r^{l-i}-\frac{1}{\operatorname{Re}}a_0, \tag*{(8)}

where

c0,i=∫Ωϕi⋅(ϕ0⋅∇)ϕ0 dV,a0,i=∫Ω∇ϕi:∇ϕ0 dV,(9)c_{0,i}=\int_{\Omega}\phi_i\cdot(\phi_0\cdot\nabla)\phi_0\,dV,\quad a_{0,i}=\int_{\Omega}\nabla\phi_i:\nabla\phi_0\,dV, \tag*{(9)}
C1,ij=∫Ωϕi⋅(ϕ0⋅∇)ϕj dV,C2,ik=∫Ωϕi⋅(ϕk⋅∇)ϕ0 dV,(10)C_{1,ij}=\int_{\Omega}\phi_i\cdot(\phi_0\cdot\nabla)\phi_j\,dV,\quad C_{2,ik}=\int_{\Omega}\phi_i\cdot(\phi_k\cdot\nabla)\phi_0\,dV, \tag*{(10)}

for all i=1,…,ri=1,\ldots,r and j,k=0,…,rj,k=0,\ldots,r.

Machine Learning Variational Multiscale ROMs (ML-VMS-ROMs)

The variational multiscale (VMS) methods are numerical discretization techniques that greatly enhance the accuracy of classical Galerkin approximations in under-resolved simulations. This situation often occurs with coarse meshes or when there are insufficient basis functions available. The VMS framework, originally proposed by Hughes and his colleagues [25], has significantly influenced various fields in computational mechanics. For comprehensive reviews of its impact, see [26, 27].

In the following, we describe the VMS-ROM framework [11]. First, we consider two reduced spaces X1:=span⁡{ϕ1,…,ϕr}X_1:=\operatorname{span}\{\phi_1,\ldots,\phi_r\} and X2:=span⁡{ϕr+1,…,ϕR}X_2:=\operatorname{span}\{\phi_{r+1},\ldots,\phi_R\}, where ϕi\phi_i is the ii-th reduced basis function, and RR is the dimension of the snapshot dataset. We note that X1X_1 represents the span of the resolved ROM scales, and X2X_2 represents the span of the unresolved scales. In addition, the two reduced spaces are orthogonal because of the POD construction.

Next, we use the best ROM approximation of uu in the space X1⊕X2X_1\oplus X_2, that is, uR∈X1⊕X2u_R\in X_1\oplus X_2, defined as

uR=∑j=1Rur,jϕj(x)=∑j=1rur,jϕj+∑j=r+1Rur,jϕj=ur+ur′,(11)u_R=\sum_{j=1}^{R}u_{r,j}\phi_j(x)=\sum_{j=1}^{r}u_{r,j}\phi_j+\sum_{j=r+1}^{R}u_{r,j}\phi_j=u_r+u_r', \tag*{(11)}

where ur∈X1u_r\in X_1 represents the resolved ROM component of uu, and u′∈X2u'\in X_2 represents the unresolved ROM component of uu.

Plugging uRu_R in (1) and projecting the resulting equation onto X1X_1, we obtain

(∂uR∂t,vi)+1Re⁡(∇uR,∇vi)+((uR⋅∇)uR,vi)=0,∀i=1,…,r.(12)\left(\frac{\partial u_R}{\partial t},v_i\right)+\frac{1}{\operatorname{Re}}(\nabla u_R,\nabla v_i)+\left((u_R\cdot\nabla)u_R,v_i\right)=0,\qquad\forall i=1,\ldots,r. \tag*{(12)}

Using the orthogonality of the ROM basis functions, one can show that

(∂uR∂t,vi)=(∂ur∂t,vi),∀ i=1,…,r,(13)\left(\frac{\partial u_R}{\partial t},v_i\right)=\left(\frac{\partial u_r}{\partial t},v_i\right),\qquad\forall\ i=1,\ldots,r, \tag*{(13)}

and (12) can be further written as: ∀ i=1,…,r\forall\ i=1,\ldots,r

(∂ur∂t,vi)+1Re⁡(∇ur,∇vi)+((ur⋅∇)ur,vi)+[1Re⁡(∇uR,∇vi)+((uR⋅∇)uR,vi)−1Re⁡(∇ur,∇vi)−((ur⋅∇)ur,vi)]=0.(14)\left(\frac{\partial u_r}{\partial t},v_i\right)+\frac{1}{\operatorname{Re}}\left(\nabla u_r,\nabla v_i\right)+\left((u_r\cdot\nabla)u_r,v_i\right)+\left[\frac{1}{\operatorname{Re}}\left(\nabla u_R,\nabla v_i\right)+\left((u_R\cdot\nabla)u_R,v_i\right)-\frac{1}{\operatorname{Re}}\left(\nabla u_r,\nabla v_i\right)-\left((u_r\cdot\nabla)u_r,v_i\right)\right]=0. \tag*{(14)}

The bracketed term in (3.11) is the VMS-ROM closure term, which models the interaction between the ROM modes {φ1,…,φr}\{\varphi_1,\ldots,\varphi_r\} and the discarded ROM modes {φr+1,…,φR}\{\varphi_{r+1},\ldots,\varphi_R\}.

Equation (3.11) is referred to as VMS-ROM and can be written as:

Bdurdt=−C(u‾r)u‾r−1Re⁡A‾ur+τVMS,(15)B\frac{d u_r}{dt}=-C(\overline{u}_r)\overline{u}_r-\frac{1}{\operatorname{Re}}\overline{A}u_r+\tau_{\mathrm{VMS}}, \tag*{(15)}

where

τVMS,i≡((uR⋅∇)uR,φi)−((ur⋅∇)ur,φi),∀ i=1,…,r.(16)\tau_{\mathrm{VMS},i}\equiv\left((u_R\cdot\nabla)u_R,\varphi_i\right)-\left((u_r\cdot\nabla)u_r,\varphi_i\right),\qquad\forall\ i=1,\ldots,r. \tag*{(16)}

Remark 3.3. If we drop the VMS-ROM closure term, we are left with the G-ROM (3.2).

Remark 3.4. The VMS-ROM closure term is a correction term in the higher-dimensional space X1⊕X2X_1\oplus X_2.

The VMS-ROM closure term τVMS\tau_{\mathrm{VMS}} is essential for the accuracy of (3.12). However, the unresolved component of uRu_R, ur′u'_r, is not available during the online stage. To overcome this issue, for each i=1,…,ri=1,\ldots,r, τVMS,i\tau_{\mathrm{VMS},i} is approximated using a generic function gi(ur)g_i(u_r) as follows:

τVMS,i≡((uR⋅∇)uR,φi)−((ur⋅∇)ur,φi)≈gi(ur).(17)\tau_{\mathrm{VMS},i}\equiv\left((u_R\cdot\nabla)u_R,\varphi_i\right)-\left((u_r\cdot\nabla)u_r,\varphi_i\right)\approx g_i(u_r). \tag*{(17)}

In the VMS-ROM training, for each i=1,…,ri=1,\ldots,r, the function gig_i is determined by minimizing the mean squared error (MSE) between the VMS-ROM closure τi\tau_i and the reconstruction made by gig_i,

MSE⁡tr,i:=1Ntr∑j=1Ntr(τij−gi(u^rj))2.(18)\operatorname{MSE}_{\mathrm{tr},i}:=\frac{1}{N_{\mathrm{tr}}}\sum_{j=1}^{N_{\mathrm{tr}}}\left(\tau_i^j-g_i(\widehat{u}_r^j)\right)^2. \tag*{(18)}

We note that NtrN_{\mathrm{tr}} is the number of training time samples, which, in our numerical experiments, is selected to be the same as the number of snapshots that are used to build the POD basis functions. The VMS-ROM closure term τij\tau_i^j is defined to be

τij≡((PRuj⋅∇)PRuj,φi)−((Pruj⋅∇)Pruj,φi),(19)\tau_i^j\equiv\left((P_Ru^j\cdot\nabla)P_Ru^j,\varphi_i\right)-\left((P_ru^j\cdot\nabla)P_ru^j,\varphi_i\right), \tag*{(19)}

where PrP_r is the operator that projects the FOM solution u∈XNu\in X^N onto the rr-dimensional reduced space XrX_r, and uju^j is the FOM solution at time tjt^j. The vector of ROM coefficients u^r\widehat{u}_r is obtained by projecting the FOM solution onto the rr-dimensional reduced space, that is, u^r,i=(uj,φi)\widehat{u}_{r,i}=(u^j,\varphi_i) for all i=1,…,ri=1,\ldots,r. We note that for i=1,…,ri=1,\ldots,r, the function gig_i is found independently.

Remark 3.5. By a slight abuse of notation, we refer to both (16) and (19) as the VMS-ROM closure term. We note that (19) is used to train the function gig_i instead of (16) because the latter requires running the RR-dimensional G-ROM, which is not practical. Moreover, the stiffness terms are ignored in both (16) and (19) because these terms are usually small in the convection-dominated regime due to the 1/Re⁡1/\operatorname{Re} scaling (see the careful discussion of the commutation error in [28]).

In the remainder of this section, we describe the four ML techniques that we employ to construct the function gig_i in the ML-VMS-ROMs: the linear regression VMS-ROM (outlined in § 3.2.1); the data-driven VMS-ROM (outlined in § 3.2.2); the novel symbolic regression VMS-ROM (outlined in § 3.2.3); and the neural network VMS-ROM (outlined in § 3.2.4). A schematic of the four different ROM closure modeling strategies that are used to construct the ML-VMS-ROMs is depicted in Figure 3.1.

Figure 3.1. Schematic of the ML-VMS-ROMs considered in this paper.

VMS-ROM with Linear Regression Closure Models (LR-ROM)

Linear regression (LR) predicts a target (dependent) variable yy (in our case, the ii-th component of the VMS ROM closure, τi\tau_i) based on one or more input (independent) variables xjx_j (in our case, the jj-th ROM coefficient, ur,ju_{r,j}) by assuming a linear dependency between them. Mathematically, for each i=1,…,ri = 1, \ldots, r, the function gig_i is postulated as

gi(ur)=β0+∑j=1rβjur,j,(20)g_i(u_r) = \beta_0 + \sum_{j=1}^{r} \beta_j u_{r,j}, \tag*{(20)}

where rr is the dimension of the reduced space and (β0,…,βr)(\beta_0, \ldots, \beta_r) are the learnable weights. The optimal weights are found by solving the linear system given by the normal equations (see equation (2.5) in [29]). To penalize the complexity of the model and avoid overfitting, the regularization term α∑j=1rβj2\alpha\sum_{j=1}^{r} \beta_j^2 is added to (18) to favor the solution with weights of smaller L2L^2 norm.

In this work, the Ridge class of the scikit-learn library is used for training the LR-ROM. The default settings are used, apart for the hyperparameter α\alpha, which is tuned through a hold-out validation based on the error metric (28). A more detailed discussion on LR can be found in [30]. The advantages of LR stem from its interpretability and simplicity. However, linear models may not always effectively describe data, and they are particularly sensitive to outliers.

Data-Driven Variational Multiscale ROMs (d2-VMS-ROM)

In [11], the authors proposed a data-driven approach to approximate the VMS-ROM closure. In particular, the structure of function gig_i is assumed to be similar to the structure of the G-ROM function and is postulated as

gi(ur)=∑j=1rA~ijur,j+∑j=1r∑k=1rB~ijkur,jur,k.(21)g_i(u_r) = \sum_{j=1}^{r} \widetilde{A}_{ij}u_{r,j} + \sum_{j=1}^{r}\sum_{k=1}^{r} \widetilde{B}_{ijk}u_{r,j}u_{r,k}. \tag*{(21)}

The entries in the r×rr \times r matrix A~\widetilde{A} and the r×r×rr \times r \times r tensor B~\widetilde{B} are determined by minimizing the training MSE (18). We note that the minimization problem is equivalent to the following least squares problem

min⁡x∥Gx−f∥2,(22)\min_{\boldsymbol{x}} \lVert G\boldsymbol{x} - \boldsymbol{f} \rVert^2, \tag*{(22)}

where G∈RNtrr×(r2+r3)G \in\mathbb{R}^{N_{\mathrm{tr}}r \times(r^2+r^3)} is a matrix whose entries are determined by u^rj\widehat{u}_{r}^{j}, for j=1,…,Ntrj = 1, \ldots, N_{\mathrm{tr}}, f∈RNtrr×1\boldsymbol{f} \in\mathbb{R}^{N_{\mathrm{tr}}r \times1} is a vector whose entries are determined by τj\tau^j for j=1,…,Ntrj = 1, \ldots, N_{\mathrm{tr}}, and x∈R(r2+r3)×1\boldsymbol{x} \in\mathbb{R}^{(r^2+r^3)\times1} is a vector whose entries are determined by A~\widetilde{A} and B~\widetilde{B}. To alleviate the ill-conditioning of the least squares problem (22) [31], we use the truncated SVD as a regularization method [11].

VMS-ROM with Symbolic Regression Closure Models (SR-ROM)

Symbolic regression (SR) is a type of regression analysis that seeks to find mathematical expressions that accurately describe the relationship between the features and the labels of a dataset [32]. Unlike traditional regression methods, which rely on predefined model structures, SR identifies both the structure of the model and its parameters.

Here, we use SR to search for the function gig_i in (17) that solves the following optimization problem

gi=arg⁡max⁡f∈F[1−∑j=1Ntr(τji−f(ujr))2∑j=1Ntr(τji−τˉi)2],(23)g_i = \underset{f \in\mathcal{F}}{\arg\max}\left[1-\frac{\sum_{j=1}^{N_{\mathrm{tr}}}\left(\tau_j^i-f\left(\mathbf{u}_j^r\right)\right)^2}{\sum_{j=1}^{N_{\mathrm{tr}}}\left(\tau_j^i-\bar{\tau}_i\right)^2}\right], \tag*{(23)}

where τˉi\bar{\tau}_i is the mean of τi\tau_i over NtrN_{\mathrm{tr}} samples and F\mathcal{F} is a set of functions generated by a primitive set, that is, a collection of mathematical operations, such as {+,−,∗,sin⁡,exp⁡}\{+,-,\ast,\sin,\exp\}, and a terminal set, made of the independent variables and some constants. We note that maximizing the objective function in (23) (called fitness) is equivalent to minimizing the MSE in (18). Apart from dealing with algebraic expressions, SR has recently been extended to field problems by including discrete differential operators [33].

A common approach to SR (also adopted in this work) is genetic programming (GP) [34, 35]. GP is a gradient-free strategy that explores a space of mathematical expressions by iteratively evolving a population of candidate models through genetic operations. A candidate model, is also known as individual, can be represented by a tree, where each node is either a primitive or a terminal (i.e., a variable or a constant), and is characterized by its fitness, as defined above. GP-based SR consists of the following steps:

  1. An initial population of individuals is randomly generated, based on the chosen primitive set;

  2. For each individual, the constants are optimized via the Levenberg–Marquardt method to maximize the fitness function;

  3. A pool of individuals are selected according to fitness-based tournaments. Specifically, for each tournament a given number (tournament size) of individuals are sampled from the population and the ones with the highest fitness are selected for the next step;

  4. The selected expressions are modified using crossover (combining parts of two parent expressions to create new offspring) and mutation (randomly altering parts of an expression);

  5. Steps 2–4 are repeated for multiple generations, until a stopping condition is satisfied or a maximum number of generations is reached.

A more detailed review of SR methods can be found in [34, 35]. A schematic of the described algorithm is reported in Figure 3.2.

Figure 3.2. Schematic of the symbolic regression approach used to model ROM closures (SR-ROM).

In this paper, we used the SR implementation of the pyoperon [36] library, because it achieves a good compromise between test performance, final model size, and training time (see, e.g., Figure 1 in [37]). We fixed the following hyperparameters for every run:

  • tournament size equal to 5;

  • number of iterations of the constant optimizer equal to 10;

  • population size equal to 1000;

  • maximum depth of the trees equal to 5.

The maximum length of the individuals (i.e., the number of nodes), the number of generations, and the primitive set were tuned through a grid search based on a hold-out validation. A description of the datasets used for training, validation, and testing can be found in § 4.

VMS-ROM with Neural Network Closure Models (NN-ROM)

Among the different architectures of neural networks (NNs), in this work, we used a multi-layer perceptron (MLP) to represent the mapping between the vector of ROM coefficients ur\mathbf{u}_r and the approximated closure term. The MLP can be represented mathematically as follows:

gi(ur):=∑k=1∣Ln∣WknLkn(ur)+bn+1,(24)g_i(\mathbf{u}_r) := \sum_{k=1}^{\lvert L^n\rvert} W_k^n L_k^n(\mathbf{u}_r) + b^{n+1}, \tag*{(24)}
Lj+1(ur):=h(WjLj+bj+1),0≤j≤n−1,(25)L^{j+1}(\mathbf{u}_r) := h\left(W^j L^j + b^{j+1}\right), \qquad0 \leq j \leq n-1, \tag*{(25)}
L0(ur):=ur,(26)L^0(\mathbf{u}_r) := \mathbf{u}_r, \tag*{(26)}

where nn is the number of the (hidden) layers, LjL^j is the output of the jj-th layer, ∣Ln∣\lvert L^n\rvert is the number of nodes of the nn-th layer, hh is the activation function, WjW^j is the weight matrix for the connections between the jj-th and the (j+1)(j+1)-th layers, and bjb^j is the bias term for the outputs of the jj-th layer.

The training of the NN amounts to adjusting its weights to minimize a loss function using backpropagation [38]. Here, as a loss function, we chose the sum of the MSE (18) and the L2L^2 regularization term, which penalizes the sum of the squares of the weights. A more detailed introduction to NNs can be found in [39, 30].

In this work, we used the skorch [40] library for training the MLP with the following settings:

  • LeakyReLU as the activation function;

  • Adam optimizer;

  • 100 training epochs;

  • batch size equal to 512.

We tuned the values of the learning rate, the number of units per layer, the L2L^2 regularization parameter, and the dropout rate through a grid search based on a hold-out validation. Details on the datasets used for training, validation, and testing can be found in § 4.

Numerical Results

In this section, we assess the four ML-VMS-ROMs introduced in § 3.2 in the numerical simulation of two test problems: (i) the 2D flow past a cylinder, and (ii) the 2D lid-driven cavity. We emphasize that, for both test problems, we consider the under-resolved regime, i.e., the case when the number of ROM basis functions is not large enough to capture the underlying dynamics. We also note that, for the 2D lid-driven cavity, we consider the convection-dominated regime. Both the under-resolved regime and the convection-dominated regime are relevant to the realistic engineering and geophysical settings (e.g., the numerical simulation of turbulent flows).

We remark that each model depends on hyperparameters, and we list the considered hyperparameters for each model below. In each model, the function gig_i is determined by minimizing the training MSE, MSEtr,i\mathrm{MSE}_{\mathrm{tr},i} (18).

  • For the LR-ROM, we select the L2L^2 regularization parameter from the set {10i,2×10i,…,9×10i}i=0,…,5\{10^i, 2 \times10^i, \ldots, 9 \times10^i\}_{i=0,\ldots,5}.

  • For the d2-VMS-ROM, we select the SVD rank from the set {1,2,…,Rank⁡(G)}\{1,2,\ldots,\operatorname{Rank}(G)\}, where GG is the matrix in the least squares problem (22), whose entries are determined by the ROM coefficients of the projected snapshots, u^r\hat{u}_r.

  • For the SR-ROM, we select the maximum length from the set {5,10,…,45,50}\{5,10,\ldots,45,50\}, the number of generations from the set {10,25,50,75,100}\{10,25,50,75,100\}, and the primitive set from the set {{add⁡,sub⁡,mul⁡,constant⁡,variable⁡},{add⁡,sub⁡,mul⁡,sin⁡,constant⁡,variable⁡},{add⁡,sub⁡,mul⁡,exp⁡,sin⁡,constant⁡,variable⁡},{add⁡,sub⁡,mul⁡,sin⁡,cos⁡,constant⁡,variable⁡},{add⁡,sub⁡,mul⁡,exp⁡,sin⁡,cos⁡,constant⁡,variable⁡},{add⁡,sub⁡,mul⁡,exp⁡,sin⁡,cos⁡,square⁡,log⁡,constant⁡,variable⁡}}\{\{\operatorname{add},\operatorname{sub},\operatorname{mul},\operatorname{constant},\operatorname{variable}\},\{\operatorname{add},\operatorname{sub},\operatorname{mul},\sin,\operatorname{constant},\operatorname{variable}\},\{\operatorname{add},\operatorname{sub},\operatorname{mul},\exp,\sin,\operatorname{constant},\operatorname{variable}\},\{\operatorname{add},\operatorname{sub},\operatorname{mul},\sin,\cos,\operatorname{constant},\operatorname{variable}\},\{\operatorname{add},\operatorname{sub},\operatorname{mul},\exp,\sin,\cos,\operatorname{constant},\operatorname{variable}\},\{\operatorname{add},\operatorname{sub},\operatorname{mul},\exp,\sin,\cos,\operatorname{square},\log,\operatorname{constant},\operatorname{variable}\}\}};

  • For the NN-ROM, we select the learning rate from the set {10−4,10−3,10−2}\{10^{-4},10^{-3},10^{-2}\}, the number of nodes per layer from the set {[64,128,256,512,256,128,64],[64,128,256,128,64],[64,128,64]}\{[64,128,256,512,256,128,64],[64,128,256,128,64],[64,128,64]\}, the L2L^2 regularization parameter from the set {10−5,10−4,10−3}\{10^{-5},10^{-4},10^{-3}\}, and the dropout rate from the set {0.3,0.4,0.5}\{0.3,0.4,0.5\}.

For each model and each hyperparameter, the selected function gig_i is then integrated with the VMS-ROM, and its performance is further evaluated in the validation interval. The optimal hyperparameters are selected to be those that maximize the coefficient of determination

Rval2:=1−MSE⁡val1Nval∑i=1Nval(EFOMi−E‾FOM)2.(27)R_{\mathrm{val}}^2 := 1-\frac{\operatorname{MSE}_{\mathrm{val}}}{\frac{1}{N_{\mathrm{val}}}\sum_{i=1}^{N_{\mathrm{val}}}\left(E_{\mathrm{FOM}}^i-\overline{E}_{\mathrm{FOM}}\right)^2}. \tag*{(27)}

where the validation error, MSE⁡val\operatorname{MSE}_{\mathrm{val}}, is defined to be the MSE between the FOM kinetic energy and the ROM kinetic energy,

MSE⁡val=1Nval∑i=1Nval(EFOMi−EROMi)2.(28)\operatorname{MSE}_{\mathrm{val}}=\frac{1}{N_{\mathrm{val}}}\sum_{i=1}^{N_{\mathrm{val}}}\left(E_{\mathrm{FOM}}^i-E_{\mathrm{ROM}}^i\right)^2. \tag*{(28)}

The kinetic energy EE at time tjt^j is defined to be

E(tj)=Ej=12(∫Ωuj⋅uj dΩ)12,(29)E(t^j)=E^j=\frac{1}{2}\left(\int_{\Omega}\boldsymbol{u}^j\cdot\boldsymbol{u}^j\,\mathrm{d}\Omega\right)^{\frac{1}{2}}, \tag*{(29)}

and the averaged kinetic energy over the NvalN_{\mathrm{val}} samples in the validation set is defined to be

E‾FOM=1Nval∑i=1NvalEFOMi.(30)\overline{E}_{\mathrm{FOM}}=\frac{1}{N_{\mathrm{val}}}\sum_{i=1}^{N_{\mathrm{val}}}E_{\mathrm{FOM}}^i. \tag*{(30)}

Once the optimal parameters are selected, the resulting ML-VMS-ROM is assessed in a testing interval with the metric

rMSE⁡test:=1Ntest∑i=1Ntest(EFOMi−EROMi)2(EFOMi)2.(31)\operatorname{rMSE}_{\mathrm{test}}:=\frac{1}{N_{\mathrm{test}}}\sum_{i=1}^{N_{\mathrm{test}}}\frac{\left(E_{\mathrm{FOM}}^i-E_{\mathrm{ROM}}^i\right)^2}{\left(E_{\mathrm{FOM}}^i\right)^2}. \tag*{(31)}

2D Flow Past a Cylinder

Our first test problem is the 2D flow past a cylinder, which is a canonical test case for ROMs. The computational domain is Ω=[−2.5D:17D]×[−5D:5D]\Omega=[-2.5D:17D]\times[-5D:5D], where DD is the cylinder diameter, and the cylinder is centered at [0,0][0,0]. We consider two Reynolds numbers, namely, Re⁡=400\operatorname{Re}=400 and Re⁡=500\operatorname{Re}=500. For each Reynolds number, we collect Ntr=2001N_{\mathrm{tr}}=2001 snapshots, {uk:=u(x,tk)−φ0}k=1Ntr\{u^k:=u(x,t^k)-\varphi_0\}_{k=1}^{N_{\mathrm{tr}}}, over the time interval [500,520][500,520] (measured in convective time units, D/UD/U, where UU is the free-stream velocity). These snapshots are taken after the von Karman vortex street has developed, with sampling time Δts=0.01\Delta t_s=0.01. The zeroth mode, φ0\varphi_0, is set to be the FOM velocity field at t=500t=500. Snapshots of the velocity magnitude for Re⁡=400\operatorname{Re}=400 and Re⁡=500\operatorname{Re}=500 are displayed in Figure 4.1.

Velocity magnitude snapshots for 2D flow past a cylinder

Figure 4.1. 2D flow past a cylinder. Velocity magnitude snapshots for Re=400\mathrm{Re}=400 (left) and Re=500\mathrm{Re}=500 (right).

The snapshot data are then used to construct the reduced basis functions {φi}i=1r\{\varphi_i\}_{i=1}^r by using POD. The ROM coefficients, {u^j}j=1Ntr\{\hat{u}_j\}_{j=1}^{N_{\mathrm{tr}}}, are computed by projecting the snapshots onto the RR-dimensional reduced space, where RR is the rank of the snapshot matrix. The VMS-ROM closure terms {τj}j=1Ntr\{\tau_j\}_{j=1}^{N_{\mathrm{tr}}} are then computed using (19). For each ML-VMS-ROM, the function gig_i is found by minimizing the training MSE (18). We select the optimal values for the hyperparameters for model training such that they minimize the validation error (28). To evaluate this error, we further run the FOM simulation and monitor the FOM kinetic energy over the time interval [520,540][520,540]. The trained function gig_i is then used within the VMS-ROM to perform a simulation over the time interval [500,540][500,540]. Finally, we assess the generalization error in (31) of all the ROMs in an extrapolation region, that is, in the time interval [540,600][540,600].

In Table 1, the relative MSE (rMSEtest\mathrm{rMSE}_{\mathrm{test}}) between the FOM kinetic energy and the predicted kinetic energy during the time interval [540,600][540,600] (extrapolation region) for all the ML-VMS-ROMs is reported with respect to the reduced space dimension r=2,3,4,5r=2,3,4,5. Because the models trained using neural networks and symbolic regression rely on stochastic optimization algorithms, the rMSEtest\mathrm{rMSE}_{\mathrm{test}} for both the NN-ROM and the SR-ROM are computed by averaging the results over 5 runs corresponding to different seeds. The results show that the SR-ROM is the most robust model with respect

Re\mathrm{Re}rrLR-ROMd2-VMS-ROMSR-ROMNN-ROM
40021.4×10−41.4 \times10^{-4}5.3×10−65.3 \times10^{-6}(7.1±1.1)×10−7\mathbf{(7.1 \pm1.1) \times10^{-7}}(5.3±3.7)×10−6(5.3 \pm3.7) \times10^{-6}
40033.3×10−43.3 \times10^{-4}nan(1.5±2.0)×10−5\mathbf{(1.5 \pm2.0) \times10^{-5}}(3.1±3.1)×10−4(3.1 \pm3.1) \times10^{-4}
40045.5×10−55.5 \times10^{-5}7.5×10−7\mathbf{7.5 \times10^{-7}}(6.7±1.0)×10−7\mathbf{(6.7 \pm1.0) \times10^{-7}}(4.4±2.7)×10−6(4.4 \pm2.7) \times10^{-6}
40054.6×10−64.6 \times10^{-6}2.0×10−52.0 \times10^{-5}(9.5±8.9)×10−7\mathbf{(9.5 \pm8.9) \times10^{-7}}(10.0±6.7)×10−7\mathbf{(10.0 \pm6.7) \times10^{-7}}
50028.1×10−68.1 \times10^{-6}3.2×10−53.2 \times10^{-5}(1.7±1.9)×10−5(1.7 \pm1.9) \times10^{-5}(4.5±1.6)×10−6\mathbf{(4.5 \pm1.6) \times10^{-6}}
50036.6×10−56.6 \times10^{-5}3.7×10−53.7 \times10^{-5}(1.8±1.0)×10−6\mathbf{(1.8 \pm1.0) \times10^{-6}}(2.7±1.6)×10−6\mathbf{(2.7 \pm1.6) \times10^{-6}}
50042.2×10−52.2 \times10^{-5}3.2×10−6\mathbf{3.2 \times10^{-6}}(7.2±5.1)×10−6(7.2 \pm5.1) \times10^{-6}(10.4±7.8)×10−6(10.4 \pm7.8) \times10^{-6}
50053.1×10−43.1 \times10^{-4}1.9×10−41.9 \times10^{-4}(1.0±1.3)×10−6\mathbf{(1.0 \pm1.3) \times10^{-6}}(4.7±2.3)×10−5(4.7 \pm2.3) \times10^{-5}

Table 1. 2D flow past a cylinder. Relative MSE (rMSEtest\mathrm{rMSE}_{\mathrm{test}}) between the FOM kinetic energy and the kinetic energy predicted in the extrapolation region (i.e., in the time interval [540,600][540,600]) by the ML-VMS-ROMs, for different values of the reduced space dimension r=2,3,4,5r=2,3,4,5. For the SR-ROM and the NN-ROM, the mean and the standard deviation over 5 independent runs (different seeds) are reported. In bold, the best performing model for each rr value is listed.

to rr for both Re, followed by the NN-ROM, the d2-VMS-ROM, and the LR-ROM. In particular, we note that for Re =400=400 and r=2r=2 and r=3r=3 the SR-ROM rMSEtest\mathrm{rMSE}_{\mathrm{test}} values in the extrapolation region are an order of magnitude lower than the rMSEtest\mathrm{rMSE}_{\mathrm{test}} values of the other ROMs. The same happens for Re =500=500 and r=5r=5.

We further measure the closeness between the VMS-ROM closure terms {τj}j=1Ntr\{\tau_j\}_{j=1}^{N_{\mathrm{tr}}} and the reconstructed closure terms for each model. In particular, the MSE for each τ\tau component, MSEtr,i\mathrm{MSE}_{\mathrm{tr},i} ((18)) is computed, and the mean MSE, which is defined as

MSE‾tr:=1r∑i=1rMSEtr,i,(32)\overline{\mathrm{MSE}}_{\mathrm{tr}} := \frac{1}{r}\sum_{i=1}^{r}\mathrm{MSE}_{\mathrm{tr},i}, \tag*{(32)}

is reported in Table 2. We observe that SR-ROM is the most robust method also in this case. We also note that the NN-ROM has a much larger MSE than the SR-ROM even though the corresponding test errors on predicted energy are comparable (Table 1). For example, for Re =500=500, the training MSE of NN-ROM with r=3r=3 is roughly two orders of magnitude larger than the training MSE of SR-ROM. However, the performance of the two models in the extrapolation region is comparable. There are two reasons for this behavior:

Table 2: 2D flow past a cylinder. Mean squared error (MSEtr\mathrm{MSE}_{\mathrm{tr}}) between the VMS-ROM closure and the closures given by the ML-VMS-ROMs for Re=400,500\mathrm{Re}=400, 500. In bold, the best performing model for each case is listed.

RerrLR-ROMd2-VMS-ROMSR-ROMNN-ROM
40022.3×10−32.3 \times10^{-3}5.2×10−45.2 \times10^{-4}(6.9±12.5)×10−7(6.9 \pm12.5) \times10^{-7}(8.5±4.6)×10−4(8.5 \pm4.6) \times10^{-4}
35.9×10−35.9 \times10^{-3}5×10−45 \times10^{-4}(1.3±2.6)×10−4(1.3 \pm2.6) \times10^{-4}(1.7±1.1)×10−3(1.7 \pm1.1) \times10^{-3}
43.1×10−33.1 \times10^{-3}6.1×10−46.1 \times10^{-4}(1.2±1.5)×10−4(1.2 \pm1.5) \times10^{-4}(1.2±0.5)×10−3(1.2 \pm0.5) \times10^{-3}
51.4×10−31.4 \times10^{-3}5.6×10−65.6 \times10^{-6}(5.7±4.9)×10−5(5.7 \pm4.9) \times10^{-5}(1.9±0.4)×10−4(1.9 \pm0.4) \times10^{-4}
50024.7×10−44.7 \times10^{-4}1.2×10−31.2 \times10^{-3}(5.5±3.3)×10−5(5.5 \pm3.3) \times10^{-5}(8.7±4.3)×10−5(8.7 \pm4.3) \times10^{-5}
35.9×10−35.9 \times10^{-3}3.5×10−43.5 \times10^{-4}(7.5±12)×10−6(7.5 \pm12) \times10^{-6}(5.5±3.6)×10−4(5.5 \pm3.6) \times10^{-4}
43.5×10−33.5 \times10^{-3}4.1×10−34.1 \times10^{-3}(1.4±2.7)×10−4(1.4 \pm2.7) \times10^{-4}(4.9±2.7)×10−4(4.9 \pm2.7) \times10^{-4}
51.0×10−31.0 \times10^{-3}4×10−44 \times10^{-4}(5.2±3.0)×10−5(5.2 \pm3.0) \times10^{-5}(8±6.1)×10−4(8 \pm6.1) \times10^{-4}

Table 2.

  1. We are using different metrics for training (closure error) and validation/testing (kinetic energy error).

  2. In the NN-ROM and LR-ROM, we consider L2L^2 regularization to prevent models from overfitting. This results in a model with smaller weights and could lead to a larger training MSE but better generalizability (smaller rMSEtest\mathrm{rMSE}_{\mathrm{test}} in the extrapolation region).

Overall, our results demonstrate that SR-ROM outperforms the other ML-VMS-ROM models in terms of generalizability.

In Figure 4.2, we show the kinetic energy behavior of the ROMs and the FOM in the time interval [500,600][500, 600] with reduced space dimension r=2,3,4,5r=2, 3, 4, 5 for Re=400\mathrm{Re}=400 and Re=500\mathrm{Re}=500. For Re=400\mathrm{Re}=400, we observe that the LR-ROM energy deviates away from the FOM energy for all rr values considered. For the d2-VMS-ROM and the NN-ROM, although the energy is better than the LR-ROM energy, it could still deviate away from the FOM energy; see, for example, the r=3r=3 case. The SR-ROM is the most stable compared to the other three ML-VMS-ROMs, and it is able to predict energy that is comparable to the FOM energy. For Re=500\mathrm{Re}=500, we again observe that the LR-ROM energy deviates

Figure 4.2. 2D flow past a cylinder at Re=400\mathrm{Re}=400 and Re=500\mathrm{Re}=500. Comparison of the kinetic energy of the FOM and the ROMs for the reduced space dimension r=2,3,4,5r=2, 3, 4, 5.

away from the FOM energy for all rr values considered. For the d2-VMS-ROM and the NN-ROM, although the energy is better than the LR-ROM energy, it could still deviate away from the FOM energy; see, for example, the r=5r=5 case. The SR-ROM is the most stable with respect to different rr values.

In Table 3, we report the best symbolic models in terms of the rMSEtest\mathrm{rMSE}_{\mathrm{test}} for both Reynolds numbers. We observe that all the expressions are simple and compact.

ReComponentExpression
400g1g_1(−0.039u1+0.002u2−0.034)sin⁡(1.719u2−3.789)+0.000(-0.039u_1 + 0.002u_2 - 0.034)\sin(1.719u_2 - 3.789) + 0.000
400g2g_2(−0.077u1+0.031u4−0.073)sin⁡(1.062u1−1.804u2)+0.020(-0.077u_1 + 0.031u_4 - 0.073)\sin(1.062u_1 - 1.804u_2) + 0.020
400g3g_3−0.384u1+0.384u3+0.039u4+0.971sin⁡(1.868u1+0.123)−0.012-0.384u_1 + 0.384u_3 + 0.039u_4 + 0.971\sin(1.868u_1 + 0.123) - 0.012
400g4g_4(7.533u2−1.921u4)(0.706u2+0.046u4−sin⁡(0.846u2))−0.008(7.533u_2 - 1.921u_4)(0.706u_2 + 0.046u_4 - \sin(0.846u_2)) - 0.008
500g1g_10.015u1+0.010u4+sin⁡(0.191u1−0.100u5+sin⁡(0.064u4)−1.569)+0.9790.015u_1 + 0.010u_4 + \sin(0.191u_1 - 0.100u_5 + \sin(0.064u_4) - 1.569) + 0.979
500g2g_21.004(0.264u2u3−exp⁡(0.282u2))(0.007u1−0.005u2−0.014u3+0.006u4)−0.0021.004(0.264u_2u_3 - \exp(0.282u_2))(0.007u_1 - 0.005u_2 - 0.014u_3 + 0.006u_4) - 0.002
500g3g_3−0.999(0.033u1−0.005u5)(1.795u2−2.109u5)−(0.040u1−0.042)exp⁡(1.895u1)-0.999(0.033u_1 - 0.005u_5)(1.795u_2 - 2.109u_5) - (0.040u_1 - 0.042)\exp(1.895u_1)
500g4g_4−0.999(0.009u1−0.004u5+0.009)(1.449u2+3.357u3+1.044u5+2.355)-0.999(0.009u_1 - 0.004u_5 + 0.009)(1.449u_2 + 3.357u_3 + 1.044u_5 + 2.355)
500g5g_5−0.017u1−0.013u32+0.011u5+0.852(0.001u1−0.001u5)(17.071u4−31.025u5)−0.009-0.017u_1 - 0.013u_3^2 + 0.011u_5 + 0.852(0.001u_1 - 0.001u_5)(17.071u_4 - 31.025u_5) - 0.009

Table 3. 2D flow past a cylinder. Expressions of the components of the SR-ROM closure model that attains the minimum rMSEtest_{\mathrm{test}} in the extrapolation region reported in Table 1.

In Figure 4.3, we further plot the rMSEtest\mathrm{rMSE}_{\mathrm{test}} as a function of the number of parameters for each ROM. We note that the number of constants is reported for the SR-ROM and the number of weights for the other ROMs. We remark that the d2-VMS-ROM results for Re=400Re = 400 and r=3r = 3 do not appear in the plot, since the model’s kinetic energy diverges. Also, although the rMSEtest_{\mathrm{test}} s of SR-ROM and NN-ROM are similar for Re=500Re = 500 (Table 1), SR-ROM has an order of magnitude fewer parameters. Overall, the SR-ROM is the best model in terms of complexity and accuracy.

Figure 4.3. 2D flow past a cylinder. rMSEtest_{\mathrm{test}} as a function of the number of parameters of the model. For the SR-ROM and the NN-ROM, the mean values out of 5 independent runs are reported. Each point is marked using the following legend: point for r=2r = 2; triangle for r=3r = 3; square for r=4r = 4; pentagon for r=5r = 5.

Finally, the structure of the closure components gig_i obtained using symbolic regression for both Reynolds numbers is analyzed in Figure 4.4. Specifically, we count the mean occurrences of the ROM coefficients, and square, trigonometric, and exponential functions over 50 independent runs after simplifying the symbolic expressions. To clarify our definition of “occurrences”, let us consider the expression of the function g1g_1 for Re=500Re = 500, as shown in Table 3,

Figure 4.4. 2D flow past a cylinder. Statistics of the occurrences (over 50 independent runs) of the ROM coefficients and primitives appearing in the function gig_i found using symbolic regression.

g1(u1,u2,u3,u4,u5)≔0.015u1+0.01u4+1.0sin⁡(0.191u1−0.1u5+sin⁡(0.064u4)−1.57)+0.98.(33)g_1(u_1,u_2,u_3,u_4,u_5) \coloneqq0.015u_1 + 0.01u_4 + 1.0\sin(0.191u_1 - 0.1u_5 + \sin(0.064u_4) - 1.57) + 0.98. \tag*{(33)}

In the function g1g_1, there are two u1u_1, two u4u_4, one u5u_5, and two sine occurrences. The same rule is applied to all the components for each run, and we compute the mean and standard deviation (std) with respect to the number of runs. From Figure 4.4, we can observe that:

  • The logarithm and square are absent from almost all the expressions, thus suggesting that the (general) primitive set that we chose for SR could be reduced without loss in accuracy of the models;

  • Except for r=5r = 5, the first ROM coefficients u1u_1 and u2u_2 appear more frequently than the others (even though their frequency has a significant standard deviation). Intuitively, this could be related to the higher importance of the first modes in the reconstruction of the solution;

  • For r=5r = 5, the mean occurrences of all the considered terms are much more spread than the previous rr values.

We note that the expressions of gig_i found via SR do not show any common structure across the two values of Reynolds number. This is possibly a consequence of the training procedure, which was performed by including only data corresponding to a single Reynolds number.

2D Lid-Driven Cavity

Our next example is the 2D lid-driven cavity (LDC) problem, which is a more challenging test problem compared to the 2D flow past a cylinder. As demonstrated in [5], the problem requires more than 60 POD modes for G-ROM to accurately reconstruct the solutions and quantities of interest. A detailed description of the FOM setup for this problem can be found in [7]. We consider three Reynolds numbers, namely, Re⁡=10000\operatorname{Re}=10000, Re⁡=15000\operatorname{Re}=15000, and Re⁡=20000\operatorname{Re}=20000. The

time evolution of the FOM kinetic energy for the three Reynolds numbers is shown in Figure 4.5. The results indicate that the solutions at these Reynolds numbers are chaotic.

Figure 4.5. 2D lid-driven cavity. The FOM kinetic energy as a function of time tt for Re⁡=10000\operatorname{Re}=10000, 1500015000, 2000020000.

For each Re⁡\operatorname{Re}, K=2001K=2001 snapshots {uk:=u(x,tk)−φ0}k=1K\{u^{k}:=u(x,t^{k})-\varphi_{0}\}_{k=1}^{K} are collected in the time interval [T0,T0+80][T_{0},T_{0}+80] with sampling time step Δts=0.04\Delta t_{s}=0.04, and the zeroth mode, φ0\varphi_{0}, is set to be the FOM velocity field at t=T0t=T_{0}. We note that the time T0T_{0} depends on the Reynolds number and is determined such that the solutions are in the statistically steady state region. Snapshots of the velocity magnitude for each considered Re⁡\operatorname{Re} are shown in Figure 4.6. The snapshot data are then used to construct the reduced basis functions {φi}i=1r\{\varphi_{i}\}_{i=1}^{r} by leveraging POD, the VMS-ROM closure matrix T\mathcal{T}, and (19). The data-driven closure of each model is trained in the time interval [T0,T0+80][T_{0},T_{0}+80] by minimizing (18), and is selected based on the optimal hyperparameters that minimize (28) in the time interval [T0+80,T0+160][T_{0}+80,T_{0}+160]. Finally, we assess the generalization capabilities of all the ROMs in the extrapolation region, that is, in the time interval [T0,T0+400][T_{0},T_{0}+400].

Velocity magnitude snapshots for three Reynolds numbers

Figure 4.6. 2D lid-driven cavity. The velocity magnitude snapshot for Re⁡=10000\operatorname{Re}=10000, 1500015000, 2000020000.

In Table 4, the rMSEtest\mathrm{rMSE}_{\mathrm{test}} between the FOM kinetic energy and the predicted kinetic energy in the extrapolation region of all models is reported with respect to reduced space dimension r=2,3,4,5,6,7r = 2, 3, 4, 5, 6, 7. The rMSE of the SR-ROM and the NN-ROM is computed by taking the mean of 5 runs with 5 different seeds because these models rely on stochastic optimization algorithms. Similar to the 2D flow past a cylinder case, the results show that the SR-ROM is the most robust model, followed by the NN-ROM, d2-VMS-ROM, and LR-ROM. In particular, we note that for Re=10000\mathrm{Re} = 10000 and r=5,6r = 5, 6, Re=15000\mathrm{Re} = 15000 and r=3,6,7r = 3, 6, 7, and Re=20000\mathrm{Re} = 20000 and r=2r = 2, the SR-ROM rMSEtest\mathrm{rMSE}_{\mathrm{test}} in the extrapolation region is orders of magnitude lower than the rMSEtest\mathrm{rMSE}_{\mathrm{test}} of the other ROMs.

RerrLR-ROMd2-VMS-ROMSR-ROMNN-ROM
1000022.8×10−52.8 \times10^{-5}2.8×10−52.8 \times10^{-5}(2.8±0.0)×10−5(2.8 \pm0.0) \times10^{-5}(4.1±4.2)×10−6(4.1 \pm4.2) \times10^{-6}
30.22.8×10−32.8 \times10^{-3}(2.2±2.2)×10−5(2.2 \pm2.2) \times10^{-5}(2.9±1.3)×10−6(2.9 \pm1.3) \times10^{-6}
42.20.7(4.9±1)×10−6(4.9 \pm1) \times10^{-6}(5.0±3.3)×10−6(5.0 \pm3.3) \times10^{-6}
51.5×10−31.5 \times10^{-3}1.5×10−31.5 \times10^{-3}(4.2±5.2)×10−5(4.2 \pm5.2) \times10^{-5}(5.4±1.5)×10−3(5.4 \pm1.5) \times10^{-3}
63.1×1023.1 \times10^{2}7(1.5±1.3)×10−5(1.5 \pm1.3) \times10^{-5}(2.7±5.5)×102(2.7 \pm5.5) \times10^{2}
72.3×10−52.3 \times10^{-5}2.5×10−52.5 \times10^{-5}(2.9±0.2)×10−5(2.9 \pm0.2) \times10^{-5}(1.9±3.1)×10−5(1.9 \pm3.1) \times10^{-5}
1500021.4×10−41.4 \times10^{-4}8.4×10−58.4 \times10^{-5}(2.8±0.6)×10−4(2.8 \pm0.6) \times10^{-4}(4.5±4.3)×10−5(4.5 \pm4.3) \times10^{-5}
35.1×10−35.1 \times10^{-3}1.8×10−41.8 \times10^{-4}(3.9±0.5)×10−5(3.9 \pm0.5) \times10^{-5}(0.6±1.2)×106(0.6 \pm1.2) \times10^{6}
44×10−44 \times10^{-4}1.6×10−41.6 \times10^{-4}(8.8±1.5)×10−5(8.8 \pm1.5) \times10^{-5}(1.0±0.4)×10−4(1.0 \pm0.4) \times10^{-4}
51.9×10−41.9 \times10^{-4}9.2×10−59.2 \times10^{-5}0.01±0.010.01 \pm0.01(7.2±3.8)×10−5(7.2 \pm3.8) \times10^{-5}
60.061.4×10−41.4 \times10^{-4}(4.8±0.8)×10−5(4.8 \pm0.8) \times10^{-5}(6.6±10)×10−4(6.6 \pm10) \times10^{-4}
70.121.7×10−41.7 \times10^{-4}(8.2±3.3)×10−5(8.2 \pm3.3) \times10^{-5}0.04±0.070.04 \pm0.07
2000021.5×10−31.5 \times10^{-3}1.1×10−31.1 \times10^{-3}(2.2±0.6)×10−4(2.2 \pm0.6) \times10^{-4}(1.7±0.1)×10−3(1.7 \pm0.1) \times10^{-3}
31.1×10−41.1 \times10^{-4}6.3×10−46.3 \times10^{-4}(1.7±0.4)×10−4(1.7 \pm0.4) \times10^{-4}(5.1±6.3)×10−4(5.1 \pm6.3) \times10^{-4}
40.36.2×10−46.2 \times10^{-4}(1.8±0.4)×10−4(1.8 \pm0.4) \times10^{-4}0.7±0.70.7 \pm0.7
52.6×10−22.6 \times10^{-2}3.6×10−33.6 \times10^{-3}(3.3±6.2)×10−3(3.3 \pm6.2) \times10^{-3}(3.8±5.2)×10−2(3.8 \pm5.2) \times10^{-2}
68.1×10−48.1 \times10^{-4}2.9×10−32.9 \times10^{-3}nan(1.8±1.3)×10−3(1.8 \pm1.3) \times10^{-3}
72×10−42 \times10^{-4}8.6×10−48.6 \times10^{-4}(3.4±2.6)×10−4(3.4 \pm2.6) \times10^{-4}0.1±0.20.1 \pm0.2

Table 4. 2D lid-driven cavity. Relative MSE (rMSEtest\mathrm{rMSE}_{\mathrm{test}}) between the FOM kinetic energy and the predicted kinetic energy in the testing region for all models and reduced space dimension r=2,3,4,5,6,7r = 2, 3, 4, 5, 6, 7. For the SR-ROM and the NN-ROM, the mean and the standard deviation over 5 independent runs (different seeds) are reported. In bold, the best performing model for each case is listed.

We further measure the closeness between the VMS-ROM closure matrix TT and the reconstructed closure for each model by computing the mean MSE (32), see Table 5. We observe that the SR-ROM is the most robust method also in this case. Furthermore, we note that, unlike in the MSE of predicted energy (Table 4), the NN-ROM has a much larger MSE than the SR-ROM. We stress again that this phenomenon is a consequence of the L2L^2 regularization, which is not used for the SR-ROM.

Table 5: 2D lid-driven cavity. Mean squared error (MSE‾tr\overline{\mathrm{MSE}}_{\mathrm{tr}}) between the VMS-ROM closure and the ML-VMS-ROM reconstructed closures for Re=10000\mathrm{Re}=10000, 1500015000, 2000020000. In bold, the best performing model for each case is listed.

RerrLR-ROMd2-VMS-ROMSR-ROMNN-ROM
1000022.0×10−72.0 \times10^{-7}2.0×10−72.0 \times10^{-7}(1.0±0.0)×10−7(1.0 \pm0.0) \times10^{-7}(8.9±3.7)×10−4(8.9 \pm3.7) \times10^{-4}
31.4×10−71.4 \times10^{-7}1.2×10−71.2 \times10^{-7}(9.4±0.5)×10−8(9.4 \pm0.5) \times10^{-8}(3.3±2.1)×10−4(3.3 \pm2.1) \times10^{-4}
41.2×10−71.2 \times10^{-7}1.4×10−71.4 \times10^{-7}(7.2±0.8)×10−8(7.2 \pm0.8) \times10^{-8}(4.7±2.6)×10−4(4.7 \pm2.6) \times10^{-4}
51.3×10−71.3 \times10^{-7}1.3×10−71.3 \times10^{-7}(6.4±0.8)×10−8(6.4 \pm0.8) \times10^{-8}(2.4±0.9)×10−7(2.4 \pm0.9) \times10^{-7}
69.4×10−89.4 \times10^{-8}9.3×10−89.3 \times10^{-8}(5.8±0.6)×10−8(5.8 \pm0.6) \times10^{-8}(1.9±1.3)×10−4(1.9 \pm1.3) \times10^{-4}
71.0×10−71.0 \times10^{-7}1.0×10−71.0 \times10^{-7}(5.2±0.7)×10−8(5.2 \pm0.7) \times10^{-8}(1.2±1.2)×10−5(1.2 \pm1.2) \times10^{-5}
1500021.3×10−51.3 \times10^{-5}1.4×10−51.4 \times10^{-5}(6.6±1.8)×10−6(6.6 \pm1.8) \times10^{-6}(4.8±1.1)×10−4(4.8 \pm1.1) \times10^{-4}
37.9×10−67.9 \times10^{-6}4.5×10−64.5 \times10^{-6}(3.5±0.0)×10−6(3.5 \pm0.0) \times10^{-6}(1.9±2.5)×10−4(1.9 \pm2.5) \times10^{-4}
45.3×10−65.3 \times10^{-6}3.1×10−63.1 \times10^{-6}(2.9±0.4)×10−7(2.9 \pm0.4) \times10^{-7}(0.1±2.2)×10−3(0.1 \pm2.2) \times10^{-3}
54.0×10−64.0 \times10^{-6}2.3×10−62.3 \times10^{-6}(2.3±0.4)×10−6(2.3 \pm0.4) \times10^{-6}(9.7±7.1)×10−6(9.7 \pm7.1) \times10^{-6}
63.6×10−63.6 \times10^{-6}2.1×10−62.1 \times10^{-6}(2±0.4)×10−6(2 \pm0.4) \times10^{-6}(8.7±3.8)×10−6(8.7 \pm3.8) \times10^{-6}
72.7×10−62.7 \times10^{-6}2.5×10−62.5 \times10^{-6}(1.7±1.9)×10−6(1.7 \pm1.9) \times10^{-6}(1.1±0.5)×10−5(1.1 \pm0.5) \times10^{-5}
2000029.0×10−69.0 \times10^{-6}9×10−69 \times10^{-6}(5.4±0.6)×10−6(5.4 \pm0.6) \times10^{-6}(9.0±0.0)×10−6(9.0 \pm0.0) \times10^{-6}
39.2×10−69.2 \times10^{-6}8.7×10−68.7 \times10^{-6}(4.7±1.2)×10−6(4.7 \pm1.2) \times10^{-6}(4.2±5.5)×10−6(4.2 \pm5.5) \times10^{-6}
41.1×10−51.1 \times10^{-5}9.3×10−69.3 \times10^{-6}(4.3±0.5)×10−6(4.3 \pm0.5) \times10^{-6}(1.1±0.1)×10−5(1.1 \pm0.1) \times10^{-5}
59.8×10−69.8 \times10^{-6}1.0×10−51.0 \times10^{-5}(6.5±0.9)×10−6(6.5 \pm0.9) \times10^{-6}(3.2±4)×10−5(3.2 \pm4) \times10^{-5}
67.4×10−67.4 \times10^{-6}7.6×10−67.6 \times10^{-6}(4.9±0.5)×10−6(4.9 \pm0.5) \times10^{-6}(1.0±1.8)×10−4(1.0 \pm1.8) \times10^{-4}
77.3×10−67.3 \times10^{-6}7.6×10−67.6 \times10^{-6}(5.3±0.8)×10−6(5.3 \pm0.8) \times10^{-6}(7.6±0.1)×10−6(7.6 \pm0.1) \times10^{-6}

Table 5.

Similarly to § 4.1, we report in Table 6 the best symbolic expressions found for the three Reynolds numbers considered. These expressions are slightly more complicated than the ones in Table 3, but this is due to the chaotic, thus more complex, nature of the problem. Also, we remark that for Re=15000\mathrm{Re} = 15000 and 20000, the best models found depend only polynomially on the FOM coefficients uru_r. Hence, also the d2-VMS-ROM could, in principle, find the same models. However, with respect to the rMSEtest\mathrm{rMSE}_{\mathrm{test}} in the extrapolation region, a worse model is found, and this is due to the ill-conditioned linear system to be solved to train this method [12].

ReComponentExpression
10000g1g_10.0002u1+0.0901u2+0.0003u3−0.0004u4+(0.0001u1−0.0001u2)∗cos⁡(3.7331u4−2.1130)+(0.0002u1+0.0010u2+0.0005u3−0.0007u4+0.0004)∗sin⁡(0.0686u1−cos⁡(0.6297u1−0.1635u3))−0.999sin⁡(0.0897u2)+0.00030.0002u_1 + 0.0901u_2 + 0.0003u_3 - 0.0004u_4 + (0.0001u_1 - 0.0001u_2) * \cos(3.7331u_4 - 2.1130) + (0.0002u_1 + 0.0010u_2 + 0.0005u_3 - 0.0007u_4 + 0.0004) * \sin(0.0686u_1 - \cos(0.6297u_1 - 0.1635u_3)) - 0.999\sin(0.0897u_2) + 0.0003
10000g2g_2−0.0003u1+0.0003u2+0.0002u3−sin⁡((0.0883u2(1.7855u1+1.8087u2)−sin⁡(1.2270u2+0.5128u4−7.0881)+0.9256)sin⁡(sin⁡(sin⁡(0.0006u3)−0.0009))∗sin⁡(sin⁡(sin⁡(sin⁡(0.8346u1))+cos⁡(0.7177u1+0.9317))))−0.0004-0.0003u_1 + 0.0003u_2 + 0.0002u_3 - \sin((0.0883u_2(1.7855u_1 + 1.8087u_2) - \sin(1.2270u_2 + 0.5128u_4 - 7.0881) + 0.9256)\sin(\sin(\sin(0.0006u_3) - 0.0009)) * \sin(\sin(\sin(\sin(0.8346u_1)) + \cos(0.7177u_1 + 0.9317)))) - 0.0004
10000g3g_30.0001u2+0.0001u3+(0.0005u1u3−0.0003u2−0.0003u4)(0.4274u1−0.1150u4+(0.0507u1−0.0376u2cos⁡(1.1193u2)+0.0342u3)(1.6577u1+0.5313u2−7.1728)+sin⁡(0.7269u2+1.0911u3+0.5615)cos⁡(0.6477u1))0.0001u_2 + 0.0001u_3 + (0.0005u_1u_3 - 0.0003u_2 - 0.0003u_4)(0.4274u_1 - 0.1150u_4 + (0.0507u_1 - 0.0376u_2\cos(1.1193u_2) + 0.0342u_3)(1.6577u_1 + 0.5313u_2 - 7.1728) + \sin(0.7269u_2 + 1.0911u_3 + 0.5615)\cos(0.6477u_1))
10000g4g_41.0010sin⁡(0.0299u2+0.0659u4+(0.3499u1−cos⁡(1.6868u1))(0.4145u4+sin⁡((0.4721u2−sin⁡(0.7269u3))cos⁡(0.6962u1)+2.7152)0.0006−sin⁡(0.0301u2+0.0659u4))1.0010\sin(0.0299u_2 + 0.0659u_4 + (0.3499u_1 - \cos(1.6868u_1))(0.4145u_4 + \sin((0.4721u_2 - \sin(0.7269u_3))\cos(0.6962u_1) + 2.7152)0.0006 - \sin(0.0301u_2 + 0.0659u_4))
15000g1g_1−0.0022−0.0006u1+0.0004u02(1.9712+0.5332u0)−0.0001u1u0−(−0.0004−0.0001u1)(0.7896u2+2.7564u0)-0.0022 - 0.0006u_1 + 0.0004u_0^2(1.9712 + 0.5332u_0) - 0.0001u_1u_0 - (-0.0004 - 0.0001u_1)(0.7896u_2 + 2.7564u_0)
15000g2g_20.0001+0.9990(20.9826+9.6691u0+7.9730u2+(1.1479+0.0092u0(−22.8110u0+27.8969u1+9.8753u2)−3.1177u1(0.1684+0.0210u2))(−13.6013−8.9292u1+(−2.5189+0.2241u0−1.2826u2)(6.0625+2.3368u0+1.1539u1−0.0070u2)))0.0001 + 0.9990(20.9826 + 9.6691u_0 + 7.9730u_2 + (1.1479 + 0.0092u_0(-22.8110u_0 + 27.8969u_1 + 9.8753u_2) - 3.1177u_1(0.1684 + 0.0210u_2))(-13.6013 - 8.9292u_1 + (-2.5189 + 0.2241u_0 - 1.2826u_2)(6.0625 + 2.3368u_0 + 1.1539u_1 - 0.0070u_2)))
15000g2g_2(−0.0002+0.0001u1+0.0002u2)(-0.0002 + 0.0001u_1 + 0.0002u_2)
15000g3g_30.0038−0.0006u0−0.0019u2+0.0001u2u0+(0.0010−0.0007u2)(−2.4952+0.0602u1+0.1949u2+(0.2424u1+0.1948u2u1)(1.1255u0−1.2423u1+2.9477u2))0.0038 - 0.0006u_0 - 0.0019u_2 + 0.0001u_2u_0 + (0.0010 - 0.0007u_2)(-2.4952 + 0.0602u_1 + 0.1949u_2 + (0.2424u_1 + 0.1948u_2u_1)(1.1255u_0 - 1.2423u_1 + 2.9477u_2))
20000g1g_10.0002u1+0.0008u2−0.0005u3+(6.0344u2−4.848u4)(0.0001u1(1.9919u1u3∗(3.2164u1+1.9255u4−3.5296)−4.4491u2−2.9491u3+8.9232))−0.00040.0002u_1 + 0.0008u_2 - 0.0005u_3 + (6.0344u_2 - 4.848u_4)(0.0001u_1(1.9919u_1u_3 * (3.2164u_1 + 1.9255u_4 - 3.5296) - 4.4491u_2 - 2.9491u_3 + 8.9232)) - 0.0004
20000g2g_21.0021(0.3193u2(0.7430u1+1.8020u4)(0.1953u2−1.4475)(5.9033u2+10.0442)+12.5300u4−(0.5016u2−4.5800u3)(1.1975u2+2.6217)(0.3743u3−0.0664))∗(0.0002u1+0.0001u2)−0.00011.0021(0.3193u_2(0.7430u_1 + 1.8020u_4)(0.1953u_2 - 1.4475)(5.9033u_2 + 10.0442) + 12.5300u_4 - (0.5016u_2 - 4.5800u_3)(1.1975u_2 + 2.6217)(0.3743u_3 - 0.0664)) * (0.0002u_1 + 0.0001u_2) - 0.0001
20000g3g_30.0003u3−0.0002u4+(0.0001u2−0.0001u3)(1.7047u2+2.8970u4−0.9176)(13.6070u1−1.5597u3+5.6265u4−6.6889)−(0.0021u2+0.0011u4)(1.5428u2−0.6498u3)+0.0020.0003u_3 - 0.0002u_4 + (0.0001u_2 - 0.0001u_3)(1.7047u_2 + 2.8970u_4 - 0.9176)(13.6070u_1 - 1.5597u_3 + 5.6265u_4 - 6.6889) - (0.0021u_2 + 0.0011u_4)(1.5428u_2 - 0.6498u_3) + 0.002
20000g4g_4−0.9953(0.0001u2+0.0001u32−0.0001u3+0.001u4−(0.0001u1+0.0010)∗(1.2730u4+0.300))(4.1908u22+9.7453u2−0.1684u3(70.4595u1+13.7008)+4.0931u4+3.2191)-0.9953(0.0001u_2 + 0.0001u_3^2 - 0.0001u_3 + 0.001u_4 - (0.0001u_1 + 0.0010) * (1.2730u_4 + 0.300))(4.1908u_2^2 + 9.7453u_2 - 0.1684u_3(70.4595u_1 + 13.7008) + 4.0931u_4 + 3.2191)

Table 6. 2D lid driven cavity. Expressions of the components of the SR-ROM closure model that attains the minimum rMSEtest_{\mathrm{test}} in the extrapolation region among those reported in Table 4.

Figure 4.8 confirms, also in this case, that the SR-ROM is the best trade-off between generalization capability (rMSEtest\mathrm{rMSE}_{\mathrm{test}}) and model complexity. Again, the SR-ROM and NN-ROM manifest a similar behavior in terms of rMSEtest\mathrm{rMSE}_{\mathrm{test}}, especially for Re=10000\mathrm{Re} = 10000 and 15000, but the SR-ROM has an order of magnitude fewer parameters. Also, the SR-ROM results for Re=20000\mathrm{Re} = 20000 and r=6r = 6 are not shown in the figure, since the kinetic energy of one of the SR-ROM best models diverges.

Figure 4.8. 2D lid-driven cavity. rMSEtestr\mathrm{MSE}_{\mathrm{test}} as a function of the number of parameters of the model. For the SR-ROM and NN-ROM, the mean values over 5 independent runs are reported. Each point is marked using the following legend: point for r=2r = 2; triangle for r=3r = 3; square for r=4r = 4; pentagon for r=5r = 5; hexagon for r=6r = 6; octagon for r=7r = 7.

As in § 4.1, we compute the mean occurrences of some terms using the same strategy. Also in this case, we can make a qualitative analysis:

  • The logarithm and square are basically absent for every rr and Re\mathrm{Re};

  • For r=5r=5, 66, 77 and for each Reynolds number, the mean occurrence of every term is uniform, i.e., none of the terms in the expressions is dominant.

As in § 4.1, no evident structure emerges in the expressions of Table 6 across different values of the Reynolds number because the training procedure of all the ROMs is carried out at a fixed Reynolds number.

In Figure 4.7, we show the kinetic energy as a function of time for the ROMs and the FOM with reduced space dimension r=3,5,7r=3,5,7, for Re=10000\mathrm{Re}=10000 and Re=20000\mathrm{Re}=20000. For Re=10000\mathrm{Re}=10000, the SR-ROM outperforms the other ROMs for r=5r=5, while for r=3,7r=3,7 the SR-ROM is still among the best performing models. We observe that the d2-VMS-ROM and LR-ROM deviate in the extrapolation region for r=3,5r=3,5, as does the NN-ROM for r=5r=5. The

Figure 4.7. 2D lid-driven cavity at Re=10000\mathrm{Re}=10000 and Re=20000\mathrm{Re}=20000. Comparison of the kinetic energy between the FOM and the ROMs with r=3,5,7r=3,5,7.

SR-ROM is the only method that remains close to the FOM in this case. Finally, for Re = 20000, the SR-ROM is the best method for r=3,5,7r = 3, 5, 7.

Figure 4.9. 2D lid-driven cavity, r=2,3,4r = 2, 3, 4. Statistics of the occurrences (over 50 independent runs) of the ROM coefficients and primitives appearing in the closure model found by SR-ROM.

Figure 4.10. 2D lid-driven cavity, r=5,6,7r = 5, 6, 7. Statistics of the occurrences (over 50 independent runs) of the ROM coefficients and primitives in the closure model found by SR-ROM.

Conclusions and Future Work

In this paper, we propose a novel symbolic regression (SR) closure strategy for ROM, which leverages available data and SR to find the ROM closure model. The new SR-ROM closure model combines the advantages of current data-driven ROM closure approaches and eliminates their drawbacks: The new data-driven SR closures yield ROMs that are interpretable, parsimonious, accurate, generalizable, and robust. To compare the SR-ROM closures with the current data-driven ROM closures (i.e., structural and machine learning ROM closures), we consider a specific ROM closure framework, the data-driven variational multiscale ROM. Within this framework, we compare the novel SR-ROM closure with three other data-driven ROM closures: a linear regression ROM closure, a quadratic data-driven ROM closure, and a neural network ROM closure. To investigate the four types of ROM closures, we consider two under-resolved, convection-dominated test problems: the flow past a cylinder and the lid-driven cavity flow. To assess the performance of the different types of closures, we monitor the performance of the ROMs obtained by adding the four ROM closure models.

Our numerical investigation yields the following conclusions:

  • For both test problems, in the predictive regime, the new SR-ROM closure yields more accurate results than the other three ROM closures (i.e., the linear regression, quadratic data-driven, and neural network ROM closures);

  • The SR-ROM is the most robust model with respect to the ROM space dimension, rr;

  • Our numerical results also show that the new SR-ROM closure has orders of magnitude fewer terms than the neural network closure model;

  • The SR strategy is more accurate than the other three data-driven approaches in reconstructing the ROM closure term.

The first steps in the numerical investigation of the new SR-ROM have been encouraging. There are, however, several other research directions that should be pursued next. For example, in this paper, we did not find common structures in the SR-ROM closures for different Reynolds numbers. This is because we used training data for a fixed Reynolds number. To allow SR to discover common structures in the new SR-ROM closures, we plan to train it with data from different Reynolds numbers. We also note that, although the SR-ROM is used in this paper for prediction in time, the Reynolds number is fixed. We plan to extend the new SR-ROM to parametric problems, and use it to predict solutions and quantities of interests at other Reynolds numbers. Another interesting research direction is the investigation of the new SR-ROM strategy in ROM closure frameworks that are different from the variational multiscale ROM closure approach considered in this paper. For example, we plan to investigate the new SR-ROM closure for large eddy simulation ROMs constructed by using ROM spatial filtering [12]. Finally, we plan to compare the new SR-ROM closure strategy with other data-driven modeling approaches, e.g., operator inference [31, 41], SINDy [42, 43], and LaSDI [44].

CRediT Authorship Contribution Statement

Simone Manti: Writing - original draft, Writing - review and editing, Data curation, Investigation, Conceptualization, Validation, Visualization, Methodology, Software. Ping-Hsuan Tsai: Writing - original draft, Writing - review and editing, Data curation, Investigation, Conceptualization, Validation, Visualization, Methodology, Software. Alessandro Lucantonio: Writing - review and editing, Conceptualization, Validation, Methodology, Resources, Funding acquisition, Supervision. Traian Iliescu: Writing - original draft, Writing - review and editing, Conceptualization, Validation, Methodology, Funding acquisition, Supervision.

Acknowledgments

The work of SM and AL is supported by the European Union (European Research Council (ERC), ALPS, 101039481). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. Computational resources have been partially provided by DeiC National HPC (DeiC-AU-N1-2023030). TI acknowledges support through National Science Foundation grants DMS-2012253 and CDS&E-MSS-1953113.

References

  1. [1]J. S. Hesthaven, G. Rozza, and B. Stamm. Certified Reduced Basis Methods for Parametrized Partial Differential Equations. Springer, 2015.DOI
  2. [2]A. Quarteroni, A. Manzoni, and F. Negri. Reduced Basis Methods for Partial Differential Equations: An Introduction, volume 92. Springer, 2015.DOI
  3. [3]P.T. Tsai, P. Fischer, and E. Solomonik. Accelerating the Galerkin reduced-order model with the tensor decomposition for turbulent flows. arXiv preprint, http://arxiv.org/abs/2311.03694, 2023.
  4. [4]S. E. Ahmed, S. Pawar, O. San, A. Rasheed, T. Iliescu, and B. R. Noack. On closures for reduced order models – A spectrum of first-principle to machine-learned avenues. Phys. Fluids, 33(9):091301, 2021.
  5. [5]L. Fick, Y. Maday, A. T. Patera, and T. Taddei. A stabilized POD model for turbulent flows over a range of Reynolds numbers: Optimal parameter sampling and constrained projection. J. Comp. Phys., 371:214–243, 2018.DOI
  6. [6]K. Kaneko. An augmented basis method for reduced order models of turbulent flow. PhD thesis, 2022.
  7. [7]K. Kaneko, P.-H. Tsai, and P. Fischer. Towards model order reduction for fluid-thermal analysis. Nucl. Eng. Des., 370:108666, 2020.DOI
  8. [8]E. J. Parish, M. Yano, I. Tezaur, and T. Iliescu. Residual-based stabilized reduced-order models of the transient convection-diffusion-reaction equation obtained through discrete and continuous projection. Arch. Comput. Methods Eng., pages 1–45, 2024.arxiv.org/abs/2302.09355
  9. [9]P.H. Tsai, P. Fischer, and T. Iliescu. A time-relaxation reduced order model for the turbulent channel flow. J. Comput. Phys., page 113563, 2024.
  10. [10]B. Sanderse, P. Stinis, R. Maulik, and S. E. Ahmed. Scientific machine learning for closure models in multiscale problems: A review. arXiv preprint arXiv:2403.02913, 2024.arxiv.org/abs/2403.02913
  11. [11]C. Mou, B. Koc, O. San, L. G. Rebholz, and T. Iliescu. Data-driven variational multiscale reduced order models. Comput. Methods Appl. Mech. Engrg., 373:113470, 2021.DOI
  12. [12]X. Xie, M. Mohebujjaman, L. G. Rebholz, and T. Iliescu. Data-driven filtered reduced order modeling of fluid flows. SIAM J. Sci. Comput., 40(3):B834–B857, 2018.arxiv.org/abs/1709.04362
  13. [13]S. E. Ahmed, O. San, A. Rasheed, T. Iliescu, and A. Veneziani. Physics guided machine learning for variational multiscale reduced order modeling. SIAM J. Sci. Comput., 45(3):B283–B313, 2023.arxiv.org/abs/2205.12419
  14. [14]X. Xie, C. Webster, and T. Iliescu. Closure learning for nonlinear model reduction using deep residual neural network. Fluids, 5(1):39, 2020.arxiv.org/abs/1910.10298
  15. [15]C. Mou, Z. Wang, D. R. Wells, X. Xie, and T. Iliescu. Reduced order models for the quasi-geostrophic equations: A brief survey. Fluids, 6(1):16, 2021.
  16. [16]T. J. R. Hughes. Multiscale phenomena: Green’s functions, the Dirichlet-to-Neumann formulation, subgrid scale models, bubbles and the origins of stabilized methods. Comput. Methods Appl. Mech. Engrg., 127(1-4):387–401, 1995.DOI
  17. [17]Y. Maday, A. T. Patera, and E. M. Ronquist. A well-posed optimal spectral element approximation for the Stokes problem. Technical report, 1987.
  18. [18]P. Fischer, M. Schmitt, and A. Tomboulides. Recent developments in spectral element simulations of moving-domain problems. In Recent Progress and Modern Challenges in Applied Mathematics, Modeling and Computational Science, pages 213–244. Springer, 2017.DOI
  19. [19]P.T. Tsai and P. Fischer. Parametric model-order-reduction development for unsteady convection. Front. Phys., 10:903169, 2022.DOI
  20. [20]G. Berkooz, P. Holmes, and J. L. Lumley. The proper orthogonal decomposition in the analysis of turbulent flows. Ann. Rev. Fluid Mech., 25(1):539–575, 1993.
  21. [21]S. Volkwein. Proper Orthogonal Decomposition: Theory and Reduced-Order Modelling. Lecture Notes, University of Konstanz, 2013. http://www.math.uni-konstanz.de/numerik/personen/volkwein/teaching/POD-Book.pdf.
  22. [22]F. Ballarin, A. Manzoni, A. Quarteroni, and G. Rozza. Supremizer stabilization of POD–Galerkin approximation of parametrized steady incompressible Navier–Stokes equations. Int. J. Numer. Meth. Engng., 102:1136–1161, 2015.DOI
  23. [23]V. DeCaria, T. Iliescu, W. Layton, M. McLaughlin, and M. Schneier. An artificial compression reduced order model. SIAM J. Numer. Anal., 58(1):565–589, 2020.arxiv.org/abs/1902.09061
  24. [24]B. R. Noack, P. Papas, and P. A. Monkewitz. The need for a pressure-term representation in empirical Galerkin models of incompressible shear flows. J. Fluid Mech., 523:339–365, 2005.DOI
  25. [25]T. J. R. Hughes, G. R. Feijóo, L. Mazzei, and J.-B. Quincy. The variational multiscale method – a paradigm for computational mechanics. Comput. Methods Appl. Mech. Engrg., 166(1):3–24, 1998.DOI
  26. [26]N. Ahmed, T. C. Rebollo, V. John, and S. Rubino. A review of variational multiscale methods for the simulation of turbulent incompressible flows. Arch. Comput. Method. E., 24(1):115–164, 2017.
  27. [27]R. Codina, S. Badia, J. Baiges, and J. Principe. Variational multiscale methods in computational fluid dynamics. Encyclopedia of Computational Mechanics Second Edition, pages 1–28, 2018.DOI
  28. [28]B. Koc, M. Mohebujjaman, C. Mou, and T. Iliescu. Commutation error in reduced order modeling of fluid flows. Adv. Comput. Math., 45(5-6):2587–2621, 2019.arxiv.org/abs/1810.00517
  29. [29]H. Trevor, T. Robert, and F. Jerome. The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer, 2009.
  30. [30]J. Gareth, W. Daniela, H. Trevor, and T. Robert. An Introduction to Statistical Learning: With Applications in R. Springer, 2021.
  31. [31]B. Peherstorfer and K. Willcox. Data-driven operator inference for nonintrusive projection-based model reduction. Comput. Methods Appl. Mech. Engrg., 306:196–215, 2016.DOI
  32. [32]G. Kronberger, B. Burlacu, M. Kommenda, S. M. Winkler, and M. Affenzeller. Symbolic Regression. CRC Press, 2024.DOI
  33. [33]S. Manti and A. Lucantonio. Discovering interpretable physical models using symbolic regression and discrete exterior calculus. Mach. Learn.: Sci. Technol., 5(1):015005, 2024.arxiv.org/abs/2310.06609
  34. [34]J. R. Koza. Genetic Programming as a Means for Programming Computers by Natural Selection, volume 4. Stat. Comput., 1994.DOI
  35. [35]R. Poli, W.B. Langdon, and N.F. McPhee. A Field Guide to Genetic Programming. Lulu Enterprises UK, 2009.
  36. [36]B. Burlacu, G. Kronberger, and M. Kommenda. Operon C++: An efficient genetic programming framework for symbolic regression. GECCO ’20, page 1562–1570, New York, NY, USA, 2020. Association for Computing Machinery.DOI
  37. [37]W. La Cava, B. Burlacu, M. Virgolini, M. Kommenda, P. Orzechowski, F. O. de França, Y. Jin, and J. H. Moore. Contemporary symbolic regression methods and their relative performance. NeurIPS, 2021(D1):1, 2021.
  38. [38]D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature, 323(6088):533–536, 1986.
  39. [39]I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
  40. [40]M. Tietz, T. J. Fan, D. Nouri, B. Bossan, and Skorch Developers. skorch: A scikit-learn compatible neural network library that wraps pytorch. https://skorch.readthedocs.io/en/stable, 2017.
  41. [41]N. Aretz and K. Willcox. Exploiting structure via nested operator inference in physics-based learning. MORe 2024, page 12.
  42. [42]S. L. Brunton, J. L. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci., 113(15):3932–3937, 2016.DOI
  43. [43]D. A. Messenger and D. M. Bortz. Weak SINDy for partial differential equations. J. Comput. Phys., 443:110525, 2021.DOI
  44. [44]W. D. Fries, X. He, and Y. Choi. Lasdi: Parametric latent space dynamics identification. Comput. Methods Appl. Mech. Engrg., 399:115436, 2022.DOI

Paper details

vlnc:W_6ct9kfkj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents