WorksComputer Science
Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics
Abstract
Model reduction for fluid flow simulation continues to be of great interest across a number of scientific and engineering fields. Here, we explore the use of Neural Ordinary Differential Equations, a recently introduced family of continuous-depth, differentiable networks (Chen et al 2018), as a way to propagate latent-space dynamics in reduced order models. We compare their behavior with two classical non-intrusive methods based on proper orthogonal decomposition and radial basis function interpolation as well as dynamic mode decomposition. The test problems we consider include incompressible flow around a cylinder as well as real-world applications of shallow water hydrodynamics in riverine and estuarine systems. Our findings indicate that Neural ODEs provide an elegant framework for stable and accurate evolution of latent-space dynamics with a promising potential of extrapolatory predictions. However, in order to facilitate their widespread adoption for large-scale systems, significant effort needs to be directed at accelerating their training times. This will enable a more comprehensive exploration of the hyperparameter space for building generalizable Neural ODE approximations over a wide range of system dynamics.
Introduction
Despite the trend of hardware improvements and significant gains in the algorithmic efficiency of standard discretization procedures, high-fidelity numerical simulation of engineering systems governed by nonlinear partial differential equations still pose a prohibitive computational challenge Quarteroni, Manzoni, and Negri [50] for several decision-making applications involving control Proctor, Brunton, and Kutz [49], optimal design and multi-fidelity optimization Peherstorfer, Willcox, and Gunzburger [48], and/or uncertainty quantification Sapsis and Majda [55]. Reduced order models (ROMs) offer a valuable alternative way to simulate such dynamical systems with considerably reduced computational cost Benner, Gugercin, and Willcox [9].
Reduced basis (RB) methods Quarteroni, Manzoni, and Negri [50] constitute a family of widely popular ROM techniques that are usually implemented with an offline-online decomposition paradigm. The offline stage involves the construction of a solution-dependent, linear basis space spanned by a set of RB “modes”, which are extracted from a collection of high-fidelity solutions, also called snapshots. The RB “modes” can be thought of as a set of global basis functions spanning a linear subspace that can be used to approximate the dynamics of the high-fidelity model. The most well known method to extract the reduced basis is called proper orthogonal decomposition (POD) Sirovich [57]; ); Berkooz, Holmes, and Lumley [10], which is particularly effective when the coherent structures of the flow can be hierarchically ranked in terms of their energy content.
In the online stage of traditional RB methods, a linear combination of the reduced order RB modes is used to approximate the high-fidelity solution for a new configuration of flow parameters. The procedure adopted to compute the expansion coefficients leads to the classification of these methods into two broad categories: intrusive and non-intrusive. In an intrusive RB method, the expansion coefficients are determined by the solution of a reduced order system of equations, which is typically obtained via a Galerkin or Petrov-Galerkin projection of the high-fidelity (full-order) system onto the RB space Lozovskiy, Farthing, and Kees [43]. Typically this projection and solution involves modification of high-fidelity simulators and hence the label intrusive. For linear systems, Galerkin projection is the most popular choice. However, in the presence of nonlinearities, an affine expansion of the nonlinear (or non-affine) differential operator must be recovered in order to make the evaluation of the projection-based reduced model independent of the number of DOFs of the high-fidelity solution.
Several different techniques, collectively referred to as hyper-reduction methods Amsallem et al. [5], have been proposed to address this problem. These include the empirical interpolation method (EIM), its discrete counterpart DEIM Chaturantabut and Sorensen [17], “gappy POD” Willcox [64], as well as the residual DEIM method Xiao et al. [66]. Beyond the need for hyper-reduction to recover efficiency, in complex nonlinear problems it is also common that some of the intrinsic structures present in the high-fidelity model may be lost during order reduction using Galerkin projection-based approaches. This is because the Galerkin projection approach inherently assumes that the residual generated by the truncated representation of the high-fidelity model is orthogonal to the reduced basis space
Copyright © 2021, for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). which leads to the loss of higher order nonlinear interaction terms in the reduced representation. This can result in qualitatively wrong solutions or instability issues [4]. As a remedy, Petrov-Galerkin projection based approaches have been proposed [14, 15, 22].
An alternative family of methods to address the issues of instability and loss of efficiency in the intrusive ROM frameworks is represented by non-intrusive reduced order models (NIROMs), and forms the subject of this study. The primary advantage of this class of methods is that complex modifications to the source code describing the physical model can be avoided, thus making it easier to develop reduced models when the legacy or proprietary source codes are not available. In these methods, instead of a Galerkin-type projection, the expansion coefficients for the reduced solution are obtained via interpolation on the space of a reduced basis extracted from snapshot data. However, since the reduced dynamics generally belong to nonlinear, matrix manifolds, a variety of interpolation techniques have been proposed that are capable of enforcing the constraints characterizing those manifolds. Regression-based non-intrusive methods have been proposed that, among others, use artificial neural networks (ANNs), in particular multi-layer perceptrons [32], Gaussian process regression (GPR) [31], and radial basis function (RBF) [7] to perform the interpolation.
Here, we will explore an alternative approach to propagating latent-space dynamics based on Neural ODEs, which are a family of continuous-depth, differentiable networks that can be seen as an extension of ResNets in the limit of a zero discretization step size [20]. Details of our approach follow below. In addition, we consider two NIROM techniques - a) based on linear dimension reduction via POD and latent space evolution via Radial Basis Functions (RBF), and b) Dynamic Mode Decomposition (DMD) that will serve as the benchmarks in our numerical experiments to provide comparisons with the Neural ODE approach. We then proceed with several numerical experiments based on incompressible flow around a cylinder and shallow water hydrodynamics in order to evaluate the methods’ performance for fast replay applications in complex fluid-dynamics problems.
Methodology
The standard ROM development framework usually consists of three stages:
identification of a low-dimensional latent (or reduced-order) space,
determining a latent-space representation of the nonlinear dynamical system in terms of the reduced basis and modeling the evolution of the system of modal coefficients, and
reconstruction in the high-fidelity space for validation and analysis.
Machine learning techniques can be introduced at any of these stages. For example, many works have explored the use of deep learning-based approaches like autoencoders as a way to introduce a nonlinear alternative for dimension reduction [45, 29]. Combining these methods with data-driven latent-space propagation (for example via fully connected or recurrent neural networks) leads to a fully non-intrusive approach [30]. On the other hand, one can also combine nonlinear dimension reduction with intrusive projection to create a hybrid method [41, 36]. In this work, we study three different data-driven strategies for accurate learning of system dynamics within the context of linear dimension reduction. In the first two methods we adopt the POD technique for identification of an optimal global basis. For the latent space evolution, we utilize a kernel-based multivariate interpolation method called radial basis function (RBF) interpolation, and a machine learning strategy designed for sequential learning of time-series data called neural ordinary differential equations (NODE). In the third strategy, the three stages of ROM development are combined together by using a classical model decomposition technique called the dynamic mode decomposition (DMD), that is supported by rigorous mathematical analysis of Koopman mode theory.
Proper orthogonal decomposition
POD is a popular technique for dimension reduction [6] of the solution manifold of a dynamical system by determining a linear reduced space spanned by an orthogonal basis with an associated energetic hierarchy. [60] provides an excellent overview of POD as well as a comparison with other dimension-reduction techniques.
Consider a snapshot matrix containing a collection of high-fidelity snapshots of the solution manifold from time to such that is the snapshot with the temporal mean value removed, i.e., where is the time-averaged solution. The goal of the POD procedure is to identify a linear subspace , () which approximates the solution manifold optimally with respect to the -norm.
The POD bases can be efficiently extracted by performing a “thin” singular value decomposition (SVD) of the snapshot matrix , where is a diagonal matrix containing the singular values arranged in decreasing order of magnitude, and is the rank of . and are and matrices respectively, whose columns are the orthonormal left and right singular vectors of such that . The column of the matrix are ordered corresponding to the singular values and these provide the desired POD basis. Let denote the matrix of the first columns of , be the matrix containing the first rows of , and be a diagonal matrix containing the first singular values from , then the high-fidelity solution at time can be approximated as,
where is a vector of modal coefficients with respect to the reduced basis. The modal coefficient matrix constitutes our training data for the latent space learning methods. Due to the Eckart-Young-Mirsky theorem, the POD basis provides an optimal rank- approximation of the snapshot matrix with a desired level of accuracy, .
The POD method has been successfully applied in statistics [34], signal analysis and pattern recognition [19], ocean models [62], air pollution models [23], convective Boussinesq flows [54], and Shallow Water Equation (SWE) models [58, 44].
Latent space evolution
In this section, we outline two non-intrusive methods for modeling the evolution of time-series data in the latent space defined by the POD basis. RBF interpolation is a classical, data-driven, kernel-based method for computing an approximate continuous response surface that aligns with the given multivariate data. The second technique called NODE is a neural-network based method to predict the continuous evolution of a vector over time, that is designed to preserve memory effects within the architecture.
Radial basis function interpolation For simplicity, let the time evolution of the modal coefficients be represented as a semi-discrete dynamical system,
where all the information about the temporal dynamics including the effects of any numerical stabilization of the high-fidelity solver and all the nonlinear terms are embedded in . In the POD-RBF NIROM framework [21], instead of the Galerkin projection, the components of the time derivative function () are approximated using RBF interpolation.
Let denote a RBF approximation of the time derivative function , which is defined by a linear combination of instances of a radial basis function ,
where denotes the set of interpolation centers and () is the unknown interpolation coefficient corresponding to the center for the component of the modal coefficient. These interpolation coefficients are computed by enforcing the interpolation function to exactly match the time derivative of the modal coefficients at test points (). Choosing the centers and the test points identically from the set of snapshot modal coefficients as such that , and making some simplifying assumptions leads to a symmetric, linear system of equations to solve for the unknown interpolation coefficients,
where
The coefficients define a unique RBF interpolant which can then be used to approximate eq. (2) and generate a non-intrusive model for the evolution of the modal coefficients . In this work, a first-order forward Euler scheme has been employed for the discretization of the time derivative, and a strictly positive-definite Matérn kernel, given by has been adopted, where is the Euclidean distance and is the RBF shape factor [24].
Adopting RBF interpolation for modeling the latent space evolution of the modal coefficients has been shown to be quite successful for nonlinear, time-dependent partial differential equations (PDEs) [67, 21], nonlinear, parametrized PDEs [7, 68], and aerodynamic shape optimization [33], to name a few.
Neural ordinary differential equations Recurrent neural network (RNN) architectures like LSTM and GRU are often employed to encode time-series data and forecast future states, as their internal memory preserving architecture allows them to incorporate state information over a sequence of input data. Although RNNs have seen great success in natural language processing tasks, they have had relatively limited success in high-fidelity scientific computing applications [25, 63], as it has been observed that a sequence generated by an RNN may fail to preserve temporal regularity of the underlying signal, and thus may not represent true continuous dynamics. [18]. With deep neural networks (DNN) such as ResNet, the evolution of the features over the network depth is equivalent to solving an ordinary differential equation (ODE) such as using the forward Euler method, and this connection between ResNet’s architecture and numerical integrators has been explored in details by [53] and others. Several other deep learning methods have been proposed for learning ODEs and PDEs. These include using PDE-based network [42], training DNNs using physics-informed soft penalty constraints [51], and using sparse regularizers and regression [13, 16], to name a few.
Chen et al. [18] proposed a ’continuous-depth’ neural network called ODE-Net that effectively replaces the layers in ResNet-like architectures with a trainable ODE solver. The memory efficiency and stability of this neural ordinary differential equation (NODE) approach was further improved in [28, 20] and others. [46] applied the NODE framework to obtain latent space closure models for ROMs of a one-dimensional advecting shock problem and a one-dimensional Burgers’ turbulence problem that exhibits multiscale behavior in the wavenumber space. Some other notable recent applications of NODE include the identification of ODE or PDE models from time-dependent data [59], modeling of irregularly spaced time series data (Rubanova, Chen, and Duvenaud 2019), modeling of spatio-temporal information in video signals [35]. Finlay et al. [26] used a combination of optimal transport theory and stability regularizations to propose a neural-ODE generative model that can be efficiently trained on large-scale datasets. Here we further explore the application of the POD-NODE methodology to complex, real-world flows characterized by systems of two-dimensional, nonlinear PDEs.
We assume that the time evolution of the modal coefficients of the high-fidelity dynamical system in the latent space can be modeled using a (first-order) ODE,
The goal is to obtain a NN approximation of the dynamics function such that . The full procedure can be outlined as follows:
Compute the time series of modal coefficients for where .
Initialize a NN approximation for the dynamics function where represents the initial NN parameters.
The NN parameters are optimized iteratively through the following steps. (a) Compute the approximate forward time trajectory of the modal coefficients by solving eq. (5) using a standard ODE integrator as,
(b) The free parameters of the NODE model are . Evaluate the differentiable loss function . (c) To optimise the loss, compute gradients with respect to the free parameters. Similar to the usual backpropagation algorithm, this can be achieved by first computing the gradient , and then a performing a reverse traversal through the intermediate states of the ODE integrator. For a memory-efficient implementation, the adjoint method [18] can be used to backpropagate the errors by solving an adjoint system for the augmented state vector backwards in time from to . (d) The gradient computed in the previous step is used to update the parameters by using an optimization algorithm like RMSProp or Adam.
The trained NODE approximation of the dynamics function can be used to compute predictions for the time trajectory of the modal coefficients.
In this work, we utilize the TEDiffEq (https://github.com/tilu1994/tfdiffeq) library that runs on the Tensorflow Eager Execution platform to train the NODE models. Although a single layer architecture guarantees upper-bounds according to the universal approximation theorem [8], deeper networks with up to four layers as well as several linear and nonlinear activation functions are also explored due to their possibly improved expressibility for more complex nonlinear dynamics [69]. RMSProp is adopted for loss minimization with an initial learning rate of 0.001, a staircase decay function with a range of variable decay schedules, and a momentum coefficient of 0.9. NODE predictions of comparable accuracy were obtained for all the numerical experiments by using both the adjoint method as well as by backpropagating gradients directly through the hidden steps of the ode solver. However, for large-scale training data the latter method may lead to memory issues, especially while computing on GPU nodes.
Dynamic mode decomposition
As a final point of comparison, we consider Dynamic mode decomposition (DMD). DMD is a data-driven ROM technique that represents the temporal dynamics of a complex, nonlinear system [56, 38] as the combination of a few linearly evolving, spatially coherent modes that oscillate at a fixed frequency, and which are closely related to the eigenvectors of the infinite-dimensional Koopman operator [37, 47]. Consider the following snapshot matrices containing a few temporally-equiespaced snapshots of a high-dimensional dynamical system:
where is the solution snapshot, is the spatial degrees of freedom of the discretized system, and is the total number of temporal snapshots. DMD involves the identification of the best-fit linear operator that relates the above matrices as , and computing its eigenvalues and eigenvectors. Computing a least-square approximation of using the Moore-Penrose pseudoinverse(†) may pose computational challenges due to the size of the discrete dynamical system. For computational efficiency, the exact DMD algorithm (adopted here) avoids computing the Moore-Penrose pseudoinverse(†) by projecting the operator on to a reduced space obtained by POD, as outlined in [2].
In recent years, Koopman mode theory has provided a rigorous theoretical background for an efficient modal decomposition in problems describing oscillations and other nonlinear dynamics using DMD [52]. Several variants of the DMD algorithm have been proposed [49, 39, 1, 40] and have been successfully applied as efficient ROM techniques for determining the optimal global basis modes for nonlinear, time-dependent problems [11, 12]. For non-parametrized PDEs, DMD presents an efficient framework that combines all the three stages of ROM development to learn a linear operator in an optimal least square sense. However, this approach cannot be directly applied to parametrized problems Alsayyari et al. [3].
Numerical experiments
In this section, we first assess the performance of different NODE architectures for a benchmark flow problem characterized by the incompressible Navier Stokes equations (NSE), and then further evaluate the relative performance of all three NIROM models for two real-world applications governed by the shallow water equations (SWE). The POD-RBF and DMD NIROM training runs were performed on a Macbook Pro 2018 with a 2.9 GHz 6-Core Intel Core i9 processor and 32 GB 2400 MHz DDR4 RAM. The NODE models were trained in serial on Vulcanite, a high performance computer at the U.S. Army Engineer Research and Development Center DoD Supercomputing Resource Center (ERDC-DSRC). Vulcanite is equipped with NVIDIA Tesla V100 PCIe GPU accelerator nodes and has 32GBytes memory/node.
Flow around a cylinder
This problem simulates a time-periodic fluid flow through a 2D pipe with a circular obstacle. The flow domain is a rectangular pipe with a circular hole of radius , denoted by . The flow is governed by
where denotes the velocity, the pressure, is the outer product (dyadic product) given by , and is the kinematic viscosity. No slip boundary conditions are specified along the lower and upper walls, and on the boundary of the circular obstacle. A parabolic inflow velocity profile is prescribed on the left wall,
and zero gradient outflow boundary conditions on the right wall. High-fidelity simulation data is obtained with OpenFOAM using an unstructured mesh with 14605 nodes at , such that the flow exhibits the periodic shedding of von Karman vortices. 313 training snapshots are collected for seconds with seconds, and the NIROM predictions are obtained for seconds with seconds.
A large collection of NODE architectures and hyperparameter configurations were trained for 50000 epochs and details of the best 8 models are presented in Table 1. A fourth-order Runge-Kutta solver was found to be the optimal choice in terms of both accuracy and efficiency among all the available solvers ranging from the fixed-step forward Euler and the midpoint solvers to the adaptive-step Dormand-Prince (dopri5) solver. The “tanh” and “elu” activation functions were found to be the most effective among all the available linear and nonlinear activation functions. Due to the nature of the activation functions, the networks
| Id | Layers | Units | Act. | LR decay steps, rate | Scaling | Augmented | MSE | Training |
| Range | 1–4 | 32–512 | linear, relu, elu, tanh, ... | 5000–25000, 0.1–0.9 | ||||
| NODE1 | 1 | 256 | elu | 10000, 0.3 | No | No | 8.87e-4 | 24.45 hrs |
| NODE2 | 1 | 256 | tanh | 5000, 0.7 | Yes | No | 9.01e-4 | 24.56 hrs |
| NODE3 | 1 | 512 | elu | 5000, 0.5 | No | No | 8.86e-4 | 24.39 hrs |
| NODE4 | 1 | 256 | tanh | 10000, 0.25 | Yes | Yes | 9.02e-4 | 22.97 hrs |
| NODE5 | 4 | 64 | tanh | 5000, 0.5 | Yes | No | 9.19e-4 | 27.98 hrs |
| NODE6 | 1 | 256 | elu | 10000, 0.1 | No | No | 8.87e-4 | 24.13 hrs |
| NODE7 | 2 | 128 | elu | 5000, 0.5 | No | No | 8.86e-4 | 25.80 hrs |
| NODE8 | 1 | 512 | tanh | 5000, 0.5 | Yes | Yes | 9.00e-4 | 24.77 hrs |
Table 1. Best NODE architectures for the cylinder example. All models were trained for 50000 epochs using the fourth-order Runge-Kutta solver and the RMSProp optimizer with an initial learning rate of and a momentum of 0.9.
with “tanh” activations were found to train better when every element of the input state vector was individually scaled to be bounded in , while networks with “elu” activations trained better without scaling of input vectors. Augmentation of input states as outlined in Dupont, Doucet, and Teh [20] was found to have no significant impact on the training. The RMSProp optimizer paired with either a step decay function or an exponential decay function were found to be equally effective. However, further numerical experiments are necessary to study the efficiency of alternative first-order and second-order optimization methods. The number of decay steps were varied in discrete increments between 5000 to 25000, and decay rates ranging from 0.1 to 0.9 were studied. It was observed that a lower initial learning rate () combined with either larger decay steps and smaller decay rates or vice versa led to a desirable training trajectory. Figure 1 shows the evolution of the , , and latent-space modal coefficients for the pressure and the x-velocity solutions, obtained using the best 8 NODE models. All the models generate accurate predictions at a finer temporal resolution than the training data, and have excellent agreement with the high-fidelity solution even while extrapolating outside the training data ( seconds).

Figure 1. Comparison of NODE models in the latent space for the cylinder example
Figure 2 compares the time trajectory of the spatial root mean square errors (RMSE) in the high-fidelity space for two of the best NODE models with two DMD NIROM solutions obtained using truncation levels of and . It is encouraging to note that even though the NODE solutions are computed using a latent-space representation that is roughly comparable to the DMD solution with a smaller truncation level (), they are superior in accuracy to the coarsely truncated DMD solutions. Furthermore, unlike the POD-RBF solution that is trained with a first-order Euler time discretization, the NODE solutions did not exhibit any significant loss in accuracy with time, even while predicting outside the training region. It is, however, important to note that the training time for any new NODE architecture was extremely high (see Table 1) when compared to generating a POD-RBF or a DMD NIROM model, which usually required less than a minute in most cases. Such long training times may pose a significant challenge for exhaustive explorations of the design space for optimal architectures and hyperparameters, and may hinder the adoption of existing packages for automated architecture search. Thus, a concerted effort needs to be directed towards acceleration of NODE training times and towards constraining the design space by a priori identification of promising architectures.
Figure 2. Comparison of RMSE of best NODE models with DMD for the cylinder example
Shallow water equations
The next two numerical examples involve flows governed by the depth-averaged SWE which is written in a conservative residual formulation as
where the state variable consists of the flow depth, , and the discharges in the and directions, given by and , respectively. Further details about the flux vectors , and the high-fidelity model equations are available in [21]. The high-fidelity numerical solutions of the SWE are obtained using the 2D depth-averaged module of the Adaptive Hydraulics (AdH) finite element suite, which is a U.S. Army Corps of Engineers (USACE) high-fidelity, finite element resource for 2D and 3D dynamics [61].
Tidal flow in San Diego bay This numerical example involves the simulation of tide-driven flow in the San Diego Bay in California, USA. The AdH high-fidelity model consists of nodes,
Bay in California, USA. The AdH high-fidelity model consists of nodes, uses tidal data obtained from NOAA/NOS Co-Ops website at a tailwater elevation inflow boundary and has no flow boundary conditions everywhere else. Further details are available in [21].
The training space is generated using 1801 high-fidelity snapshots obtained between minutes to hours at a time interval of seconds. The predicted ROM solutions are computed for the same time interval with seconds. A latent space of dimension 265 is generated by using a POD truncation tolerance of for each solution component. The RBF NIROM approximation is computed using a shape factor, . The simulation time points provided as input to the NODE model are normalized to lie in . The ‘dopri5’ ODE solver is adapted for computing the hidden states both forward and backward in time. Learning from the conclusions of the cylinder example, a network consisting of a single hidden layer with 256 neurons is deployed and the RMSProp optimizer with an initial learning rate of 0.001, a staircase decay rate of 0.5 every 5000 epochs, and a momentum of 0.9 is utilized for training the model over 20000 epochs. For the DMD NIROM, the simulation time points are normalized to a unit time step, and a truncation level of is used to compute the DMD eigen-spectrum.
Figure 3 shows the NIROM solutions (top row) for at hours and the corresponding error plots.

Figure 3. NIROM solutions of and errors at hours for the San Diego example
Figure 4 shows the spatial RMSE over time for the depth (left) and the x-velocity (right) NIROM solutions. The NODE NIROM solution has comparable accuracy to the DMD NIROM solution and unlike the RBF NIROM solution, does not exhibit any appreciable accumulation of error over time.
Figure 4. NIROM RMSEs for the San Diego example
Riverine flow in Red River The final numerical example involves an application of the 2D SWE to simulate riverine flow in a section of the Red River in Louisiana, USA. The AdH high-fidelity model uses nodes, has a natural inflow velocity condition upstream, a tailwater elevation boundary downstream, and no flow boundary along the river bank. For further details see [21].

Figure 5. NIROM solutions of and errors at hours for the Red River example
Figure 6. NIROM RMSEs for the Red River example
The training space is generated by using 1081 high-fidelity snapshots obtained between minutes to hours at a time interval of seconds. The predicted ROM solutions are computed for the same time interval with seconds. A latent space spanned by 54 modes is generated by using a POD truncation tolerance of for each solution component. The RBF NIROM approximation is computed using a shape factor, . For consistency, the NODE network architecture is kept identical to the San Diego example and the training is also performed for 20000 epochs. Also, similar to the previous example, the simulation time points for DMD input are normalized to an unit time step. However, a smaller truncation level of is used to compute the DMD eigen-spectrum.
Figure 5 shows the NIROM solutions (top row) for at hours and the corresponding error plots.
Figure 6 shows the spatial RMSE over time of the depth (left) and the x-velocity (right) NIROM solutions for the Red River example. It can be seen that the DMD NIROM solution has a relatively higher RMSE owing to the lower truncation level chosen for this example, while the RBF NIROM is far more accurate. The NODE NIROM solution seems to match the performance of the RBF NIROM solution. This indicates that the NODE NIROM framework is successful with two distinct real-world flow regimes and holds promise for more widespread applicability to model the evolution of latent space dynamics.
Conclusion
We have studied Neural ODEs as a non-intrusive machine-learning algorithm to model the evolution of modal coefficients of a system of nonlinear, time-dependent PDEs in the linearly embedded latent space characterized by a truncated POD basis. Numerical experiments were carried out with a benchmark periodic flow problem governed by the incompressible Navier Stokes equations and two real-world applications of estuarine and riverine flow dynamics governed by the two-dimensional shallow water equations.The NODE formulation demonstrated a stable and accurate learning trajectory in modeling reduced basis dynamics, even in comparison to two classical ROM techniques utilizing dynamic mode decomposition and radial basis function interpolation. The DMD NIROM exhibited superior accuracy in most of the examples and was found to be most promising for long-term predictions. However, the POD-RBF NIROM technique is easily applicable to parametrized model reduction scenarios involving parametric training manifolds of very high dimension, whereas the DMD algorithm does not have a natural extension to such a setting. The POD-NODE formulation also produced extremely promising extrapolatory predictions for the flow around a cylinder example. This presents an exciting prospect for future exploration as even for an isolated system, unperturbed by unseen external forcings, truly extrapolative predictions of reduced order dynamics in flow regimes that do not correspond to the training data is a rare feature for most well-established ROM frameworks. This study leads to several promising avenues of research.
To begin with, an exhaustive search for an optimal NODE network architecture and optimal model hyperparameters needs to be conducted for a wide range of flow dynamics in order to gain insight of the learning trajectory and to design more generalizable NODE NIROM formulations with faster training times. With the goal of long-term predictive formulations in mind, embedding uncertainty estimates in the NODE NIROM framework might facilitate the development of adaptive models capable of re-assessing learning trajectories through in-situ measurements. The construction of a set of response functions for modeling the prediction error using machine learning [27] or Gaussian Process Regression (GPR) [65] are some recent works in this direction. Another exciting field of study would be to combine the NODE framework with machine-learning strategies for the generation of nonlinear manifolds [41, 36] that are suitable for an efficient reduced representation of the system dynamics for advection-dominated problems and in the presence of sharp gradients where a truncated linear subspace offers a poor solution representation. All the relevant data and codes for this study will be made available in a public repository at https://github.com/erdc/node_nirom upon publication.
Acknowledgements
This research was supported in part by an appointment of the first author to the Postgraduate Research Participation Program at the U.S. Army Engineer Research and Development Center, Coastal and Hydraulics Laboratory (ERDC-CHL) administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the U.S. Department of Energy and ERDC. The authors would also like to thank Dr. Gaurav Savant for his valuable help in using the Adaptive Hydraulics suite (AdH) [61] for the high-fidelity numerical simulation of the 2D shallow water flow examples. Permission was granted by the Chief of Engineers to publish this information.
References
- [1]Alekseev, A.; Bistrian, D.; Bondarev, A.; and Navon, I. 2016. On linear and nonlinear aspects of dynamic mode decomposition. Int. J. Numer. Methods Fluids 82(6): 348–371. doi:10.1002/fld.4221.DOI
- [2]Alla, A.; and Kutz, J. N. 2017. Nonlinear model order reduction via dynamic mode decomposition. SIAM J. Sci. Comput. 39(5): B778–B796. doi:10.1137/16M1059308.DOI
- [3]Alsayyari, F.; Perkó, Z.; Tiberga, M.; Kloosterman, J. L.; and Lathouwers, D. 2021. A fully adaptive nonintrusive reduced-order modeling approach for parametrized time-dependent problems. Comput. Methods Appl. Mech. Eng. 373: 114843. doi:10.1016/j.cma.2020.113483. URL https://doi.org/10.1016/j.cma.2020.113483.
- [4]Amsallem, D.; and Farhat, C. 2012. Stabilization of projection-based reduced order models. Int. J. Numer. Methods Eng. 91: 358–377. doi:10.1002/nme.4274.DOI
- [5]Amsallem, D.; Zahr, M.; Choi, Y.; and Farhat, C. 2015. Design optimization using hyper-reduced-order models. Struct. Multi. Opt. 51(4): 919–940. doi:10.1007/s00158-014-1183-y.DOI
- [6]Antoulas, A.; and Sorensen, D. 2001. Approximation of Large-Scale Dynamical Systems: An overview. Int. J. Appl. Math. Comput. Sci. 11(5): 1093–1121.
- [7]Audouze, C.; De Vuyst, F.; and Nair, P. 2013. Nonintrusive Reduced-Order Modeling of Parametrized Time-Dependent Partial Differential Equations. Numer. Methods Partial Differ. Equation 29(5): 1587–1628. doi:10.1002/num.21768.DOI
- [8]Barron, A. R. 1993. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inf. Theory 39(3): 930–945. doi:10.1109/18.256500.DOI
- [9]Benner, P.; Gugercin, S.; and Willcox, K. 2015. A Survey of Projection-Based Model Reduction Methods for Parametric Dynamical Systems. SIAM Rev. 57(4): 483–531.DOI
- [10]Berkooz, G.; Holmes, P.; and Lumley, J. 1993. The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows. Annu. Rev. Fluid Mech. 25(1): 539–575. doi:10.1146/annurev.fl.25.010193.002543.DOI
- [11]Bistrian, D.; and Navon, I. 2015. An improved algorithm for the shallow water equations model reduction: Dynamic Mode Decomposition vs POD. Int. J. Numer. Methods Fluids.DOI
- [12]Bistrian, D.; and Navon, I. 2017. Randomized dynamic mode decomposition for nonintrusive reduced order modelling. Int. J. Numer. Methods Eng. 112: 3–25. doi:10.1002/nme.5499.DOI
- [13]Brunton, S. L.; Proctor, J. L.; Kutz, J. N.; and Bialek, W. 2016. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. U. S. A. 113(15): 3932–3937. ISSN 10916490. doi:10.1073/pnas.1517384113.DOI
- [14]Carlberg, K.; Bou-Mosleh, C.; and Farhat, C. 2011. Efficient nonlinear model reduction via a least-squares Petrov-Galerkin projection and compressive tensor approximations. Int. J. Numer. Methods Eng. 86: 155–181. doi:10.1002/nme.3050.DOI
- [15]Carlberg, K.; Farhat, C.; Cortial, J.; and Amsallem, D. 2013. The GNAT method for nonlinear model reduction: Effective implementation and application to computational fluid dynamics and turbulent flows. J. Comput. Phys. 242: 623–647. doi:10.1016/j.jcp.2013.02.028.DOI
- [16]Champion, K.; Lusch, B.; Nathan Kutz, J.; and Brunton, S. L. 2019. Data-driven discovery of coordinates and governing equations. Proc. Natl. Acad. Sci. U. S. A. 116(45): 22445–22451. ISSN 10916490. doi:10.1073/pnas.1906995116.DOI
- [17]Chaturantabut, S.; and Sorensen, D. 2010. Nonlinear model reduction via Discrete Empirical Interpolation. SIAM J. Sci. Comput. 32(5): 2737–2764. doi:10.1137/090746498.DOI
- [18]Chen, R. T. Q.; Rubanova, Y.; Bettencourt, J.; and Duvenaud, D. 2018. Neural Ordinary Differential Equations. doi:10.1007/978-3-662-55774-7_3.
- [19]Deheuvels, P.; and Martynov, G. 2008. A Karhunen-Loeve decomposition of a Gaussian process generated by independent pairs of exponential random variables. J. Func. Anal. 255(9): 2363–2394. doi:10.1016/j.jfa.2008.07.021.DOI
- [20]Dupont, E.; Doucet, A.; and Teh, Y. W. 2019. Augmented Neural ODEs. URL http://arxiv.org/abs/1904.01681.
- [21]Dutta, S.; Farthing, M. W.; Perracchione, E.; Savant, G.; and Putti, M. 2020. A greedy non-intrusive reduced order model for shallow water equations. URL https://arxiv.org/abs/2002.11329.
- [22]Fang, F.; Pain, C.; Navon, I.; ElSheikh, A.; Du, J.; and Xiao, D. 2013. Non-linear Petrov-Galerkin methods for reduced order hyperbolic equations and discontinuous finite element methods. J. Comput. Phys. 234: 540–559. doi:10.1016/j.jcp.2012.10.011.DOI
- [23]Fang, F.; Zhang, T.; Pavlidis, D.; Pain, C.; Buchanan, A.; and Navon, I. 2014. Reduced order modelling of an unstructured mesh air pollution model and application in 2D/3D urban street canyons. Atmos. Env. 96: 96–106.DOI
- [24]Fasshauer, G. 2007. Meshfree Approximation Methods with MATLAB, volume 6 of Interdisciplinary Mathematical Sciences. World Scientific Publishing Company.DOI
- [25]Ferrandis, J. d. A.; Triantafyllou, M.; Chryssostomidis, C.; and Karniadakis, G. 2019. Learning functionals via LSTM neural networks for predicting vessel dynamics in extreme sea states. URL https://arxiv.org/abs/1912.13382.
- [26]Finlay, C.; Jacobsen, J.-H.; Nurbekyan, L.; and Oberman, A. M. 2020. How to train your neural ODE: the world of Jacobian and kinetic regularization. URL http://arxiv.org/abs/2002.02798.
- [27]Freno, B. A.; and Carlberg, K. T. 2019. Machine-learning error models for approximate solutions to parameterized systems of nonlinear equations. Comput. Methods Appl. Mech. Eng. 348: 250–296. doi:10.1016/j.cma.2019.01.024. URL https://doi.org/10.1016/j.cma.2019.01.024.
- [28]Gholami, A.; Keutzer, K.; and Birós, G. 2019. ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs. URL https://arxiv.org/abs/1902.10298.
- [29]Ghorbanidehno, H.; Lee, J.; Farthing, M.; Hesser, T.; Darve, E. F.; and Kitanidis, P. K. 2021. Deep learning technique for fast inference of large-scale riverine bathymetry. Advances in Water Resources 147: 103715. ISSN 0309-1708. doi:https://doi.org/10.1016/j.advwatres.2020.103715. URL https://www.sciencedirect.com/science/article/pii/S0309170819309418.
- [30]Gonzalez, F. J.; and Balajewicz, M. 2018. Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems.
- [31]Guo, M.; and Hesthaven, J. 2019. Data-driven reduced order modeling for time-dependent problems. Comput. Methods Appl. Mech. Eng. 345: 75–99. doi:10.1016/j.cma.2018.10.029.
- [32]Hesthaven, J.; and Ubbiali, S. 2018. Non-intrusive reduced order modeling of nonlinear problems using neural networks. J. Comput. Phys. 363: 55–78. doi:10.1016/j.jcp.2018.02.037.
- [33]Iuliano, E.; and Quagliarella, D. 2013. Aerodynamic shape optimization via non-intrusive POD-based surrogate modelling. In 2013 IEEE Congr. Evol. Comput. CEC 2013, 1467–1474. IEEE. doi:10.1109/CEC.2013.6557736.
- [34]Jolliffe, I. 1986. Principal Component Analysis. Springer New York, USA.
- [35]Kanaa, D.; Voleti, V.; Kahou, S.; and Pal, C. 2019. Simple Video Generation using Neural ODEs. In Annu. Conf. Neural Inf. Process. Syst. 2019, NeurIPS 2019. Vancouver, BC, Canada. URL https://voletiv.github.io/docs/publications/2019e_NeurIPSW_EncODEdec.pdf.
- [36]Kim, Y.; Choi, Y.; Widemann, D.; and Zohdi, T. 2020. A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder. URL http://arxiv.org/abs/2009.11990.
- [37]Koopman, B. O. 1931. Hamiltonian Systems and Transformation in Hilbert Space. Proc. Natl. Acad. Sci. U. S. A. 17(5): 315–8.
- [38]Kutz, J. N.; Brunton, S. L.; Brunton, B. W.; and Proctor, J. L. 2016. Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems. Philadelphia, PA: Society for Industrial and Applied Mathematics, third edition. ISBN 978-1-61197-449-2. doi:10.1137/1.9781611974508.DOI
- [39]Kutz, J. N.; Fu, X.; and Brunton, S. L. 2016. Multiresolution dynamic mode decomposition. SIAM J. Appl. Dyn. Syst. 15(2): 713–735. doi:10.1137/15M1023543.DOI
- [40]Le Clainche, S.; and Vega, J. M. 2017. Higher order dynamic mode decomposition. SIAM J. Appl. Dyn. Syst. 16(2): 882–925. doi:10.1137/15M1054924.DOI
- [41]Lee, K.; and Carlberg, K. T. 2020. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. J. Comput. Phys. 404: 108973. doi:10.1016/j.jcp.2019.108973. URL https://doi.org/10.1016/j.jcp.2019.108973.
- [42]Long, Z.; Lu, Y.; and Dong, B. 2019. PDE-Net 2.0: Learning PDEs from data with a numeric-symbolic hybrid deep network. J. Comput. Phys. 399: 108925. doi:10.1016/j.jcp.2019.108925. URL https://doi.org/10.1016/j.jcp.2019.108925.
- [43]Lozovskiy, A.; Farthing, M.; and Kees, C. 2017. Evaluation of Galerkin and Petrov-Galerkin model reduction for finite element approximations of the shallow water equations. Comput. Methods Appli. Mech. Eng. 318: 537–571. doi:10.1016/j.cma.2017.01.027.
- [44]Lozovskiy, A.; Farthing, M.; Kees, C.; and Gildin, E. 2016. POD-based Model Reduction for Stabilized Finite Element Approximations of Shallow Water Flows. J. Comput. Appl. Math. 302: 50–70.
- [45]Lusch, B.; Nathan Kutz, J.; and Brunton, S. L. 2018. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Comm. 9(1). doi:https://doi.org/10.1038/s41467-018-07210-0.
- [46]Maulik, R.; Mohan, A.; Lusch, B.; Madireddy, S.; Balaprakash, P.; and Livescu, D. 2020. Time-series learning of latent-space dynamics for reduced-order model closure. Phys. D Nonlinear Phenom. 405: 132368. doi:10.1016/j.physd.2020.132368. URL https://doi.org/10.1016/j.physd.2020.132368.
- [47]Mezić, I. 2013. Analysis of Fluid Flows via Spectral Properties of the Koopman Operator. Ann. Rev. Flu. Mech. 45(1): 357–378. doi:10.1146/annurev-fluid-011212-140652.
- [48]Peherstorfer, B.; Willcox, K.; and Gunzburger, M. A. X. 2016. Optimal model management for multifidelity Monte Carlo estimation. SIAM J. Sci. Comput. 38(5): A3163–A3194. doi:10.1137/15M1046472.
- [49]Proctor, J. L.; Brunton, S. L.; and Kutz, J. N. 2016. Dynamic mode decomposition with control. SIAM J. Appl. Dyn. Syst. 15(1): 142–161. doi:10.1137/15M1013857.DOI
- [50]Quarteroni, A.; Manzoni, A.; and Negri, F. 2016. Reduced Basis Methods for Partial Differential Equations. Springer, Cham. doi:10.1007/978-3-319-15431-2.
- [51]Raissi, M.; Perdikaris, P.; and Karniadakis, G. E. 2019. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378: 686–707. ISSN 10902716. doi:10.1016/j.jcp.2018.10.045. URL https://doi.org/10.1016/j.jcp.2018.10.045.
- [52]Rowley, C.; Mezi, I.; Bagheri, S.; Schlatter, P.; and Henningson, D. 2009. Spectral analysis of nonlinear flows. J. Fluid Mech. 641(Rowley 2005): 115–127. doi:10.1017/S0022112009992059.DOI
- [53]Ruthotto, L.; and Haber, E. 2019. Deep Neural Networks Motivated by Partial Differential Equations. J. Math. Imaging Vis. 2–10. doi:10.1007/s10851-019-00903-1.DOI
- [54]San, O.; and Borggaard, J. 2015. Principal interval decomposition framework for POD reduced-order modeling of convective Boussinesq flow. Int. J. Numer. Methods Fluids 78(1): 37–62.DOI
- [55]Sapsis, T. P.; and Majda, A. J. 2013. Statistically accurate low-order models for uncertainty quantification in turbulent dynamical systems. Proc. Natl. Acad. Sci. U. S. A. 110(34): 13705–13710. doi:10.1073/pnas.1313065110.DOI
- [56]Schmid, P. 2010. Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656(July 2010): 5–28. doi:10.1109/PIERS-FALL.2017.8293532.
- [57]Sirovich, L. 1987. Turbulence and the dynamics of coherent structures. Part I:Coherent structures. Quart. Appl. Math. 45: 561–571.
- [58]Stefanescu, R.; Sandu, A.; and Navon, I. 2014. Comparison of POD reduced order strategies for the nonlinear 2D shallow water equations. Int. J. Numer. Methods Fluids 76(8): 497–521.DOI
- [59]Sun, Y.; Zhang, L.; and Schaeffer, H. 2020. NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data. Proc. Mach. Learn. Res. 107(2016): 352–372.arxiv.org/abs/1908.03190
- [60]Taira, K.; Hemati, M. S.; Brunton, S. L.; Sun, Y.; Duraisamy, K.; Bagheri, S.; Dawson, S. T.; and Yeh, C. A. 2020. Modal analysis of fluid flows: Applications and outlook. AIAA J. 58(3): 998–1022. doi:10.2514/1.J058462.DOI
- [61]Trahan, C.; Savant, G.; Berger, R.; Farthing, M.; McAlpin, T.; Petty, L.; Choudhary, G.; and Dawson, C. 2018. Formulation and application of the adaptive hydraulics three-dimensional shallow water and transport models. J. Comput. Phys. 374: 47–90. doi:10.1016/j.jcp.2018.04.055.DOI
- [62]Vermeulen, P.; and Heemink, A. 2006. Model-reduced variational data assimilation. Monthly Weather Review 134: 2888–2899.DOI
- [63]Wang, Q.; Ripamonti, N.; and Hesthaven, J. S. 2020. Recurrent neural network closure of parametric POD-Galerkin reduced-order models based on the Mori–Zwanzig formalism. J. Comput. Phys. 410: 102402. doi:10.1016/j.jcp.2020.109402. URL https://doi.org/10.1016/j.jcp.2020.109402.
- [64]Willcox, K. 2006. Unsteady flow sensing and estimation via the gappy Proper Orthogonal Decomposition. Comput. Fluids 35(2): 208–226.
- [65]Xiao, D. 2019. Error estimation of the parametric non-intrusive reduced order model using machine learning. Comput. Methods Appl. Mech. Eng. 355: 513–534. doi:10.1016/j.cma.2019.06.018. URL https://doi.org/10.1016/j.cma.2019.06.018.
- [66]Xiao, D.; Fang, F.; Buchan, A.; Pain, C.; Navon, I.; Du, J.; and Hu, G. 2014. Non-linear model reduction for the Navier–Stokes equations using residual DEIM method. J. Comput. Phys. 263: 1–18. doi:10.1016/j.jcp.2014.01.011.DOI
- [67]Xiao, D.; Fang, F.; Pain, C.; and Hu, G. 2015. Non-intrusive reduced-order modelling of the Navier–Stokes equations based on RBF interpolation. Int. J. Numer. Methods Fluids 79: 580–595. doi:10.1002/fld.406.
- [68]Xiao, D.; Fang, F.; Pain, C.; and Navon, I. 2017. A parameterized non-intrusive reduced order model and error analysis for general time-dependent nonlinear partial differential equations and its applications. Comput. Methods Appl. Mech. Eng. 317: 868–889. doi:10.1016/j.cma.2016.12.033.DOI
- [69]Zhang, H.; Gao, X.; Unterman, J.; and Arodz, T. 2019. Approximation Capabilities of Neural ODEs and Invertible Residual Networks. CoRR abs/1907.12998. URL http://arxiv.org/abs/1907.12998.


