betaFeedback

WorksMathematics

Fast-Slow Neural Networks for Learning Singularly Perturbed Dynamical Systems

Abstract

Singularly perturbed dynamical systems play a crucial role in climate dynamics and plasma physics. A powerful and well-known tool to address these systems is the Fenichel normal form, which significantly simplifies fast dynamics near slow manifolds through a transformation. However, this normal form is difficult to realize in conventional numerical algorithms. In this work, we explore an alternative way of realizing it through structure-preserving machine learning. Specifically, a fast-slow neural network (FSNN) is proposed for learning data-driven models of singularly perturbed dynamical systems with dissipative fast timescale dynamics. Our method enforces the existence of a trainable, attracting invariant slow manifold as a hard constraint. Closed-form representation of the slow manifold enables efficient integration on the slow time scale and significantly improves prediction accuracy beyond the training data. We demonstrate the FSNN on examples including the Grad moment system, two-scale Lorenz96 equations, and Abraham-Lorentz dynamics modeling radiation reaction of electrons.

Introduction

In this article, a new architecture called the Fast-Slow Neural Network (FSNN) is developed for learning data-driven models of singularly perturbed dynamical systems with dissipative fast timescale dynamics. The constraint is imposed by way of a novel neural network architecture with structured weight matrices. By leveraging the neural ODE framework from [1], our technique produces deterministic evolution equations that approximately reproduce trajectory training data. When the training data comes from an unknown dynamical system, the method achieves data-driven full-order model discovery with stability guarantees due to provable existence of the slow manifold. In short, our architecture is, by design, a singularly perturbed dynamical system which contains an attractor.

Various dynamical systems encountered in nature exhibit a timescale separation such that the fast timescale is dissipative. Prominent examples include the kinetic dynamics of strongly-collisional gasses and plasmas, and radiative transfer in optically-thick media. For ordinary differential equations (ODEs) of this type, Fenichel theory [2] reveals that fast timescale dissipation implies existence of an invariant slow manifold that attracts nearby trajectories and governs the long-term dynamics. For partial differential equations (PDEs), or more

Funding: This work was partially supported by the U.S. Department of Energy Advanced Scientific Computing Research (ASCR) under DOE-FOA-2493 “Data-intensive scientific machine learning”. It was also partially supported by the ASCR program of Mathematical Multifaceted Integrated Capability Center (MMICC). Alvarez Loya is supported by the U.S. National Science Foundation under NSF-DMS-2213261. generally integro-differential equations, Fenichel theory generally only applies formally, but its predictions are often valid anyway. For PDEs, inertial manifold theory [3] has been studied and shown to be useful in applications such as model reductions of fluid dynamics.

Nonlinear dimensionality reduction inspired by inertial manifold theory has a rich history (see [4] and the references therein) originating in the 1980s and 1990s [3, 5, 6, 7, 8, 9, 10, 11]. Data-driven approaches for discovering inertial manifolds in dynamical systems and using separations of scales to evolve a lower-dimensional reduced order model first materialized in the 90s [12, 13, 14, 15, 16, 17, 18] and recently reemerged in popularity due to boom in machine-learning-based modeling [19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]. The work in this manuscript tackles a special class of systems with attracting invariant sets, namely fast-slow systems with dissipative fast timescale dynamics [32]. Unlike previous data-driven approaches, the FSNN, a special form of neural ODE [1], is a full-order model which, using neural network architectures with special properties, strictly enforces the existence of an attracting slow manifold. This method may be viewed as a major step along a pathway in developing data-driven model discovery techniques with embedded invariant manifolds. In future work, we plan to address the general case where explicit timescale separation is not used as a crutch.

The secondary goal of the proposed network is to discover an efficient and asymptotic-preserving reduced-order model. In the common methodology of physics-based model reduction, one first decides a latent space (e.g. lower-order moments of a distribution) and then looks for a closure using tools such as asymptotic analysis. Many complicated closures have been discovered through this approach. The classical examples in rarefied gases and plasma physics include Grad’s moment method [33], Braginskii closure [34] and Hammett-Perkins closure [35]. The challenge of this methodology is that an exact closure may not exist. The accuracy of the approximate closure is related to the dimension of the latent space. In contrast, Fenichel theory guarantees the existence of an exact closure in the neighborhood of the singular limit of the equations in the form of a diffeomorphic coordinate transformation. This transformation exposes an invariant manifold by flattening the fast variable space. Often in practice, as asymptotic expansion is the best analytical approximation to the exact closure. In our work, we leverage structured neural network architectures and optimization to learn an approximation to the exact closure that is hidden in the dynamics. The data-driven approach described in this work can discover such a transformation only using short-term trajectory data. Therefore, this approach may be useful when the underlying dynamical system is known but the targeting latent space or a formal closure are not known.

The design of the FSNN architecture is motivated by the first few theorems discovered by Fenichel [36]. The components of the architecture are constructed using structured weight matrices. The coordinate transformation into Fenichel normal form is based on an invertible coupling flow network [37] in which we introduced the additional structure-preserving property of bi-Lipschitz control through the use of bi-Lipschitz affine transformation (bLAT) layers. This network and its inverse can be evaluated using closed-form expressions with identical computational complexity. We note that “vanilla” neural networks are generally non-invertible [38, 39, 40]. We also introduce the Schur form network, which is a novel architecture based on the Schur decomposition for controlling the locations of eigenvalues in the complex plane, to parameterize the linear term in the dynamics. Additionally, a low-rank bilinear map network is included to increase the approximation power in the fast variable space. Our approach is then integrated using adaptive time stepping that treats the linear term implicitly, further leveraging back-substitution on the Schur form. Some of the structures introduced are related to the idea of trivializing manifold optimization through parameterizations [41]. Structure-preserving flow maps through machine learning are an active area of research, and include such as the symplectic neural networks for Hamiltonian dynamics [42, 43]. Previous studies on flow map learning for multi-scale dynamical systems include [44] and [45]. In particular, [45] simultaneously preserves symplecticity and adiabatic invariance in a multi-scale setting.

Before going into a detailed presentation of our new method, it is worth highlighting an important shortcoming it suffers from, and how that shortcoming might be mitigated in practice. The method assumes that the dimension of the attracting invariant manifold is known in advance. While this is not a limitation in the surrogate modeling context, where the (expensive to simulate) dynamical system is known in detail, it does present a moderate challenge in the model discovery context. As such, when performing data-driven model discovery using our method, it will be necessary to first determine the dimension of the slow manifold, either through physical reasoning, or through application of auxiliary data mining techniques such as diffusion maps [46], autoencoders [25], or other techniques from reduced order modeling [4]. For a review on various manifold learning techniques, we refer the reader to [47]. In many cases, including the examples considered in this manuscript, the form of the slow manifold and therefore its dimensionality are known in the singular limit ϵ=0\epsilon= 0 as a power series expansion of ϵ\epsilon. However, the exact form for ϵ>0\epsilon> 0 and the transformation into Fenichel normal form is unknown. In order to learn the slow manifold effectively, a sufficient number of trajectories for various choices of ϵ\epsilon and initial conditions are necessary. In the final example of the paper, we apply our network in the nontrivial setting of learning the slow manifold for the Abraham-Lorentz equations which, for trajectories starting off of the slow manifold, are unstable forward in time.

The remainder of the manuscript is organized as follows. A review of fast-slow dynamical systems is presented in § 2. The fast-slow neural network architecture is described in detail in § 3. The numerical implementation of the architecture is presented in § 4. Results from applying the architecture to various examples are shown in § 6. Concluding remarks are made in § 7.

Fast-Slow Dynamics and the Fenichel Normal Form

In this paper, we develop a novel machine learning (ML)-inspired architecture, the Fast-Slow Neural Network (FSNN). FSNNs are a parameterization for a class of dynamical systems exhibiting multiscale behavior in time that, to leading order, are dissipative on the fast time scale. Fast-slow systems, which exhibit both short-term and long-term behavior, are relevant to many physical systems observed in nature, including the climate and plasma systems. In this section, we motivate the FSNN architecture with a review of the theory of fast-slow systems.

Definition 2.1. A fast-slow system is an ODE of the form [32, 36].

ϵddτy=f(x,y,ϵ),(1)\epsilon\frac{d}{d\tau} y = f(x,y,\epsilon), \tag*{(1)}
ddτx=g(x,y,ϵ),(2)\frac{d}{d\tau} x = g(x,y,\epsilon), \tag*{(2)}

where x=x(τ)∈RNxx=x(\tau)\in\mathbb{R}^{N_x}, y=y(τ)∈RNyy=y(\tau)\in\mathbb{R}^{N_y}, ff and gg are smooth functions of their arguments, and Dyf(x,y,0)D_yf(x,y,0) is invertible for all (x,y)(x,y) where f(x,y,0)=0f(x,y,0)=0.

In the above definition, yy and xx are the fast and slow variables, respectively, over corresponding timescale variables tt and τ=ϵt\tau=\epsilon t.

In some instances, a dynamical system may exhibit a fast-slow split, but a representation for the fast and slow variables is unknown. Consider the following representation of a dynamical system for z(τ)∈RNz(\tau)\in\mathbb{R}^{N}, ϵ>0\epsilon>0,

ϵddτz=U(z,ϵ),(3)\epsilon\frac{d}{d\tau}z=U(z,\epsilon), \tag*{(3)}

where UU is a sufficiently smooth function of zz and ϵ\epsilon.

Definition 2.2. The dynamical system (3) exhibits a fast-slow split if there is an ϵ\epsilon-dependent invertible transformation of the dependent variables into fast-slow coordinates (x,y)(x,y), x∈RNxx\in\mathbb{R}^{N_x}, y∈RNyy\in\mathbb{R}^{N_y}, N=Nx+NyN=N_x+N_y, where the dynamics are described by a fast-slow system.

In this paper, we restrict our attention to the normally stable fast-slow system, which is a fast-slow system where the fast directions are stable.

Definition 2.3. A normally stable fast-slow system is a fast-slow system where the eigenvalues of the Jacobian, Dyf(x,y,0)∈RNy×NyD_yf(x,y,0)\in\mathbb{R}^{N_y\times N_y}, are strictly to the left of the imaginary axis when f(x,y,0)=0f(x,y,0)=0.

Due to the dissipative fast-scale dynamics of normally stable fast-slow systems, the solutions will be attracted to an invariant slow manifold in forward time. We will now define the notion of the slow manifold for normally stable fast-slow systems. Consider a fast-slow system in the form (2.1).

Lemma 2.4. For ϵ>0\epsilon>0 sufficiently small, there exists a function, y∗(x,ϵ)y^*(x,\epsilon), such that the graph

Mϵ={(x,y):y=y∗(x,ϵ)},(4)M_\epsilon=\{(x,y):y=y^*(x,\epsilon)\}, \tag*{(4)}

is locally invariant under (2.1). We define MϵM_\epsilon to be the slow manifold.

Here a bounded hypersurface is called a locally invariant manifold if the vector field defining the dynamical system is tangent to the hypersurface at all of its points [48]. This lemma is proved following Theorem 4 in [36]. The purpose of the FSNN is to be a universal approximator for normally stable fast-slow systems. The network is based on the following well known result due to Fenichel.

Lemma 2.5. When ϵ>0\epsilon>0 is sufficiently small and (x,y)(x,y) lies is in a neighborhood around the slow manifold MϵM_\epsilon, there exists a change of coordinates (x,y)→(x~,y~)(x,y)\to(\tilde{x},\tilde{y}) where the dynamics of normally stable fast-slow systems can be written as,

ddty~=A(x~,y~,ϵ)y~,(5)\frac{d}{dt}\tilde{y}=A(\tilde{x},\tilde{y},\epsilon)\tilde{y}, \tag*{(5)}
ddtx~=ϵg~(x~,y~,ϵ),(6)\frac{d}{dt}\tilde{x}=\epsilon\tilde{g}(\tilde{x},\tilde{y},\epsilon), \tag*{(6)}

where A(x~,y~,ϵ)∈RNy×NyA(\tilde{x},\tilde{y},\epsilon) \in\mathbb{R}^{N_y \times N_y}, g~(x~,y~,ϵ)∈RNx\tilde{g}(\tilde{x},\tilde{y},\epsilon) \in\mathbb{R}^{N_x}, and A(x~,0,0)A(\tilde{x},0,0) is a stable matrix, i.e., its eigenvalues are on the left of the imaginary axis.

The above lemma is proved in Chapter 3.2 of [36]. The lemma introduces (2.4), which is often called the Fenichel normal form due to the pioneering work of Neil Fenichel [2]. Note that the Fenichel normal form given in [36] is more general than (2.4), including fixed points that are either stable or unstable in the linearization of the fast variable. In this work, we only consider fast variables that are long-time stable. Moreover, due to the existence of the coordinate transformation, we consider the general form of the singularly perturbed systems (3). However, the advantage of the split form (2.1) is that the slow manifold dimension is known.

Our FSNN takes advantage of a modified formulation of Fenichel normal form which will be described in the following discussion. The following theorem forms the framework for the FSNN architecture.

Theorem 2.6. Suppose the dynamics of z(t;ϵ)∈RNz(t;\epsilon) \in\mathbb{R}^{N}, ϵ>0\epsilon> 0, admit a fast-slow split and can be described by normally stable fast-slow system. Then there exists a ϵ\epsilon-dependent invertible transformation

\begin{bmatrix} y \\ x \end{bmatrix} = h(z;\epsilon), \tag*{(7)}\end{bmatrix}

where x(t)∈RNxx(t) \in\mathbb{R}^{N_x}, y(t)∈RNyy(t) \in\mathbb{R}^{N_y}, are slow and fast variables, respectively, and for ∣y∣≤Δ|y| \le\Delta, where Δ>0\Delta> 0, the dynamics for (x,y)(x,y) can be described using

ddty=T(x)y+B(x,y,ϵ)(y,y)+ϵC(x,y,ϵ)y,(8)\frac{d}{dt}y = T(x)y + B(x,y,\epsilon)(y,y) + \epsilon C(x,y,\epsilon)y, \tag*{(8)}
ddtx=ϵg(x,y,ϵ).(9)\frac{d}{dt}x = \epsilon g(x,y,\epsilon). \tag*{(9)}

T(x)∈RNy×NyT(x) \in\mathbb{R}^{N_y \times N_y} is a block upper triangular matrix consisting of either R2×2\mathbb{R}^{2 \times2} or R\mathbb{R} blocks on the diagonal and the eigenvalues of these diagonal blocks are strictly to the left of the imaginary axis for all xx. B(x,y,ϵ):RNy×RNy→RNyB(x,y,\epsilon): \mathbb{R}^{N_y} \times\mathbb{R}^{N_y} \to\mathbb{R}^{N_y} is a bilinear map, C(x,y,ϵ)∈RNy×NyC(x,y,\epsilon) \in\mathbb{R}^{N_y \times N_y} is a matrix, and g(x,y,ϵ)∈RNxg(x,y,\epsilon) \in\mathbb{R}^{N_x}. TT, BB, CC, and gg are CrC^r for r≥1r \ge1.

In the following discussion, we will prove Theorem 2.6 by refining the Fenichel normal form in (2.4) and applying the matrix Schur decomposition. First, we modify (2.4) by performing a Taylor expansion of the term A(x~,y~,ϵ)y~A(\tilde{x},\tilde{y},\epsilon)\tilde{y} about y~=0\tilde{y} = 0 and ϵ=0\epsilon= 0. Assuming that AA is in CrC^r for r≥1r \ge1, we have

A(x~,y~,ϵ)y~=A(x~,0,0)y~+B(x~,y~,ϵ)(y~,y~)+ϵC(x~,y~,ϵ)y~,(10)A(\tilde{x},\tilde{y},\epsilon)\tilde{y} = A(\tilde{x},0,0)\tilde{y} + B(\tilde{x},\tilde{y},\epsilon)(\tilde{y},\tilde{y}) + \epsilon C(\tilde{x},\tilde{y},\epsilon)\tilde{y}, \tag*{(10)}

where B(x~,y~,ϵ):RNy×RNy→RNyB(\tilde{x},\tilde{y},\epsilon): \mathbb{R}^{N_y} \times\mathbb{R}^{N_y} \to\mathbb{R}^{N_y} is a bilinear map and C(x~,y~,ϵ)∈RNy×NyC(\tilde{x},\tilde{y},\epsilon) \in\mathbb{R}^{N_y \times N_y} are remainder terms. Next, the Schur decomposition of A(x~,0,0)A(\tilde{x},0,0) will be used to define a new set of fast coordinates.

Definition 2.7. The Schur form of a matrix, A∈RM×MA \in\mathbb{R}^{M \times M}, is a matrix T∈RM×MT \in\mathbb{R}^{M \times M} that forms the Schur decomposition A=QTQTA = Q T Q^{T}, where Q∈RM×MQ \in\mathbb{R}^{M \times M} is orthogonal (QTQ=IQ^{T}Q = I) and TT is block upper triangular where the diagonal blocks consist of R\mathbb{R} and R2×2\mathbb{R}^{2 \times2} blocks.

The existence of the Schur decomposition is proven in many references, including [49, 50], though for any given matrix, the decomposition is not unique. However, the eigenvalues of AA are given by the eigenvalues of the diagonal blocks of TT. Let the Schur decomposition of A(x~,0,0)A(\tilde{x},0,0) be given by

A(x~,0,0)=Q(x~)T(x~)Q(x~)T,(11)A(\tilde{x},0,0)=Q(\tilde{x})T(\tilde{x})Q(\tilde{x})^{\mathrm{T}}, \tag*{(11)}

where Q(x~)∈RNy×NyQ(\tilde{x}) \in\mathbb{R}^{N_y \times N_y} is an orthogonal matrix and T(x~)∈RNy×NyT(\tilde{x}) \in\mathbb{R}^{N_y \times N_y} is the Schur form of A(x~,0,0)A(\tilde{x},0,0). Due to our assumptions on A(x~,0,0)A(\tilde{x},0,0), all of the eigenvalues of T(x~)T(\tilde{x}) are to the left of the imaginary axis. Consider the coordinate transformation y^=Q(x~)Ty~\hat{y}=Q(\tilde{x})^{\mathrm{T}}\tilde{y}. We form the relevant ODE for y^\hat{y} to use in place of the ODE for y~\tilde{y}.

ddty^=Q(x~)Tddty~+(∇Q(x~)T)(ddtx~,y~)=Q(x~)TQ(x~)T(x~)Q(x~)Ty~+Q(x~)TB(x~,Q(x~)y^,ϵ)(Q(x~)y^,Q(x~)y^)+ϵQ(x~)TC(x~,ϵ)Q(x~)y^+ϵ(∇Q(x~)T)(gϵ(x~,y^),y^)=T(x~)y^+Q(x~)TB(x~,Q(x~)y^,ϵ)(Q(x~)y^,Q(x~)y^)+ϵQ(x~)TC(x~,ϵ)Q(x~)y^+ϵ(∇Q(x~)T)(gϵ(x~,y^),y^).\begin{aligned} \frac{d}{dt}\hat{y}={}&Q(\tilde{x})^{\mathrm{T}}\frac{d}{dt}\tilde{y}+\left(\nabla Q(\tilde{x})^{\mathrm{T}}\right)\left(\frac{d}{dt}\tilde{x},\tilde{y}\right)\\ ={}&Q(\tilde{x})^{\mathrm{T}}Q(\tilde{x})T(\tilde{x})Q(\tilde{x})^{\mathrm{T}}\tilde{y}+Q(\tilde{x})^{\mathrm{T}}B(\tilde{x},Q(\tilde{x})\hat{y},\epsilon)(Q(\tilde{x})\hat{y},Q(\tilde{x})\hat{y})\\ &+\epsilon Q(\tilde{x})^{\mathrm{T}}C(\tilde{x},\epsilon)Q(\tilde{x})\hat{y}+\epsilon\left(\nabla Q(\tilde{x})^{\mathrm{T}}\right)(g_{\epsilon}(\tilde{x},\hat{y}),\hat{y})\\ ={}&T(\tilde{x})\hat{y}+Q(\tilde{x})^{\mathrm{T}}B(\tilde{x},Q(\tilde{x})\hat{y},\epsilon)(Q(\tilde{x})\hat{y},Q(\tilde{x})\hat{y})\\ &+\epsilon Q(\tilde{x})^{\mathrm{T}}C(\tilde{x},\epsilon)Q(\tilde{x})\hat{y}+\epsilon\left(\nabla Q(\tilde{x})^{\mathrm{T}}\right)(g_{\epsilon}(\tilde{x},\hat{y}),\hat{y}). \end{aligned}

By redefining terms, it can be shown that the ODE is equivalent to

ddty^=T(x~)y^+B~(x~,y^,ϵ)(y^,y^)+ϵC~(x~,y^,ϵ)y^,\frac{d}{dt}\hat{y}=T(\tilde{x})\hat{y}+\tilde{B}(\tilde{x},\hat{y},\epsilon)(\hat{y},\hat{y})+\epsilon\tilde{C}(\tilde{x},\hat{y},\epsilon)\hat{y},

where B~(x~,y^,ϵ)\tilde{B}(\tilde{x},\hat{y},\epsilon) is a bilinear map and C~(x~,y^,ϵ)\tilde{C}(\tilde{x},\hat{y},\epsilon) is a linear map. This establishes an equivalence to (2.6).

Theorem 2.6 takes advantage of many useful properties. In the fast-slow coordinate system of Theorem 2.6, the definition of the slow-manifold simplifies significantly.

Theorem 2.8. The graph, M={(x,y):x∈RNx,y=0}M=\{(x,y):x\in\mathbb{R}^{N_x},y=0\} is locally invariant under (2.6).

This theorem can be easily verified by substituting y=0y=0 into (8) and observing that ddty=0\frac{d}{dt}y=0 on the slow manifold. Trajectories on the slow manifold will remain on the slow manifold for all time. In this case, the dynamics can be reduced to an ODE for the slow variable,

ddτx=g(x,0,ϵ),(x,y)∈M,(12)\frac{d}{d\tau}x=g(x,0,\epsilon),\qquad(x,y)\in M, \tag*{(12)}

where τ\tau is the slow timescale. In the case when the coordinates (x,y)(x,y) are close to, but not on MM, the following theorem states that trajectories emanating from (x,y)(x,y) will approach MM exponentially in forward time.

Theorem 2.9. If ϵ>0\epsilon>0 is sufficiently small, then, for ∣y∣≤Δ|y|\leq\Delta, where Δ>0\Delta>0, there are κ>0\kappa>0 and α<0\alpha<0 such that

d((x(t),y(t)),M)≤κexp⁡(αt),(13)d((x(t),y(t)),M)\leq\kappa\exp(\alpha t), \tag*{(13)}

where dd represents the Euclidean distance.

This theorem is proved following Theorem 5 in [36].

Theorem 2.6 provides a framework for the FSNN architecture, which expresses each of the functions hh, TT, BB, CC, and gg using neural networks with specially designed architectures described in the following section. There are a few considerations that went into using the modified Fenichel normal form. By expanding A(x~,y,ϵ~)A(\tilde{x},y,\tilde{\epsilon}) in (10), we expose three separate terms with known structures. We further use the Schur decomposition to explicitly expose the eigenvalues of A(x~,0,0)A(\tilde{x},0,0), which are required to be left of the imaginary axis. Finally, we further simplify the linear term by absorbing the matrix Q(x~)Q(\tilde{x}) into the representation. Though we use the Schur decomposition to represent A(x~,0,0)A(\tilde{x},0,0) in this paper, other decompositions for representing matrices with eigenvalues to the left of the imaginary axis have been studied in literature [50, 51].

Fast-Slow Neural Networks (FSNNs)

In this section, we present a neural network-based approach to approximating the set of solutions to normally stable fast-slow systems using the framework of Theorem 2.6. We refer to this approach as the fast-slow neural network (FSNN), which is defined below and summarized in Figure 1.

Figure 1. Schematic of the fast-slow neural network (FSNN). The original coordinates are projected into fast-slow coordinates using the invertible coupling flow network. In these coordinates, numerical integration of the Fenichel normal form (2.6) is used to advance the solution to later time steps. The inverse function of the invertible coupling flow network is used to convert the trajectories in the fast-slow coordinates into the original coordinate system.

Definition 3.1. A fast slow neural network (FSNN) is a special neural ODE that represents a pullback vector field on a phase space Z∋zZ \ni z of the form V(z)=(Dh(z))−1v(h(z))V(z)=(Dh(z))^{-1}v(h(z)), where hh is a parametric diffeomorphism and vv is a parametric vector field in the modified Fenichel normal form (2.6). The quantities hh, TT, CC, BB, gg are parameterized as follows. The diffeomorphism hh is represented as an invertible coupling flow network; TT is represented using a negative Schur form network; BB is represented using a low-rank bilinear map network; and CC, gg are represented using multi-layer perceptrons.

Note that the corresponding flow maps FΔtF_{\Delta t} for VV and fΔtf_{\Delta t} for vv are related according to FΔt=h−1∘fΔt∘hF_{\Delta t}=h^{-1}\circ f_{\Delta t}\circ h. It follows that computing the flow map FΔtF_{\Delta t} from a FSNN reduces to first computing the flow map fΔtf_{\Delta t} for vv and then appropriately applying hh or h−1h^{-1}. The key components of Definition 3.1, including the invertible coupling flow network, Schur form network, and low-rank bilinear map network, will be defined in the following sections. First we will begin by stating the main result of the paper.

Theorem 3.2. The FSNN is a universal approximator for the set of solutions of normally stable fast-slow systems in the neighborhood of the slow manifold.

Due to Theorem 2.6, normally stable fast-slow systems can be represented using (2.5)–(2.6). Therefore, the FSNN inherits the universal approximation properties of its components, which we will later show are universal approximators for their corresponding class of functions. The invertible coupling flow network which represents hh is described in § 3.1 and was proven to be a universal approximator to invertible maps in [37]. The universal approximation theorems of the negative Schur form network representing TT and the bilinear map network representing BB are proved in § 3.2 and § 3.3, respectively. The universal approximation of nonlinear functions using feedforward neural networks for CC and gg are well known, e.g. [52]. Finally, we describe a consistent numerical scheme to obtain trajectories in time, which has an error bound that vanishes as the time steps Δtn=tn−tn−1→0\Delta t_n=t_n-t_{n-1}\to0.

Our ML-based approach naturally comes with a set of hyperparameters. The standard hyperparameters are those related to the architectures of hh, TT, CC and gg. The only systemic hyperparameters are the dimensions of the slow and fast subspaces, NxN_x and NyN_y, respectively. In the examples considered in § 6, the true dimensions of slow manifolds are known and used. In applications when the size of the slow manifold dimension is not known, tools like diffusion maps [46] or proper orthogonal decomposition (POD) [53] may be used to infer the size of the reduced dimension before applying the proposed FSNN.

The Invertible Coupling Flow Network

Now we will describe further details of the FSNN architecture. The first component is the invertible function, h(z;ϵ)h(z;\epsilon), which represents the transformation into fast-slow coordinates. The network is based on the invertible coupling flow network developed in [54], where the authors closely follow the architecture provided by [55]. This network is an example of an invertible neural network (INN). In particular, invertible coupling flow networks have been shown to be universal approximators to invertible functions in [37]. In this work, we modify the original network by introducing a regularization property which controls the Lipschitz constant of both the forward and inverse map. Such a regularization improves the robustness of the network during training. The constant is a tunable parameter of the network that, similar to the hidden dimensions of the network, can be made arbitrarily large to increase the expressiveness of the network. Therefore, this modification is compatible with the universal approximation result derived in [37]. As such, we introduce the key ingredient of Lipschitz control.

Definition 3.3. The bi-Lipschitz affine transformation (bLAT) layer is given by the transformation, f:RN→RMf:\mathbb{R}^{N}\to\mathbb{R}^{M}, defined by

f(x)=UΣVTx+b,(14)f(x)=U\Sigma V^{\mathsf{T}}x+b, \tag*{(14)}

where U∈RM×rU\in\mathbb{R}^{M\times r}, V∈RN×rV\in\mathbb{R}^{N\times r}, Σ∈Rr×r\Sigma\in\mathbb{R}^{r\times r}, b∈RMb\in\mathbb{R}^{M}, r=min⁡(M,N)r=\min(M,N), UU and VV are orthogonal

matrices satisfying UTU=VTV=IU^{\mathsf{T}}U=V^{\mathsf{T}}V=I, and

Σ=diag⁡[σ1 ⋯ σr],(15)\Sigma=\operatorname{diag}\left[\sigma_1\ \cdots\ \sigma_r\right], \tag*{(15)}

where 1L≤σi≤L\frac{1}{L}\leq\sigma_i\leq L for i=1,…,ri=1,\ldots,r and some L≥1L\geq1.

We remark that when M=NM=N, the mapping is invertible.

f−1(x)=VΣ−1UT(x−b).(16)f^{-1}(x)=V\Sigma^{-1}U^{\mathsf{T}}(x-b). \tag*{(16)}

Our new contribution is to include the matrix term UΣVTU\Sigma V^{\mathsf{T}}, which is written as a singular value decomposition (SVD), which exists for every matrix in RM×N\mathbb{R}^{M\times N}. One advantage of the bLAT layer is the following theorem.

Theorem 3.4. The forward and inverse bLAT layers are both LL-bi-Lipschitz.

This follows from the fact that 1L≤∥∇f(x)∥2≤L\frac{1}{L}\leq\lVert\nabla f(x)\rVert_2\leq L and 1L≤∥∇f−1(x)∥2≤L\frac{1}{L}\leq\lVert\nabla f^{-1}(x)\rVert_2\leq L. Another advantage of using the SVD is that, by construction, the evaluation of f(x)f(x) and f−1(x)f^{-1}(x) have an equivalent computational cost. In our implementation of the M=NM=N case, the orthogonal matrices of the bLAT layers are parameterized using the matrix exponential of a skew-symmetric matrix input, i.e. U=expm⁡(S)U=\operatorname{expm}(S), where S=−STS=-S^{\mathsf{T}}. When M≠NM\ne N, the orthogonal matrices are parameterized using the Householder factorization.

The second component of the invertible coupling flow network is the affine coupling flow layer. We adapt the definition provided in Section 2.1 of [37] and introduce a similar regularization idea to the above.

Definition 3.5. The affine coupling flow layer is defined by the transformation y=g(x)\mathbf{y}=g(\mathbf{x}), x,y∈RM\mathbf{x},\mathbf{y}\in\mathbb{R}^{M}, which is implicitly defined by

\begin{bmatrix} y_1 \\ y_2 \end{bmatrix} = \begin{bmatrix} \operatorname{diag}(F(x_2)) & 0 \\ 0 & \operatorname{diag}(G(y_1)) \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} + \begin{bmatrix} B(x_2) \\ C(y_1) \end{bmatrix}, \tag*{(17)}\end{bmatrix}

where x=[x1x2]Tx=\begin{bmatrix}x_1 & x_2\end{bmatrix}^{\mathsf{T}}, y=[y1y2]Ty=\begin{bmatrix}y_1 & y_2\end{bmatrix}^{\mathsf{T}}, x1,y1∈Rdx_1,y_1\in\mathbb{R}^{d}, x2,y2∈RM−dx_2,y_2\in\mathbb{R}^{M-d}, F,B:RM−d→RdF,B:\mathbb{R}^{M-d}\to\mathbb{R}^{d}, G,C:Rd→RM−dG,C:\mathbb{R}^{d}\to\mathbb{R}^{M-d}, 1≤d<M1\leq d<M and the range of the output elements of FF and GG is constrained to [1L,L]\left[\frac{1}{L},L\right] for some L≥1L\geq1. The inverse mapping, x=f−1(y)x=f^{-1}(y), is implicitly defined by

\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} \operatorname{diag}(F(x_2))^{-1} & 0 \\ 0 & \operatorname{diag}(G(y_1))^{-1} \end{bmatrix} \left( \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} - \begin{bmatrix} B(x_2) \\ C(y_1) \end{bmatrix} \right). \tag*{(18)}\end{bmatrix}

The functions F,G,B,CF,G,B,C are represented using feed-forward neural networks. In order to constrain outputs to the range [1L,L]\left[\frac{1}{L},L\right], we apply the map θL:R→[1L,L]\theta_L:\mathbb{R}\to\left[\frac{1}{L},L\right] defined by θL(s)=exp⁡(log⁡(L)tanh⁡(s))\theta_L(s)=\exp(\log(L)\tanh(s)). This constraint ensures invertibility of the affine coupling layer. The Lipschitz constant is a function of LL and ∇F,∇G,∇B,∇C\nabla F,\nabla G,\nabla B,\nabla C and can optionally be constrained using bLAT layers in the feedforward networks.

The invertible coupling flow network combines bLAT layers and affine coupling flow layers into a network.

Definition 3.6. Let f1,…,fK:RM→RMf_1,\ldots,f_K:\mathbb{R}^{M}\to\mathbb{R}^{M} define a set of bLAT layers and g1,…,gK:RM→RMg_1,\ldots,g_K:\mathbb{R}^{M}\to\mathbb{R}^{M} define a set of affine coupling flow layers. The invertible coupling flow network, h:RM→RMh:\mathbb{R}^{M}\to\mathbb{R}^{M} is defined as the composition

h=gK∘fK∘⋯∘g1∘f1(19)h=g_K\circ f_K\circ\cdots\circ g_1\circ f_1 \tag*{(19)}

and the inverse map is defined by

h−1=f1−1∘g1−1∘⋯∘fK−1∘gK−1.(20)h^{-1}=f_1^{-1}\circ g_1^{-1}\circ\cdots\circ f_K^{-1}\circ g_K^{-1}. \tag*{(20)}

The free parameter LL in each of the layers adjusts the complexity of the overall network and can be used as an effective regularizer. Despite the introduction of this regularization parameter, the network can be made arbitrarily expressive with the tuning of LL, and therefore the universal approximation result from [37] still holds true.

Theorem 3.7. The invertible coupling flow network is a universal approximator for invertible maps.

In Theorem 2.6, the invertible transformation is ϵ\epsilon-dependent. Therefore, we incorporate this dependence into the network by including ϵ\epsilon as a functional dependence in F,G,B,CF,G,B,C.

The Negative Schur Form Network

Next, a parameterization suitable for representing TT is discussed. We present the negative Schur form network, which is a differentiable parameterization for Schur forms with eigenvalues to the left of the imaginary axis. The Schur form network is composed of a feedforward neural network in which the outputs are re-formatted into a negative Schur form matrix, which is defined below.

Definition 3.8. The negative Schur form matrix is given by the block matrix T∈RNy×NyT\in\mathbb{R}^{N_y\times N_y}, where

T=[T11T12⋯T1pT22⋯T2p⋱⋮Tpp].(21)T=\begin{bmatrix} T_{11} & T_{12} & \cdots& T_{1p}\\ & T_{22} & \cdots& T_{2p}\\ & & \ddots& \vdots\\ & & & T_{pp} \end{bmatrix}. \tag*{(21)}

When MM is even, p=M/2p=M/2 and Tij∈R2×2T_{ij}\in\mathbb{R}^{2\times2} for i=1,…,pi=1,\ldots,p, j=i,…,pj=i,\ldots,p. When MM is odd, p=(M+1)/2p=(M+1)/2, Tij∈R2×2T_{ij}\in\mathbb{R}^{2\times2} for i=1,…,p−1i=1,\ldots,p-1, j=i,…,p−1j=i,\ldots,p-1, Tip∈R2T_{ip}\in\mathbb{R}^{2} for i=1,…,p−1i=1,\ldots,p-1, and Tpp∈(−∞,0)T_{pp}\in(-\infty,0). Each of the R2×2\mathbb{R}^{2\times2} blocks on the diagonal are further parameterized using the map

M(R,r,θ,ϕ)=R[cos⁡(θ)sin⁡(θ)−sin⁡(θ)cos⁡(θ)]+r[cos⁡(ϕ)sin⁡(ϕ)sin⁡(ϕ)−cos⁡(ϕ)],(22)\mathcal{M}(R,r,\theta,\phi)=R\begin{bmatrix} \cos(\theta) & \sin(\theta)\\ -\sin(\theta) & \cos(\theta) \end{bmatrix} +r\begin{bmatrix} \cos(\phi) & \sin(\phi)\\ \sin(\phi) & -\cos(\phi) \end{bmatrix}, \tag*{(22)}

where the region of parameters is restricted to

∣R∣≥∣r∣,R<0,θ∈(−π2,π2),ϕ∈R(23)\lvert R\rvert\ge\lvert r\rvert,\qquad R<0,\qquad\theta\in\left(-\frac{\pi}{2},\frac{\pi}{2}\right),\qquad\phi\in\mathbb{R} \tag*{(23)}

Due to this parameterization, we have the following theorem.

Theorem 3.9. The negative Schur form network is a parameterization for the set of Schur forms with eigenvalues to the left of the imaginary axis.

This can be proven by showing that (22)–(23) can smoothly represent both complex conjugate pairs and real pairs of eigenvalues with negative real part. Equation (22) represents the R2×2\mathbb{R}^{2\times2} matrix as a 4-parameter family sum of a symmetric and antisymmetric operator [56]. Let m=12Tr⁡(M)=Rcos⁡(θ)m = \frac{1}{2}\operatorname{Tr}(\mathcal{M}) = R\cos(\theta) and p=det⁡(M)=R2−r2p = \det(\mathcal{M}) = R^2-r^2. Then the eigenvalues of M\mathcal{M} are given by

λ±=m±m2−p.(24)\lambda_{\pm}=m\pm\sqrt{m^2-p}. \tag*{(24)}

The real parts of the eigenvalues are given by

Re⁡(λ±)=m±max⁡(m2−p,0).(25)\operatorname{Re}(\lambda_{\pm})=m\pm\sqrt{\max(m^2-p,0)}. \tag*{(25)}

Figure 2 shows the regions of the mm-pp plane where both eigenvalues are less than zero (green region), greater than zero (red region), complex (blue region), and real (outside of the blue region). In order for Re⁡(λ±)<0\operatorname{Re}(\lambda_{\pm})<0, we must have m<0m<0 and p>0p>0. In order to enforce these constraints, we enforce the restrictions in (23). We remark that for the first constraint, Rcos⁡(θ)<0R\cos(\theta)<0, RR and cos⁡(θ)\cos(\theta) must have opposite signs. If we focus on the region θ∈(−π/2,π/2)\theta\in(-\pi/2,\pi/2) where cos⁡(θ)>0\cos(\theta)>0, we would require R<0R<0. If the region of θ\theta were instead shifted by π\pi, then we would require R>0R>0. However, since M(R,r,θ+π,ϕ)=M(−R,r,θ,ϕ)\mathcal{M}(R,r,\theta+\pi,\phi)=\mathcal{M}(-R,r,\theta,\phi), these choices are equivalent.

Plot showing the eigenvalues encountered in various regions of the m-p plane

Figure 2. Plot showing the eigenvalues encountered in various regions of the mm-pp plane, where m=12Tr⁡(M)=Rcos⁡(θ)m = \frac{1}{2}\operatorname{Tr}(\mathcal{M}) = R\cos(\theta) and p=det⁡(M)=R2−r2p = \det(\mathcal{M}) = R^2-r^2. The green region corresponds to linear stability, Re⁡(λ+,λ−)<0\operatorname{Re}(\lambda_+,\lambda_-)<0, and the red region corresponds to linear instability, Re⁡(λ+,λ−)>0\operatorname{Re}(\lambda_+,\lambda_-)>0. Above the parabola p=m2p=m^2 (in the blue region), eigenvalues are complex. Below the parabola (outside of the blue region), the eigenvalues are real.

In the implementation of the network, each of the R2×2\mathbb{R}^{2\times2} blocks have four tunable parameters, Rˉ\bar{R}, rˉ\bar{r}, θˉ\bar{\theta}, ϕˉ∈R\bar{\phi}\in\mathbb{R}, that are mapped onto the restricted domain using

R=−∣rˉ∣e∣Rˉ∣,r=rˉ,θ=π2tanh⁡(θˉ),ϕ=ϕˉ.(26)R=-|\bar{r}|e^{|\bar{R}|},\qquad r=\bar{r},\qquad\theta=\frac{\pi}{2}\tanh(\bar{\theta}),\qquad\phi=\bar{\phi}. \tag*{(26)}

The remaining off-diagonal terms are unrestricted in parameter space. In order to achieve dependence with respect to xx, (i.e. T(x)T(x)), we introduce a feedforward network which takes $x as an input and outputs both the tunable parameters Rˉ\bar{R}, rˉ\bar{r}, θˉ\bar{\theta}, ϕˉ\bar{\phi} for each diagonal block and the elements of the remaining blocks. The resulting negative Schur form matrix is returned as the output of T(x)T(x).

The Low-Rank bilinear map Network

Finally, we introduce an approximation for the bilinear map term, BB.

Definition 3.10. The low-rank bilinear form network B:RNx×RNy→RNzB : \mathbb{R}^{N_x} \times\mathbb{R}^{N_y} \to\mathbb{R}^{N_z} is defined by

B(x,y)=∑r=1R(C(r)x)⊙(D(r)y),(27)B(x,y) = \sum_{r=1}^{R}(C^{(r)}x) \odot(D^{(r)}y), \tag*{(27)}

where x∈RNxx \in\mathbb{R}^{N_x}, y∈RNyy \in\mathbb{R}^{N_y}, C(r)∈RNz×NxC^{(r)} \in\mathbb{R}^{N_z \times N_x}, D(r)∈RNz×NyD^{(r)} \in\mathbb{R}^{N_z \times N_y}, ⊙\odot represents the Hadamard product (element-wise multiplication), and RR, 1≤R≤min⁡(Nx,Ny)1 \le R \le\min(N_x,N_y), represents the rank of approximation.

Here, RR is a tunable parameter. Increasing RR gives more expressive power at the expense of computational cost. This network has the following property:

Theorem 3.11. The low-rank bilinear map network is a universal approximator to the set of bilinear forms.

Consider the bilinear form, B~:RNx×RNy→RNz\widetilde{B} : \mathbb{R}^{N_x} \times\mathbb{R}^{N_y} \to\mathbb{R}^{N_z}. The tensor representation of B~\widetilde{B} is written as B~ijk\widetilde{B}_{ijk} where B~i∈RNx×Ny\widetilde{B}_i \in\mathbb{R}^{N_x \times N_y} for i=1,…,Nzi=1,\ldots,N_z. Let p=min⁡(Nx,Ny)p=\min(N_x,N_y), then we can use the singular value decomposition to represent the matrix B~i\widetilde{B}_i as

B~i=∑r=1pσi(r)ui(r)(vi(r))T,(28)\widetilde{B}_i = \sum_{r=1}^{p} \sigma_i^{(r)} u_i^{(r)} (v_i^{(r)})^{T}, \tag*{(28)}

where σi(r)\sigma_i^{(r)}, u(r)u^{(r)}, v(r)v^{(r)}, r=1,…,pr=1,\ldots,p, are the singular values, left singular vectors, and right singular vectors of B~i\widetilde{B}_i, respectively, and σ1≥⋯≥σp≥0\sigma_1 \ge\cdots\ge\sigma_p \ge0. The ithi^{\mathrm{th}} term of B~(x,y)\widetilde{B}(x,y) is given by

xTB~iy=∑r=1pσi(r)xTui(r)(vi(r))Ty=∑r=1pσi(r)(∑j=1Nxuij(r)xj)(∑k=1Nyvik(r)yk),(29)x^{T}\widetilde{B}_i y = \sum_{r=1}^{p} \sigma_i^{(r)} x^{T}u_i^{(r)}(v_i^{(r)})^{T}y = \sum_{r=1}^{p} \sigma_i^{(r)} \left(\sum_{j=1}^{N_x}u_{ij}^{(r)}x_j\right)\left(\sum_{k=1}^{N_y}v_{ik}^{(r)}y_k\right), \tag*{(29)}

We can find some C∈RNz×NxC \in\mathbb{R}^{N_z \times N_x} and D∈RNz×NyD \in\mathbb{R}^{N_z \times N_y} such that

xTB~iy=∑r=1p(∑j=1NxCij(r)xj)(∑k=1NyDik(r)yk).(30)x^{T}\widetilde{B}_i y = \sum_{r=1}^{p}\left(\sum_{j=1}^{N_x}C_{ij}^{(r)}x_j\right)\left(\sum_{k=1}^{N_y}D_{ik}^{(r)}y_k\right). \tag*{(30)}

This is equivalent to

B~(x,y)=∑r=1p(C(r)x)⊙(D(r)y).(31)\widetilde{B}(x,y) = \sum_{r=1}^{p}(C^{(r)}x) \odot(D^{(r)}y). \tag*{(31)}

The network uses a truncated version of this sum in equation (27) to approximate the action of the bilinear map B~\widetilde{B}. The approximation error of the low-rank bilinear map network is ∥Bi−B~i∥2=σi(R+1)\left\|B_i-\widetilde{B}_i\right\|_2=\sigma_i^{(R+1)}. This error can be decreased by increasing RR.

In our formulation, the operator BB has coefficients which are functions of xx, yy, and ϵ\epsilon. We inject this functional dependence by making the elements of C(r)C^{(r)} and D(r)D^{(r)} outputs of a feed-forward network with inputs (x,y,ϵ)(x,y,\epsilon).

Numerical Integration

The Schur form, TT will have eigenvalues that can be of arbitrarily large magnitude. Due to this, the dynamics of (2.6) will be stiff. Therefore, an Implicit-Explicit (IMEX) time integrator can be used to solve (2.6) efficiently. The IMEX scheme has been applied to a neural ODE framework in [57]. As shown below, an additional benefit of the proposed FSNN is to eliminate the need for iterative or direct linear solvers, thereby simplifying the training process.

Consider an IMEX Runge-Kutta method consisting of a diagonally implicit method for treating the stiff linear term and an explicit method for treating the nonlinear term. The method can be represented by the following Butcher tableau [58]

a11α21a21a22α21⋮⋮⋱⋮⋱as1⋯⋯assαs1⋯αs,s−1b1⋯⋯bsβ1⋯βs−1βs\begin{array}{cccc|cccc} a_{11} & & & & \alpha_{21} & & & \\ a_{21} & a_{22} & & & \alpha_{21} & & & \\ \vdots& \vdots& \ddots& & \vdots& \ddots& & \\ a_{s1} & \cdots& \cdots& a_{ss} & \alpha_{s1} & \cdots& \alpha_{s,s-1} & \\ \hline b_1 & \cdots& \cdots& b_s & \beta_1 & \cdots& \beta_{s-1} & \beta_s \end{array}

Let xn≈x(tn)x_n \approx x(t_n), yn≈y(tn)y_n \approx y(t_n), and Δt=tn−tn−1\Delta t=t_n-t_{n-1}. Let f(x,y,ϵ)=B(x,y,ϵ)(y,y)+ϵC(x,y,ϵ)yf(x,y,\epsilon)=B(x,y,\epsilon)(y,y)+\epsilon C(x,y,\epsilon)y. The intermediate stages are implicitly defined by

Xi=xn−1+ϵΔt∑j=1i−1αijg(Xj,Yj,ϵ),(32)X^i=x_{n-1}+\epsilon\Delta t\sum_{j=1}^{i-1}\alpha_{ij}g(X^j,Y^j,\epsilon), \tag*{(32)}
Yi=yn−1+Δt∑j=1iaijT(Xj)Yj+Δt∑j=1i−1αijf(Xj,Yj,ϵ)(33)Y^i=y_{n-1}+\Delta t\sum_{j=1}^{i}a_{ij}T(X^j)Y^j+\Delta t\sum_{j=1}^{i-1}\alpha_{ij}f(X^j,Y^j,\epsilon) \tag*{(33)}

for i=1,…,si=1,\ldots,s. In our notation, we take the above summation when i=1i=1 to be 0 (i.e., ∑j=10(⋅)=0\sum_{j=1}^{0}(\cdot)=0). Equation (33) can be rewritten to solve for YiY^i.

Yi=(I−ΔtaiiT(Xi))−1(yn−1+Δt∑j=1i−1(aijT(Xj)Yj+αijf(Xj,Yj,ϵ)))(34)Y^i=(I-\Delta t a_{ii}T(X^i))^{-1}\left(y_{n-1}+\Delta t\sum_{j=1}^{i-1}\left(a_{ij}T(X^j)Y^j+\alpha_{ij}f(X^j,Y^j,\epsilon)\right)\right) \tag*{(34)}

Note that the matrix given by I−ΔtaiiT(Xi)I-\Delta t a_{ii}T(X^i) is a block 2-by-2 upper triangular matrix. Due to the proposed architecture, this inverse can be handled efficiently using a backward substitution solve. Since TT additionally has functional dependence, we treat this in a vectorization-friendly matrix-free fashion, which avoids storing multiple matrices T(Xi)T(X^i) for each data pair in the batch set during training. The solution at the next time step is obtained through the update

xn=xn−1+ϵΔt∑i=1sβig(Xi,Yi,ϵ),(35)x_n=x_{n-1}+\epsilon\Delta t\sum_{i=1}^{s}\beta_i g(X^i,Y^i,\epsilon), \tag*{(35)}
yn=yn−1+Δt∑i=1s(biT(Xi)Yi+βif(Xi,Yi,ϵ)).(36)y_n=y_{n-1}+\Delta t\sum_{i=1}^{s}\left(b_i^{\mathsf{T}}(X^i)Y^i+\beta_i f(X^i,Y^i,\epsilon)\right). \tag*{(36)}

In our implementation, we use the two-stage second-order accurate IMEX strong-stability-preserving L-stable scheme defined in [58].

During the prediction of the learned system, one can use the above IMEX time integrator to integrate the full system, using small time steps to resolve the fast time scale. Alternatively, the proposed approach offers a unique capability to take larger time steps that are on the slow time scale when only the slow dynamics is of interest. When the solution is initially on the slow manifold (y=0y=0), the stages YiY^i for i=1,…,si=1,\ldots,s evaluate exactly to zero and therefore the numerical scheme can be simplified to an integration of the slow variables given by

xn=xn−1+ϵΔt∑i=1sβig(Xi,0,ϵ),(37)x_n=x_{n-1}+\epsilon\Delta t\sum_{i=1}^{s}\beta_i g(X^i,0,\epsilon), \tag*{(37)}
Xi=xn−1+ϵΔt∑j=1i−1αijg(Xj,0,ϵ),i=1,…,s.(38)X^i=x_{n-1}+\epsilon\Delta t\sum_{j=1}^{i-1}\alpha_{ij}g(X^j,0,\epsilon),\qquad i=1,\ldots,s. \tag*{(38)}

In this case, one can take larger time steps (e.g. Δt∼1ϵ\Delta t\sim\frac{1}{\epsilon}) and step over the fast dynamics formally. A hybrid scheme can also be used where the full system in (3.21) can be integrated until ∥y∥\lVert y\rVert is sufficiently small, after which slow integration in (3.22) is pursued using larger time steps. We note that some of the first works to use Runge-Kutta based residual networks to learn equations from time series was pursued in [59, 60].

Data-driven closure. A closure discovery procedure is implied through the above integration with large time steps. For a given full state vector z0z_0, the transformation hh first maps to the separate state of (x0,y0)(x_0,y_0). The state vector is then projected onto the closure of y=0y=0; the system for xx is thus closed and evolved using (3.22). Finally, the evolved state of xx is lifted back to the full state zz using h−1h^{-1}.

Numerical Implementation

In this section, we describe the numerical implementation of the fast-slow neural network and its corresponding training algorithm. The computation of the loss function and its gradient is summarized in Figure 3. The training set consists of both trajectory data, which describes the behavior of the dynamics, and manifold data, which describes the limiting behavior of the system. The trajectory dataset, D\mathcal{D}, contains time series, {(tn(k),zn(k))}n=1Nt\{(t_n^{(k)},z_n^{(k)})\}_{n=1}^{N_t}, and parameters ϵ(k)\epsilon^{(k)}, for k=1,…,Ndk=1,\ldots,N_d, where NdN_d is the number of trajectory data samples. The manifold dataset, M\mathcal{M}, contains multiple examples of the solution on the slow manifold, z(k)z^{(k)}, with corresponding parameters ϵ(k)\epsilon^{(k)}, for k=1,…,Nmk=1,\ldots,N_m. Examples of manifold data can be obtained either experimentally by integrating a model for a sufficiently long time with a small ϵ\epsilon or analytically if the equations permit an ϵ=0\epsilon=0 reduction

Figure 3. Schematic of training the fast-slow neural network (FSNN). The network is trained on dynamics data, D\mathcal{D}, which contains examples of trajectories on both the fast and slow timescales and manifold data, M\mathcal{M}, which contains examples of solution vectors that lie on the slow manifold. This data interacts both with the full network and the component involving the invertible neural network (INN). The overall loss is a combination of the dynamics loss, which includes L2L_2 losses in the original coordinates (Lsystem\mathcal{L}_{\mathrm{system}}), fast coordinates (Lfast\mathcal{L}_{\mathrm{fast}}), and slow coordinates (Lslow\mathcal{L}_{\mathrm{slow}}), and the manifold loss (Lmanifold\mathcal{L}_{\mathrm{manifold}}), which is an L2L_2 loss of the fast coordinate, which should map to y=0y=0 on the slow manifold.

Numerical Implementation

or asymptotic expansion. For many examples using the zeroth order expansion with ϵ=0\epsilon=0 is sufficient. Often obtaining a higher order expansion is nontrivial.

The loss function is given by

L=Lsystem+Lfast+Lslow+Lmanifold.(39)\mathcal{L}=\mathcal{L}_{\mathrm{system}}+\mathcal{L}_{\mathrm{fast}}+\mathcal{L}_{\mathrm{slow}}+\mathcal{L}_{\mathrm{manifold}}. \tag*{(39)}

The first three terms are evaluated using the trajectory data set, D\mathcal{D}, and the last term is evaluated using the manifold data set, M\mathcal{M}. For the dynamics-oriented loss terms, the fast-slow neural network is applied to initial conditions in D\mathcal{D} to obtain the model trajectories, {(tn(k),x~n(k),y~n(k),z~n(k))}\{(t_n^{(k)},\tilde{x}_n^{(k)},\tilde{y}_n^{(k)},\tilde{z}_n^{(k)})\}. Additionally, the invertible coupling flow network is applied to the trajectory in D\mathcal{D} to augment the data {(tn(k),xn(k),yn(k),zn(k))}\{(t_n^{(k)},x_n^{(k)},y_n^{(k)},z_n^{(k)})\}. Let LL be an operator representing the approximation to the time integral of squared error, given by

L(t,z,z~)=∑n=0Nt−112((zn+1−z~n+1)2+(zn−z~n)2)(tn+1−tn).(40)L(t,z,\tilde{z})=\sum_{n=0}^{N_t-1}\frac{1}{2}\left((z_{n+1}-\tilde{z}_{n+1})^2+(z_n-\tilde{z}_n)^2\right)(t_{n+1}-t_n). \tag*{(40)}

Then we choose

Lsystem=1Nd∑k=1NdL(t,z(k),z~(k)),(41)\mathcal{L}_{\mathrm{system}} = \frac{1}{N_d}\sum_{k=1}^{N_d} L(t,z^{(k)},\widetilde{z}^{(k)}), \tag*{(41)}
Lfast=1Nd∑k=1NdL(t,y(k),y~(k)),(42)\mathcal{L}_{\mathrm{fast}} = \frac{1}{N_d}\sum_{k=1}^{N_d} L(t,y^{(k)},\widetilde{y}^{(k)}), \tag*{(42)}
Lslow=1Nd∑k=1NdL(t,x(k),x~(k))ϵ(k).(43)\mathcal{L}_{\mathrm{slow}} = \frac{1}{N_d}\sum_{k=1}^{N_d} \frac{L(t,x^{(k)},\widetilde{x}^{(k)})}{\epsilon^{(k)}}. \tag*{(43)}

Including the three terms individually encourages the network to learn the dynamics for the original and fast-slow coordinates to an adequate level.

In practice, we split the trajectory dataset, D\mathcal{D}, into a fast-scale subset, with time steps chosen to smoothly capture the fast dynamics converging to the slow manifold, and a slow-scale subset, with larger time steps chosen to capture the variation of the slow variables after the dynamics sufficiently close to the slow manifold. The trajectories in fast-scale subset are treated using the full integration in (3.21) while the trajectories from the slow-scale subset are treated using the slow-manifold integration in (3.22).

For the manifold loss term, the invertible coupling flow network is applied to examples in M\mathcal{M} to obtain fast-slow coordinates (x(k),y(k))(x^{(k)},y^{(k)}). If the data is on the slow manifold, then y(k)y^{(k)} should be zero. We therefore include the following as a penalty term.

Lmanifold=1Nm∑k=1Nm(y(k))2(44)\mathcal{L}_{\mathrm{manifold}} = \frac{1}{N_m}\sum_{k=1}^{N_m}\left(y^{(k)}\right)^2 \tag*{(44)}

Our implementation of the fast-slow neural network uses Jax [61] as a backend. Autodifferentiation is used to compute gradients of the loss function with respect to the tunable parameters and the Adam optimizer from the Optax [62] package is used adjust the weights towards an locally optimal solution for the dataset.

Examples

In this section, we consider multiple example problems to demonstrate the applicability of the FSNN to various problems, including multi-scale problems arising in hydrodynamics and plasma physics. In training, we explored various combinations of meta parameters for each problem. We pursued a policy to train several models in parallel and choose the model with the best performance. The meta parameters for the best models are reported in each respective section. We remark that though increasing the number of network parameters increases the expressiveness of the model, it comes at a higher computational cost and increases the risk of overfitting.

Visualization of the Attracting Slow Manifold Property

A significant advantage of the FSNN is that, for any choice of model parameters, the network will represent a fast-slow dynamical system with an attracting slow manifold that, due to Theorem 2.6, will attract solutions for sufficiently small values of ϵ\epsilon. The slow manifold is represented by the graph, y=0y = 0. Here we visualize the property of the FSNN without performing any training from dynamical system data. Instead, we demonstrate the property by randomly generating the trainable weights of FSNNs for a singularly perturbed dynamical system with Ny=Nx=1N_y = N_x = 1. The goal is to show that any initial condition for a randomly generated FSNN will naturally converge to its slow manifold. Note that for such a simple system, its slow manifold is known analytically.

In Figure 4, network a, b, and c are representing different FSNNs with different weights and therefore different slow manifolds. Multiple initial conditions are drawn randomly from a standard normal distribution (left plot) and are integrated using each FSNN to a later time t=4t = 4 corresponding to ϵ=0\epsilon= 0. The trajectory points at t=4t = 4 (blue dots) and the resulting slow manifold (orange curves) are plotted at this final time, demonstrating convergence of solutions to the slow manifolds inherent to each network.

Convergence to Slow Manifold for Multiple Network Initializations

Figure 4. Demonstration of attracting slow manifold property of the FSNN. For three different random initializations of the FSNN, initial conditions are drawn randomly from a standard normal distribution and are evolved in time to a later time (t=4t = 4). The dynamics are shown to limit to the curve representing the slow manifold.

Learning a Simple Attracting Manifold

We demonstrate the ability of the FSNN to learn dynamics for a simple test problem where the dynamics can be described by the following system of ODEs for z1,z2∈Rz_1, z_2 \in\mathbb{R}.

dz1dt=−ϵ(sin⁡(z1)+z2)(45)\frac{d z_1}{d t} = -\epsilon(\sin(z_1) + z_2) \tag*{(45)}
dz2dt=λ(z1)(z2−θ(z1))(46)\frac{d z_2}{d t} = \lambda(z_1)(z_2 - \theta(z_1)) \tag*{(46)}

In this example, z1z_1 represents the slow variable and z2z_2 represents the fast variable. When ϵ=0\epsilon= 0, the slow manifold is given by θ(z1)\theta(z_1). In this example, we choose λ\lambda and θ\theta to be

λ(x)=−1−110cos⁡(2x),θ(x)=2tanh⁡(x).(47)\lambda(x) = -1 - \frac{1}{10}\cos(2x), \qquad\theta(x) = 2\tanh(x). \tag*{(47)}

The trajectory dataset, D\mathcal{D}, consists of 10310^3 fast-scale trajectories, corresponding to initial conditions for z1,z2z_1, z_2 randomly generated uniformly in [−1,1][-1, 1], ϵ\epsilon randomly generated loguniformly in [10−5,10−2][10^{-5}, 10^{-2}], and integrated forward 5 timesteps using an adaptive RK45 solver with an error tolerance of 10−810^{-8}. The manifold dataset, M\mathcal{M}, consists of 10310^3 examples of data pairs ((z1,z2),ϵ=0)((z_1,z_2),\epsilon=0) on the graph z2=θ(z1)z_2=\theta(z_1), where z1z_1 is randomly generated uniformly in [−1,1][-1,1].

A FSNN is trained on this dataset using the Adam optimizer from Optax. The INN consists of one outer layer, where the inner coupling flow layers consists of feedforward networks with one layer and a hidden dimension of 10. Each of the feedforward neural networks on the RHS of the neural ODE system consists of 10 layers and the bilinear map has a rank of 2.

Figure 5 shows a comparison of the trained model and the reference solution in phase space for various choices of ϵ\epsilon. As can be seen by the corresponding error plots, the model shows reasonable agreement to the reference solution. In all of the phase plots, it is evident that the trajectories eventually get trapped in a slow manifold. In the rightmost plot on the top row, a comparison of the learned eigenvalues with that of the ground truth, λ(z1)\lambda(z_1), is shown. Additionally in the rightmost plot on the bottom row, the error between the learned slow manifold and true slow manifold is shown. The results demonstrate that the FSNN is able to successfully recover details of the original operator.

Results of learning the dynamics of the simple attracting manifold example

Figure 5. Results of learning the dynamics of the simple attracting manifold example described by (5.1). Left three plots on top row: phase portraits comparing the ground truth (g.t.) reference solution (bold curves) and that of the trained model (dots) for ϵ=0,0.001,0.01\epsilon= 0, 0.001, 0.01. Left three plots on bottom row: corresponding errors in time for each trajectory in the above phase portraits. Rightmost plot on top row: comparison of the eigenvalue of the negative Schur form as a function of z1z_1 learned by the FSNN with the ground truth eigenvalue given by λ\lambda in (47). Rightmost plot on bottom row: error between the learned slow manifold and the ground truth slow manifold given by θ\theta in (47).

Linear Grad Moment System

In this example, we show that the FSNN can be applied towards learning hydrodynamic slow manifolds. We consider a linear Grad moment system [33, 63], defined by

∂tρ=−ϵ∇⋅u,(48)\partial_t \rho= -\epsilon\nabla\cdot\mathbf{u}, \tag*{(48)}
∂tu=−ϵ(∇ρ+∇T+∇⋅σ),(49)\partial_t \mathbf{u} = -\epsilon\left(\nabla\rho+ \nabla T + \nabla\cdot\boldsymbol{\sigma}\right), \tag*{(49)}
∂tT=−2ϵ3(∇⋅u+∇⋅q),(50)\partial_t T = -\frac{2\epsilon}{3}\left(\nabla\cdot\mathbf{u} + \nabla\cdot\mathbf{q}\right), \tag*{(50)}
∂tσ=−σ−ϵ(2∇u‾+45∇q‾),(51)\partial_t \boldsymbol{\sigma} = -\boldsymbol{\sigma} - \epsilon\left(2\overline{\nabla\mathbf{u}} + \frac{4}{5}\overline{\nabla\mathbf{q}}\right), \tag*{(51)}
∂tq=−23q−ϵ(52∇T+∇⋅σ),(52)\partial_t \mathbf{q} = -\frac{2}{3}\mathbf{q} - \epsilon\left(\frac{5}{2}\nabla T + \nabla\cdot\boldsymbol{\sigma}\right), \tag*{(52)}

where ρ(x,t)\rho(\mathbf{x},t) is the density, u(x,t)\mathbf{u}(\mathbf{x},t) is velocity, T(x,t)T(\mathbf{x},t) is temperature, σ(x,t)\boldsymbol{\sigma}(\mathbf{x},t) is the stress tensor, q(x,t)\mathbf{q}(\mathbf{x},t) is the heat flux, and x=(x,y,z)T∈R3\mathbf{x}=(x,y,z)^{T}\in\mathbb{R}^{3} is the spatial coordinate. The overline represents the symmetric, traceless part of a tensor, e.g.

a‾=12(a+aT)−13ITr⁡(a).(53)\overline{\mathbf{a}}=\frac{1}{2}\left(\mathbf{a}+\mathbf{a}^{T}\right)-\frac{1}{3}\mathbf{I}\operatorname{Tr}(\mathbf{a}). \tag*{(53)}

We consider solutions that are one-dimensional (∂y=∂z=0\partial_y=\partial_z=0) and 2π2\pi-periodic in space, e.g.

ρ(x,t)=∑k=−∞∞ρk(t)eikx.(54)\rho(\mathbf{x},t)=\sum_{k=-\infty}^{\infty}\rho_k(t)e^{ikx}. \tag*{(54)}

The Fourier coefficients are governed by the following system of ODEs.

ddt[ρkukTkσkqk]=[0−ikϵ000−ikϵ0−ikϵ−ikϵ00−23ikϵ00−23ikϵ0−43ikϵ0−1−815ikϵ00−52ikϵ−ikϵ−23][ρkukTkσkqk],k=0,±1,…(55)\frac{d}{dt} \begin{bmatrix} \rho_k \\ u_k \\ T_k \\ \sigma_k \\ q_k \end{bmatrix} = \begin{bmatrix} 0 & -ik\epsilon& 0 & 0 & 0 \\ -ik\epsilon& 0 & -ik\epsilon& -ik\epsilon& 0 \\ 0 & -\frac{2}{3}ik\epsilon& 0 & 0 & -\frac{2}{3}ik\epsilon\\ 0 & -\frac{4}{3}ik\epsilon& 0 & -1 & -\frac{8}{15}ik\epsilon\\ 0 & 0 & -\frac{5}{2}ik\epsilon& -ik\epsilon& -\frac{2}{3} \end{bmatrix} \begin{bmatrix} \rho_k \\ u_k \\ T_k \\ \sigma_k \\ q_k \end{bmatrix}, \quad k = 0, \pm1, \ldots \tag*{(55)}

For this example, we truncate the Fourier series to consider only the k=0,±1k = 0, \pm1 models. Initial conditions are randomly generated for each Fourier coefficient uniformly in the complex region [−1,1]×[−i,i][-1,1] \times[-i,i]. The solution is computed for 50 time steps both using Δtfast=14\Delta t_{\mathrm{fast}} = \frac{1}{4} and Δtslow=14\Delta t_{\mathrm{slow}} = \frac{1}{4}. The model is trained using only the information from the first time step and tested using the remaining of the trajectory. The slow-scale trajectories are offset Δtslow\Delta t_{\mathrm{slow}} from t=0t = 0 to ensure the solution is sufficiently close to the slow manifold. A FSNN is trained on this dataset using the Adam optimizer from Optax. The INN consists of one outer layer, where the inner coupling flow layers consists of feedforward networks with one layer and a hidden dimension of 40. The feedforward neural networks on the RHS of the neural ODE system each consist of 50 layers and the bilinear map has a rank of 1. Considering each coefficient and its real and imaginary parts, the dimension of the learned model is R30\mathbb{R}^{30}, with Nx=18N_x = 18 and Ny=12N_y = 12.

Figure 6 shows comparisons of the trained model and reference solutions on the fast time scale and slow time scale. The results demonstrate that reasonable accuracy can be achieved on both time scales. We observe that model for the slow dynamics is slightly off in phase which causes error to grow in time.

Comparison of fast scale and slow scale dynamics between the ground truth and trained model for the Grad moment system example

Figure 6. Comparison of fast scale (top) and slow scale (bottom) dynamics between the ground truth and trained model for the Grad moment system example.

Table 1 shows a comparison of the eigenvalues of the trained model with the analytical eigenvalues. Quantitative agreement is observed, which implies that the model was successful in mapping the slow manifold approximately to the true slow manifold.

Grad Eigenvalues
g.t.modelerrorg.t.modelerror
−1-1−0.9971-0.99712.9e−032.9\mathrm{e}{-03}−23-\frac{2}{3}−0.6659-0.66598.2e−048.2\mathrm{e}{-04}
−1-1−0.9969-0.99693.1e−033.1\mathrm{e}{-03}−23-\frac{2}{3}−0.6658-0.66588.3e−048.3\mathrm{e}{-04}
−1-1−0.9969-0.99693.1e−033.1\mathrm{e}{-03}−23-\frac{2}{3}−0.6658-0.66589.1e−049.1\mathrm{e}{-04}
−1-1−0.9968-0.99683.2e−033.2\mathrm{e}{-03}−23-\frac{2}{3}−0.6657-0.66579.3e−049.3\mathrm{e}{-04}
−1-1−0.9968-0.99683.2e−033.2\mathrm{e}{-03}−23-\frac{2}{3}−0.6657-0.66579.4e−049.4\mathrm{e}{-04}
−1-1−0.9968-0.99683.2e−033.2\mathrm{e}{-03}−23-\frac{2}{3}−0.6657-0.66579.4e−049.4\mathrm{e}{-04}

Table 1. Comparison of ground truth eigenvalues of the Jacobian of the fast equations with that of the negative Schur form of the trained model for the Grad Moment System problem.

It is critical to stress that the slow dynamics comparison in Figure 6 is done through integration on the slow manifold using (3.22). Specifically, the initial condition state vector z0z_0 is first transformed into the fast-slow coordinates (x0,y0)(x_0, y_0) and projected onto the learned slow manifold (x0,0)(x_0, 0) before integrating the slow variable xx on the slow time scale only and then transforming back into original coordinates using h−1h^{-1}. This approach is significantly more efficient than integrating the full system using a standard numerical integrator. The excellent accuracy of the predicted slow variable solution indicates a data-driven closure is learned through this procedure. The closure of (5.3) formally exists when ϵ\epsilon approaches 0. It is readily seen that the reduced system is a linearized Euler system. When ϵ=0\epsilon= 0, the critical manifold is simply the center subspace. When weak nonlinearity is added, there are center manifolds tangent to the center subspace. In general these center manifolds are not unique, but there is a unique smoothest center manifold known as the spectral submanifold, see [64, 65]. In future work, it would be interesting to explore whether our methods can be adapted to ensure the learned slow manifold agrees with the spectral submanifold.

Multi-Scale Lorenz96 Equations

In this section we consider the two-scale Lorenz96 equations, which was a model proposed in [66] as a test bed model for multi-scale behavior in atmospheric dynamics. Multiscale techniques for this system have been explored in many works, such as in [67]. The original system is formulated as

ddτXk=−Xk−1(Xk−2−Xk+1)−Xk+F−hcb∑j=1JYj,k,(56)\frac{d}{d\tau} X_k = -X_{k-1}(X_{k-2} - X_{k+1}) - X_k + F - \frac{hc}{b} \sum_{j=1}^{J} Y_{j,k}, \tag*{(56)}
ddτYj,k=−cbYj+1,k(Yj+2,k−Yj−1,k)−cYj,k+(hcb)Xk,(57)\frac{d}{d\tau} Y_{j,k} = -cbY_{j+1,k}(Y_{j+2,k} - Y_{j-1,k}) - cY_{j,k} + \left(\frac{hc}{b}\right)X_k, \tag*{(57)}

for j=1,…,Jj = 1, \ldots, J and k=1,…,Kk = 1, \ldots, K, where the solutions satisfy periodicity (Xk+K=Xk,Yj,k+K=Yj,k,Yj+J,k=Yj,k)(X_{k+K} = X_k, Y_{j,k+K} = Y_{j,k}, Y_{j+J,k} = Y_{j,k}). Let c=b=1/ϵc = b = 1/\epsilon, h=1h = 1, Y=ϵyY = \epsilon y, τ=ϵt\tau= \epsilon t, and F=0F = 0. The system becomes

ddtxk=−ϵxk−1(xk−2−xk+1)−ϵxk−ϵ2∑j=1Jyj,k.(58)\frac{d}{dt}x_k = -\epsilon x_{k-1}(x_{k-2} - x_{k+1}) - \epsilon x_k - \epsilon^2 \sum_{j=1}^{J} y_{j,k}. \tag*{(58)}
ddtyj,k=−yj+1,k(yj+2,k−yj−1,k)−yj,k+xk.(59)\frac{d}{dt}y_{j,k} = -y_{j+1,k}(y_{j+2,k} - y_{j-1,k}) - y_{j,k} + x_k. \tag*{(59)}

Since the transformation is singular at ϵ=0\epsilon= 0, we study the equations in the region 0<ϵ≪10 < \epsilon\ll1.

Since the FSNN is limited to representing normally stable fast-slow systems, we explore the region of applicability where (5.8) is a normally fast-slow system. The normally stable region of Lorenz96 is identified by the following theorem.

Theorem 5.1. The system of equations defined by (5.8) represent a normally stable fast-slow system when −12<xk<89-\frac{1}{2} < x_k < \frac{8}{9}, for k=1,…,Kk = 1, \ldots, K.

Using Definition 2.3, we need to show that the Jacobian of the right-hand-side of (59) has eigenvalues strictly to the left of the imaginary axis. Consider the following nonlinear equation f(y;χ)∈Rf(y;\chi) \in\mathbb{R} for y∈RJy \in\mathbb{R}^{J} and χ∈R\chi\in\mathbb{R}.

f(y;χ)=−yj+1(yj+2−yj−1)−yj+χ.(60)f(y;\chi) = -y_{j+1}(y_{j+2} - y_{j-1}) - y_j + \chi. \tag*{(60)}

The Jacobian of this equation, DyfD_yf is given by

(Dyf)jk=∂fj∂yk=−δjk−δj+1,k(yj+2−yj−1)−yj+1(δj+2,k−δj−1,k),(61)(D_yf)_{jk} = \frac{\partial f_j}{\partial y_k} = -\delta_{jk} - \delta_{j+1,k}(y_{j+2} - y_{j-1}) - y_{j+1}(\delta_{j+2,k} - \delta_{j-1,k}), \tag*{(61)}

where δjk\delta_{jk} represents the Kronecker delta. We are interested in the eigenvalues of DyD_y when f(y;χ)=0f(y;\chi) = 0. Note that one solution to f(y;χ)=0f(y;\chi) = 0 is given by yj=χy_j = \chi for j=1,…,Jj = 1, \ldots, J. At this fixed point, the Jacobian becomes

(Dyf)jk∣y=χ=−δjk−χ(δj+2,k−δj−1,k)(62)(D_yf)_{jk}\big|_{y=\chi} = -\delta_{jk} - \chi(\delta_{j+2,k} - \delta_{j-1,k}) \tag*{(62)}

This represents a circulant matrix, of which the eigenvalues are given by

λj=−1−χ(eij4π/J−e−ij2π/J),j=0,1,…,J−1.(63)\lambda_j = -1 - \chi(e^{ij4\pi/J} - e^{-ij2\pi/J}), \qquad j = 0, 1, \ldots, J - 1. \tag*{(63)}

Note that

Re⁡(λj)=−1−χ(cos⁡(4ξj)−cos⁡(2ξj)),(64)\operatorname{Re}(\lambda_j) = -1 - \chi(\cos(4\xi_j) - \cos(2\xi_j)), \tag*{(64)}

where ξj=4πj/J\xi_j = 4\pi j/J. We are interested in the region for χ\chi where Re⁡(λj)<0\operatorname{Re}(\lambda_j) < 0 for all jj. As J→∞J \to\infty, we have the bound −89≤cos⁡(4ξj)−cos⁡(2ξj)≤2-\frac{8}{9} \le\cos(4\xi_j) - \cos(2\xi_j) \le2. This implies that the region is given by −12<x<89-\frac{1}{2} < x < \frac{8}{9}. For a finite JJ, the bounds of cos⁡(4ξj)−cos⁡(2ξj)\cos(4\xi_j) - \cos(2\xi_j) are determined by the resolution of ξj\xi_j, which is a discrete function of jj. This affords us to take a slightly wider region of xx where Re⁡(λj)<0\operatorname{Re}(\lambda_j) < 0 for all jj. For example, when J=4J = 4, −12<x<1-\frac{1}{2} < x < 1.

Homotopy continuation can be used on the polynomial system of equations defined by ff to reveal a high-order polynomial containing all possible roots. Maple was used to evaluate all the roots for multiple values of χ\chi and for J=4,5,6J = 4,5,6. Within the region χ∈(−12,89)\chi\in\left(-\frac{1}{2},\frac{8}{9}\right), the only real-valued solution is given by yj=χy_j = \chi for j=1,…,Jj = 1,\ldots,J. We conjecture that there is only a single real-valued solution to (60) when χ∈(−12,89)\chi\in\left(-\frac{1}{2},\frac{8}{9}\right) for J>6J > 6.

We have shown that the multi-scale Lorenz96 equations are normally-stable for a restricted range of xkx_k. However, for any xkx_k, yj,ly_{j,l}, ϵ>0\epsilon> 0, the system is energy stable, which implies that, for any initial condition, the energy will decay in time and the dynamics will eventually be able to be described by a normally fast-slow system for some time t>0t > 0.

Theorem 5.2. The equations defined by (5.8) are energy-stable for ϵ>0\epsilon> 0.

Consider again the original system. We multiply the equations by xkx_k and yj,ky_{j,k}, respectively. Let (⋅)′=ddt(⋅)(\cdot)' = \frac{d}{dt}(\cdot), then

xkxk′=−ϵxkxk−1(xk−2−xk+1)−ϵxk2−ϵ2∑j=1Jxkyj,k(65)x_k x_k' = -\epsilon x_k x_{k-1}(x_{k-2} - x_{k+1}) - \epsilon x_k^2 - \epsilon^2 \sum_{j=1}^{J} x_k y_{j,k} \tag*{(65)}
yj,kyj,k′=−yj,kyj+1,k(yj+2,k−yj−1,k)−yj,k2+xkyj,k.(66)y_{j,k} y_{j,k}' = -y_{j,k}y_{j+1,k}(y_{j+2,k} - y_{j-1,k}) - y_{j,k}^2 + x_k y_{j,k}. \tag*{(66)}

We collect the terms and take the sum over the discrete variables.

12∑k=1K(xk2)′=−ϵ∑k=1Kxkxk−1xk−2+ϵ∑k=1Kxk+1xkxk−1−ϵ∑k=1Kxk2−ϵ2∑k=1K∑j=1Jxkyj,k\frac{1}{2}\sum_{k=1}^{K}(x_k^2)' = -\epsilon\sum_{k=1}^{K}x_kx_{k-1}x_{k-2} + \epsilon\sum_{k=1}^{K}x_{k+1}x_kx_{k-1} - \epsilon\sum_{k=1}^{K}x_k^2 - \epsilon^2\sum_{k=1}^{K}\sum_{j=1}^{J}x_ky_{j,k}
12∑j=1J(yj,k2)′=−∑j=1Jyj,kyj+1,kyj+2,k+∑j=1Jyj−1,kyj,kyj+1,k−∑j=1Jyj,k2+∑j=1Jxkyj,k\frac{1}{2}\sum_{j=1}^{J}(y_{j,k}^2)' = -\sum_{j=1}^{J}y_{j,k}y_{j+1,k}y_{j+2,k} + \sum_{j=1}^{J}y_{j-1,k}y_{j,k}y_{j+1,k} - \sum_{j=1}^{J}y_{j,k}^2 + \sum_{j=1}^{J}x_ky_{j,k}

Due to periodicity, the cubic terms evaluate to zero. The remaining terms give us the following energy estimate.

12∑k=1K(xk2)′+12ϵ2∑k=1K∑j=1J(yj,k2)′=−ϵ∑k=1Kxk2−ϵ2∑k=1K∑j=1Jyj,k2<0(67)\frac{1}{2}\sum_{k=1}^{K}(x_k^2)' + \frac{1}{2}\epsilon^2\sum_{k=1}^{K}\sum_{j=1}^{J}(y_{j,k}^2)' = -\epsilon\sum_{k=1}^{K}x_k^2 - \epsilon^2\sum_{k=1}^{K}\sum_{j=1}^{J}y_{j,k}^2 < 0 \tag*{(67)}

We train a FSNN to learn the multi-scale Lorenz96 equations from data choosing J=K=4J = K = 4. The trajectory dataset, D\mathcal{D}, consists of 10410^4 fast-scale trajectories, corresponding to initial conditions for xkx_k randomly generated uniformly in [−12,1]\left[-\frac{1}{2},1\right] and yj,ky_{j,k} randomly generated from a normal distribution, ϵ\epsilon randomly generated log-uniformly in [10−5,10−2]\left[10^{-5},10^{-2}\right], and integrated forward 10 timesteps using an adaptive RK45 solver with error tolerance 10−810^{-8}. The manifold dataset, M\mathcal{M} consists of 10410^4 examples of data pairs ((xk,yj,k),ϵ=0)((x_k,y_{j,k}),\epsilon= 0) on the graph yj,k=xky_{j,k} = x_k, where xkx_k is randomly generated uniformly in [−12,1]\left[-\frac{1}{2},1\right].

A FSNN is trained on this dataset using the Adam optimizer from Optax. The INN consists of one outer layer, where the inner coupling flow layers consists of feedforward networks with one layer and a hidden dimension of 40. The feedforward neural networks on the RHS of the neural ODE system each consist of 40 layers and the bilinear map has a rank of 4.

Figure 7 shows a comparison of the trained model and the reference solution for an example initial condition with ϵ=1.1⋅10−3\epsilon= 1.1 \cdot10^{-3}. The model shows quantitative agreement to the reference solution. Although the globally attracting solution is the zero solution, the error remains away from 0. This can be explained by approximation error of the model and can be improved through including more data in the training set. Figure 8 shows a comparison of the eigenvalues of the trained model with the analytical eigenvalues. Agreement is observed, implying that the model was successful in mapping the slow manifold approximately to the true slow manifold. In this example, the time interval of all the training data is much shorter than t=3t = 3. Nevertheless, the prediction can be done up to T=3000T = 3000 or longer with excellent accuracy.

Dynamics Comparison, Lorenz96

Figure 7. The top five plots are trajectory plots comparing the reference solution computed using RK45 (bold curves) and that of the trained model (dotted markers) for an initial condition with ϵ=1.1⋅10−3\epsilon= 1.1 \cdot10^{-3} for the multi-scale Lorenz96 example described by the dynamics in (5.8). The bottom five plots show the corresponding errors as a function of time. The plots in the left column show the solution and error on the slow time scale (t∈[0,3000]t \in[0, 3000]) while the plots in the right column zoom in on the fast time scale (t∈[0,3]t \in[0, 3]).

Eigenvalues of $T(x)$ Comparison, Lorenz96

Figure 8. Comparison of the ground truth eigenvalues of the Jacobian of the fast dynamics (bold lines) with that of the negative Schur form learned from the trained model (dots) for the multi-scale Lorenz96 example described by the dynamics in (5.8)

Abraham-Lorentz Equations

In this section, we consider the Abraham-Lorentz equations [32], which model the radiation reaction in the case of a single electron with charge ee and mass mm moving through a static external magnetic field B(x)∈R3\mathbf{B}(\mathbf{x}) \in\mathbb{R}^{3}. The equations are given by

2e23c3dadτ=ma−ecv×B(x),(68)\frac{2e^{2}}{3c^{3}}\frac{\mathrm{d}a}{\mathrm{d}\tau}=ma-\frac{e}{c}\mathbf{v}\times\mathbf{B}(\mathbf{x}), \tag*{(68)}
dvdτ=a,(69)\frac{\mathrm{d}\mathbf{v}}{\mathrm{d}\tau}=\mathbf{a}, \tag*{(69)}
dxdτ=v,(70)\frac{\mathrm{d}\mathbf{x}}{\mathrm{d}\tau}=\mathbf{v}, \tag*{(70)}

where x(τ)\mathbf{x}(\tau), v(τ)\mathbf{v}(\tau), a(τ)∈R3\mathbf{a}(\tau) \in\mathbb{R}^{3} are the position, velocity, and acceleration, respectively, and cc is the speed of light. We rescale these equations following the approach in [32]. Time is scaled by the observer time scale TT as τ=Tτˉ\tau=T\bar{\tau}, space by the observer length scale LL as x=Lxˉ\mathbf{x}=L\bar{\mathbf{x}}, velocity as v=(L/T)vˉ\mathbf{v}=(L/T)\bar{\mathbf{v}}, the magnetic field as B(x)=B0Bˉ(x)\mathbf{B}(\mathbf{x})=B_{0}\bar{\mathbf{B}}(\mathbf{x}), and acceleration as a=(L/T)(∣e∣B0)(mc)−1aˉ\mathbf{a}=(L/T)(|e|B_{0})(mc)^{-1}\bar{\mathbf{a}}. We reformulate the rescaled Abraham-Lorentz equations in terms of the nondimensional variables and drop the overline for notational convenience.

(r0cT)23dadτ=a−ζv×B(x),(71)\left(\frac{r_{0}}{cT}\right)\frac{2}{3}\frac{\mathrm{d}\mathbf{a}}{\mathrm{d}\tau}=\mathbf{a}-\zeta\mathbf{v}\times\mathbf{B}(\mathbf{x}), \tag*{(71)}
dvdτ=(ωcT)a,(72)\frac{\mathrm{d}\mathbf{v}}{\mathrm{d}\tau}=(\omega_{c}T)\mathbf{a}, \tag*{(72)}
dxdτ=v,(73)\frac{\mathrm{d}\mathbf{x}}{\mathrm{d}\tau}=\mathbf{v}, \tag*{(73)}

where we have introduced the cyclotron frequency ωc=∣e∣B0/(mc)\omega_{c}=|e|B_{0}/(mc), the classical electron radius r0=e2/(mc2)r_{0}=e^{2}/(mc^{2}), and the sign of the charge ζ∈{−1,+1}\zeta\in\{-1,+1\}, where ζ=−1\zeta=-1 corresponds to electrons and ζ=+1\zeta= +1 to positrons. The dimensionless parameters

γR=r0cT,γB=1ωcT,(74)\gamma_{\mathrm{R}} = \frac{r_0}{cT}, \qquad\gamma_{\mathrm{B}} = \frac{1}{\omega_{\mathrm{c}}T}, \tag*{(74)}

represent the ratio of the electron size to the distance light travels during a time interval TT, and ratio of the cyclotron period to TT, respectively. The system can be transformed into a two-scale fast-slow system with the choice γB=1\gamma_{\mathrm{B}} = 1 and γR=ϵ≪1\gamma_{\mathrm{R}} = \epsilon\ll1. We scale time through the transformation τ=23ϵt\tau= \frac{2}{3}\epsilon t to reveal a normally-unstable fast-slow system given by

dadt=a−ζv×B(x),(75)\frac{d\mathbf{a}}{dt} = \mathbf{a} - \zeta\mathbf{v} \times\mathbf{B}(\mathbf{x}), \tag*{(75)}
dvdt=ϵ23a,(76)\frac{d\mathbf{v}}{dt} = \epsilon\frac{2}{3}\mathbf{a}, \tag*{(76)}
dxdt=ϵ23v.(77)\frac{d\mathbf{x}}{dt} = \epsilon\frac{2}{3}\mathbf{v}. \tag*{(77)}

We will refer to (5.19) as the forward-time Abraham-Lorentz system. A normally-stable fast-slow system can be obtained by reversing time using t−=−tt_- = -t,

dadt−=−a+ζv×B(x),(78)\frac{d\mathbf{a}}{dt_-} = -\mathbf{a} + \zeta\mathbf{v} \times\mathbf{B}(\mathbf{x}), \tag*{(78)}
dvdt−=−ϵ23a,(79)\frac{d\mathbf{v}}{dt_-} = -\epsilon\frac{2}{3}\mathbf{a}, \tag*{(79)}
dxdt−=−ϵ23v.(80)\frac{d\mathbf{x}}{dt_-} = -\epsilon\frac{2}{3}\mathbf{v}. \tag*{(80)}

We refer to (5.20) as the reverse-time Abraham-Lorentz system. In both systems, a\mathbf{a} is the fast variable and v,x\mathbf{v}, \mathbf{x} are the slow variables. In reverse time, the acceleration, a\mathbf{a} will be attracted to the slow manifold, given by

a∗(x,v)=ζv×B(x)+32ϵv×B(x)×B(x)+O(ϵ2).(81)\mathbf{a}^*(\mathbf{x},\mathbf{v}) = \zeta\mathbf{v} \times\mathbf{B}(\mathbf{x}) + \frac{3}{2}\epsilon\mathbf{v} \times\mathbf{B}(\mathbf{x}) \times\mathbf{B}(\mathbf{x}) + O(\epsilon^2). \tag*{(81)}

Though the reverse-time system is stable on the fast time scale, the system exhibits an instability tangential to the slow manifold for which solutions blow up on the slow time scale. This is a more challenging test than the previous ones.

It is of interest to obtain forward-time trajectories of electrons along the slow manifold. A standard numerical integration of (5.19) will result in unstable trajectories on the fast time scale since any numerical error will push trajectories off of the slow-manifold, resulting in exponential growth of the acceleration. We instead apply FSNNs to learn the Fenichel normal form of the system in reverse time. We then obtain forward-time trajectories along the learned slow manifold by performing an integration of the slow variables along y=0y = 0 using (3.22). This test is therefore a more practical application of a data-driven closure.

We train an FSNN to learn the Abraham-Lorentz equations from data. The trajectory dataset, D\mathcal{D}, consists of 10410^4 trajectories, corresponding to initial conditions randomly generated in the region, R\mathcal{R}, specified by

R={x,v,a:2≤r≤4,∣z∣≤1,∥v∥≤1,∥a−a∗(x,v)∥≤1}.\mathcal{R} = \{\mathbf{x}, \mathbf{v}, \mathbf{a}: \quad2 \le r \le4,\quad|z| \le1,\quad\lVert\mathbf{v}\rVert\le1,\quad\lVert\mathbf{a} - \mathbf{a}^*(\mathbf{x},\mathbf{v})\rVert\le1\}.

where r=x12+x22r=\sqrt{x_1^2+x_2^2} and z=x3z=x_3. ϵ\epsilon is randomly generated log-uniformly in the range [10−5,10−2][10^{-5},10^{-2}]. The trajectories are integrated in reverse time using equations (5.20) using an adaptive RK45 solver set to an error tolerance of 10−810^{-8}. The first 40 time steps batched in sequences of length 10 are used as fast-scale training data. These trajectories are also interpolated over a slower time scale with Δt=0.025/ϵ\Delta t=0.025/\epsilon and 10410^4 sequences of length 10 are chosen at random for slow-scale training data. The manifold dataset, M\mathcal{M} consists of 10410^4 examples of data pairs ((x,v,a),ϵ=0)((\mathbf{x},\mathbf{v},\mathbf{a}),\epsilon=0) on the graph a=v×B(x)\mathbf{a}=\mathbf{v}\times\mathbf{B}(\mathbf{x}), where x,v\mathbf{x},\mathbf{v} are randomly generated in the region R\mathcal{R}.

The magnetic field is chosen to be B=Brer+Bϕeϕ+Bzez\mathbf{B}=B_r\mathbf{e}_r+B_\phi\mathbf{e}_\phi+B_z\mathbf{e}_z, where er,eϕ,ez\mathbf{e}_r,\mathbf{e}_\phi,\mathbf{e}_z represent basis vectors for cylindrical coordinates, and

Br=−zq(r,z)rB0,Bϕ=R0rB0,Bz=(r−R0)q(r,z)rB0(82)B_r=-\frac{z}{q(r,z)r}B_0,\qquad B_\phi=\frac{R_0}{r}B_0,\qquad B_z=\frac{(r-R_0)}{q(r,z)r}B_0 \tag*{(82)}

where q(r,z)=q0+q2((r−R0)(r−R0)+z2)/a2q(r,z)=q_0+q_2((r-R_0)(r-R_0)+z^2)/a^2 and q0=1.2q_0=1.2, q2=2.8q_2=2.8, a=1.5a=1.5, R0=3.0R_0=3.0, Z0=3.0Z_0=3.0, and B0=1.0B_0=1.0. This field is an idealized equilibrium of a tokamak, where the field is physical inside a torus with major radius of R0R_0 and minor radius of aa. Here qq is called a safety factor [68] and the poloidal magnetic flux function is ψ=B0a22q2ln⁡(q)\psi=\frac{B_0a^2}{2q_2}\ln(q). Under this definition, the given field can be rewritten as

B=1R∇ψ×eϕ+F0Reϕ.(83)\mathbf{B}=\frac{1}{R}\nabla\psi\times\mathbf{e}_\phi+\frac{F_0}{R}\mathbf{e}_\phi. \tag*{(83)}

where the poloidal field is a constant of F0=R0B0F_0=R_0B_0. This test thus models a charged particle moving through a tokamak with radiation reaction.

A FSNN is trained on this dataset using the Adam optimizer from Optax. The INN consists of one outer layer, where the inner coupling flow layers consists of feedforward networks with one layer and a hidden dimension of 50. The function gg on the right-hand-side of the neural ODE uses a single-layer feedforward neural network with hidden dimension 100 and is added to a bilinear form network with rank 1. The remaining feedforward neural networks on the right-hand-side of the neural ODE system each consist of 50 layers and the bilinear map has a rank of 4.

Figure 9 shows comparisons of the solutions in rr-zz space between the trained network and trajectories computed using RK45 on the reverse-time system. Using the reverse-time trajectories, endpoints were supplied as initial conditions to the trained network to predict forward in time. For both the reverse-time predictions (red arrows) and forward-time predictions (green arrows), the trained model agrees well with the solution computed numerically. The corresponding L2L2 errors are shown for position, velocity, and acceleration in Figure 10, demonstrating that errors are low when integrating on the slow time scale.

Six trajectory plots for reverse-time and forward-time Abraham–Lorentz trajectories

Figure 9. Displayed are the trajectories for three values of ϵ\epsilon evolved backwards in time along the slow manifold (top three plots, red arrows) and three values of ϵ\epsilon evolved forwards in time along the slow manifold (bottom three plots, green arrows). The solid lines represent the ground truth trajectories and the dotted lines represent the model.

Error trajectories for the reverse time and forward time simulations

Figure 10. Error trajectories for the reverse time (top three plots) and forward time simulations are displayed. For each simulation, the L2L_2 norm of position, velocity and acceleration are plotted.

Table 2 shows the comparison of the true eigenvalues of the linear part of the fast dynamics of the reverse-time system. The eigenvalues were recovered with a great deal of accuracy, demonstrating that the fast-scale dynamics are learned accurately.

Reverse-Time Abraham-Lorentz System Eigenvalues
g.t.modelerror
-1-0.999623.8e-04
-1-0.999584.2e-04
-1-0.999584.2e-04

Table 2. Comparison of the true eigenvalues Jacobian of the fast dynamics to the eigenvalues of the negative Schur form of the trained model for the Abraham-Lorentz problem.

Figure 11 shows comparisons of orbits between the trained model and orbits obtained when integrating the system by substituting the first two terms of the asymptotic expansion in (81). Note again that when integrating along the slow manifold, only the slow variable is evolved using the transformation hh and the closure y=0y=0. The trained model is able to qualitatively capture the gyro-motion behavior while also accurately capturing the guiding center dynamics over the slow time scale. Despite training using trajectories over a relatively short time frame (2 or less gyration periods) due to the slow time scale instability of the reverse-time system, the trained network is highly capable of predicting long-time dynamics in forward-time for many more gyration cycles. We remind the reader that standard numerical methods cannot integrate the forward-time Abraham-Lorentz system along the slow manifold without experiencing an exponential growth instability. Direct integration of the asymptotic expansion removes the instability, but introduces an error dependent on ϵ\epsilon due to the asymptotic approximation. This result highlights the unique capability of the proposed network.

Comparison of forward-time Abraham–Lorentz slow manifold orbits for asymptotic expansion and model

Figure 11. Comparison of orbits of the forward-time Abraham-Lorentz system (5.19) along the slow manifold for the first two terms of the asymptotic expansion in (81) (left) and the trained model (right) for two different initial conditions and choices of ϵ\epsilon.

Conclusions

In this paper, we introduced the fast-slow neural network (FSNN), which is a data-driven approach suitable for learning singularly perturbed dynamical systems where the fast-scale dynamics are dissipative. Our method enforces the existence of a trainable, attracting invariant slow manifold as a hard constraint. Invertible neural networks and neural ODEs are key components of the FSNN. We introduce bi-Lipschitz affine transformation (bLAT) layers for enforcing regularity of the invertible neural network. We also develop a negative Schur form network which parameterizes matrices with eigenvalues to the left of the imaginary axis. This in addition to the use of an additive L-stable diagonally-implicit Runge Kutta scheme for integration are key pieces in ensuring that the fast-scale dynamics are stable. The Schur form network also enables the use of back-substitution to efficiently handle the implicit solve. Analytical representation of the slow manifold enables efficient integration on the slow time scale. We demonstrate the FSNN on many examples that exhibit two timescales, including the Grad moment system from hydrodynamics, two-scale Lorenz96 equations for modeling atmospheric dynamics, and Abraham-Lorentz dynamics modeling radiation reaction of electrons in a magnetic field.

In order to effectively train the FSNN, a sufficient number of trajectories across multiple values of ϵ\epsilon and initial conditions are necessary. Despite this, we demonstrate that the network can be used to discover a full-order model with long-term accuracy near a slow manifold when only short duration trajectories are available. Therefore the proposed network is also applicable to discover a reduced-order model as a unique closure discovery procedure.

In the examples we consider, the equations and slow manifold reduction in the limit ϵ=0\epsilon= 0 are known a priori. Therefore, we were able to bypass an important shortcoming of our approach, which is the need to determine the dimensionality of the slow manifold. While a plethora of techniques exist in the literature [47], it would be interesting to combine them to our technique to tackle problems where the dimensionality is unknown. A future direction of our work involves considering hyperbolic fast slow systems, which can be represented using a more general Fenichel normal form consisting of both stable and unstable directions.

Acknowledgment. This research used resources provided by the Los Alamos National Laboratory Institutional Computing Program, which is supported by the U.S. Department of Energy National Nuclear Security Administration under Contract No. 89233218CNA000001, and the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory, operated under Contract No. DE-AC02-05CH11231 using NERSC award ASCR-ERCAP0023112.

References

  1. [1]R. T. Chen, Y. Rubanova, J. Bettencourt, D. K. Duvenaud, Neural ordinary differential equations, Advances in neural information processing systems 31.
  2. [2]N. Fenichel, Geometric singular perturbation theory for ordinary differential equations, Journal of Differential Equations 31 (1) (1979) 53–98.DOI
  3. [3]C. Foisal, G. R. Sell, R. Temam, Inertial manifolds for nonlinear evolutionary equations, Journal of differential equations 73 (2) (1988) 309–353.
  4. [4]E. D. Koronaki, N. Evangelou, C. P. Martin-Linares, E. S. Titi, I. G. Kevrekidis, Nonlinear dimensionality reduction then and now: AIMs for dissipative pdes in the ml era, Journal of Computational Physics 506 (2024) 112910.
  5. [5]C. Foias, M. Jolly, I. Kerekridis, G. Sell, E. Titi, On the computation of inertial manifolds, Physics Letters A 131 (7-8) (1988) 433–436.DOI
  6. [6]C. Foias, G. Sell, E. Titi, Exponential tracking and approximation of inertial manifolds for dissipative nonlinear equations, Journal of Dynamics and Differential Equations 1 (1989) 199–244.DOI
  7. [7]M. S. Jolly, Explicit construction of an inertial manifold for a reaction diffusion equation, Journal of Differential Equations 78 (2) (1989) 220–261.DOI
  8. [8]R. Temam, Do inertial manifolds apply to turbulence?, Physica D: Nonlinear Phenomena 37 (1) (1989) 146–152.DOI
  9. [9]R. Temam, Induced trajectories and approximate inertial manifolds, ESAIM: M2AN 23 (3) (1989) 541–561.DOI
  10. [10]F. Jauberteau, C. Rosier, R. Temam, A nonlinear galerkin method for the navier-stokes equations, Computer Methods in Applied Mechanics and Engineering 80 (1) (1990) 245–260.DOI
  11. [11]J.-L. Guermond, S. Prudhomme, A fully discrete nonlinear galerkin method for the 3d navier-stokes equations, Numerical Methods for Partial Differential Equations 24 (2008) 759 – 775.DOI
  12. [12]M. S. Jolly, I. Kevrekidis, E. S. Titi, Approximate inertial manifolds for the Kuramoto-Sivashinsky equation: analysis and computations, Physica D: Nonlinear Phenomena 44 (1-2) (1990) 36–60.DOI
  13. [13]E. S. Titi, On approximate inertial manifolds to the navier-stokes equations, Journal of Mathematical Analysis and Applications 149 (2) (1990) 540–557.DOI
  14. [14]M. A. Kramer, Nonlinear principal component analysis using autoassociative neural networks, AIChE Journal 37 (2) (1991) 233–243.DOI
  15. [15]K. Krischer, R. Rico-Martínez, I. G. Kevrekidis, H. H. Rotermund, G. Ertl, J. L. Hudson, Model identification of a spatiotemporally varying catalytic reaction, AIChE Journal 39 (1) (1993) 89–98.DOI
  16. [16]S. Y. Shvartsman, I. G. Kevrekidis, Nonlinear model reduction for control of distributed systems: A computer-assisted study, AIChE Journal 44 (7) (1998) 1579–1595.DOI
  17. [17]S. Shvartsman, C. Theodoropoulos, R. Rico-Martínez, I. Kevrekidis, E. Titi, T. Mountziaris, Order reduction for nonlinear dynamic models of distributed reacting systems, Journal of Process Control 10 (2) (2000) 177–184.
  18. [18]S. Shvartsman, C. Theodoropoulos, R. Rico-Martínez, I. Kevrekidis, E. Titi, T. Mountziaris, Order reduction for nonlinear dynamic models of distributed reacting systems, Journal of Process Control 10 (2) (2000) 177–184.
  19. [19]P. Benner, S. Gugercin, K. Willcox, A survey of projection-based model reduction methods for parametric dynamical systems, SIAM Review 57 (4) (2015) 483–531.DOI
  20. [20]A. J. Chorin, F. Lu, Discrete approach to stochastic parametrization and dimension reduction in nonlinear dynamics, Proceedings of the National Academy of Sciences 112 (32) (2015) 9804–9809.
  21. [21]F. Lu, K. K. Lin, A. J. Chorin, Data-based stochastic model reduction for the kuramoto-sivashinsky equation, Physica D: Nonlinear Phenomena 340 (2017) 46–57.arxiv.org/abs/1509.09279
  22. [22]Y. Bar-Sinai, S. Hoyer, J. Hickey, M. P. Brenner, Learning data-driven discretizations for partial differential equations, Proceedings of the National Academy of Sciences 116 (31) (2019) 15344–15349.
  23. [23]K. Lee, K. T. Carlberg, Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, Journal of Computational Physics 404 (2020) 108973.arxiv.org/abs/1812.08373
  24. [24]A. J. Linot, M. D. Graham, Deep learning to discover and predict dynamics on an inertial manifold, Phys. Rev. E 101 (2020) 062209.DOI
  25. [25]K. Zeng, A. J. Linot, M. D. Graham, Data-driven control of spatiotemporal chaos with reduced-order neural ode-based models and reinforcement learning, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 478 (2267) (2022) 20220297.DOI
  26. [26]A. J. Linot, M. D. Graham, Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations, Chaos: An Interdisciplinary Journal of Nonlinear Science 32 (7) (2022) 073110.DOI
  27. [27]K. Zeng, C. E. P. D. Jesús, A. J. Fox, M. D. Graham, Autoencoders for discovering manifold dimension and coordinates in data from complex dynamical systems (2023).
  28. [28]C. E. P. De Jesús, M. D. Graham, Data-driven low-dimensional dynamic model of kolmogorov flow, Phys. Rev. Fluids 8 (2023) 044402.
  29. [29]A. J. Linot, K. Zeng, M. D. Graham, Turbulence control in plane couette flow using low-dimensional neural ode-based models and deep reinforcement learning, International Journal of Heat and Fluid Flow 101 (2023) 109139.arxiv.org/abs/2301.12098
  30. [30]A. J. Linot, J. W. Burby, Q. Tang, P. Balaprakash, M. D. Graham, R. Maulik, Stabilized neural ordinary differential equations for long-time forecasting of dynamical systems, Journal of Computational Physics 474 (2023) 111838.DOI
  31. [31]D. G. Patsatzis, G. Fabiani, L. Russo, C. Siettos, Slow invariant manifolds of singularly perturbed systems via physics-informed machine learning, arXiv preprint arXiv:2309.07946.
  32. [32]J. W. Burby, T. Klotz, Slow manifold reduction for plasma science, Commun. Nonlinear Sci. Numer. Simul. 89 (2020) 105289.DOI
  33. [33]H. Grad, On the kinetic theory of rarefied gases, Communications on Pure and Applied Mathematics 2 (4) (1949) 331–407.DOI
  34. [34]S. Braginskii, Transport processes in a plasma, Reviews of plasma physics 1 (1965) 205.
  35. [35]G. W. Hammett, F. W. Perkins, Fluid moment models for Landau damping with application to the ion-temperature-gradient instability, Physical review letters 64 (25) (1990) 3019.
  36. [36]C. Jones, Geometric singular perturbation theory, 1995.DOI
  37. [37]T. Teshima, I. Ishikawa, K. Tojo, K. Oono, M. Ikeda, M. Sugiyama, Coupling-based invertible neural networks are universal diffeomorphism approximators, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Curran Associates Inc., Red Hook, NY, USA, 2020.
  38. [38]N. Gicquel, J. Anderson, I. Kevrekidis, Noninvertibility and resonance in discrete-time neural networks for time-series processing, Physics Letters A 238 (1) (1998) 8–18.DOI
  39. [39]R. Rico-Martínez, R. Adomaitis, I. Kevrekidis, Noninvertibility in neural networks, Computers & Chemical Engineering 24 (2000) 2417–2433.
  40. [40]T. Cui, T. Bertalan, G. J. Pappas, M. Morari, I. G. Kevrekidis, M. Fazlyab, Certified invertibility in neural networks via mixed-integer programming (2023).
  41. [41]M. Lezcano-Casado, Trivializations for gradient-based optimization on manifolds, in: Advances in Neural Information Processing Systems, NeurIPS, 2019, pp. 9154–9164.
  42. [42]P. Jin, Z. Zhang, A. Zhu, Y. Tang, G. E. Karniadakis, SympNets: Intrinsic structure-preserving symplectic networks for identifying hamiltonian systems, Neural Networks 132 (2020) 166–179.arxiv.org/abs/2001.03750
  43. [43]J. Burby, Q. Tang, R. Maulik, Fast neural Poincaré maps for toroidal magnetic fields, Plasma Physics and Controlled Fusion 63.
  44. [44]Y. Liu, J. N. Kutz, S. L. Brunton, Hierarchical deep learning of multiscale differential equation time-steppers, Philosophical Transactions of the Royal Society A 380 (2229) (2022) 20210200.
  45. [45]V. Duruisseaux, J. W. Burby, Q. Tang, Approximation of nearly-periodic symplectic maps via structure-preserving neural networks, Scientific Reports 13 (1) (2023) 8351.
  46. [46]R. R. Coifman, S. Lafon, Diffusion maps, Applied and computational harmonic analysis 21 (1) (2006) 5–30.DOI
  47. [47]M. Meilă, H. Zhang, Manifold learning: What, how, and why, Annual Review of Statistics and Its Application 11 (Volume 11, 2024) (2024) 393–417.arxiv.org/abs/2311.03757
  48. [48]A. Sawant, A. Acharya, Model reduction via parametrized locally invariant manifolds: some examples, Computer methods in applied mechanics and engineering 195 (44-47) (2006) 6287–6311.DOI
  49. [49]R. A. Horn, C. R. Johnson, Matrix Analysis, Cambridge University Press, 1985.
  50. [50]Z.-Z. Bai, J.-Y. Pan, Matrix Analysis and Computations, 2021.DOI
  51. [51]G.-R. Duan, R. J. Patton, A note on hurwitz stability of matrices, Automatica 34 (4) (1998) 509–511.DOI
  52. [52]G. V. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems 2 (1989) 303–314.
  53. [53]G. Berkooz, P. Holmes, J. L. Lumley, The proper orthogonal decomposition in the analysis of turbulent flows, Annual review of fluid mechanics 25 (1) (1993) 539–575.
  54. [54]L. Ardizzone, J. Kruse, S. Wirkert, D. Rahner, E. W. Pellegrini, R. S. Klessen, L. Maier-Hein, C. Rother, U. Köthe, Analyzing inverse problems with invertible neural networks, arXiv preprint arXiv:1808.04730.arxiv.org/abs/1808.04730
  55. [55]L. Dinh, J. Sohl-Dickstein, S. Bengio, Density estimation using real NVP, arXiv preprint arXiv:1605.08803.arxiv.org/abs/1605.08803
  56. [56]J. F. Blinn, Consider the lowly 2 × 2 matrix, IEEE Computer Graphics and Applications 16 (1996) 82–88.
  57. [57]H. Zhang, Y. Liu, R. Maulik, Semi-implicit neural ordinary differential equations for learning chaotic systems, in: NeurIPS 2023 Workshop Heavy Tails in Machine Learning, 2023.arxiv.org/abs/2412.11301
  58. [58]L. Pareschi, G. Russo, Implicit–explicit Runge–Kutta schemes and applications to hyperbolic systems with relaxation, Journal of Scientific Computing 25 (2005) 129–155.
  59. [59]R. Rico-Martínez, K. Krischer, I. G. Kevrekidis, M. C. Kube, J. L. Hudson, Discrete- vs. continuous-time nonlinear signal processing of Cu electro dissolution data, Chemical Engineering Communications 118 (1) (1992) 25–48.DOI
  60. [60]R. Rico-Martínez, I. Kevrekidis, K. Krischer, Nonlinear system identification using neural networks: dynamics and instabilities, Neural networks for chemical engineers (1995) 409–442.
  61. [61]J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, A. VanderPlas, S. Wanderman-Milne, Q. Zhang, JAX: composable transformations of Python+NumPy programs (2018).
  62. [62]DeepMind, I. Babuschkin, K. Bauml, A. Bell, S. Bhupatiraju, J. Bruce, P. Buchlovsky, D. Budden, T. Cai, A. Clark, I. Danihelka, A. Dedieu, C. Fantacci, J. Godwin, C. Jones, R. Hemsley, T. Hennigan, M. Hessel, S. Hou, S. Kapturowski, T. Keck, I. Kemaev, M. King, M. Kunesch, L. Martens, H. Merzic, V. Mikulik, T. Norman, G. Papamakarios, J. Quan, R. Ring, F. Ruiz, A. Sanchez, L. Sartran, R. Schneider, E. Sezener, S. Spencer, S. Srinivasan, M. Stanojevic, W. Stokowiec, L. Wang, G. Zhou, F. Viola, The DeepMind JAX Ecosystem (2020).
  63. [63]A. Gorban, I. Karlin, Hilbert’s 6th problem: Exact and approximate hydrodynamic manifolds for kinetic equations, Bulletin of the American Mathematical Society 51 (2013) 186–246.
  64. [64]G. Haller, S. Ponsioen, Nonlinear normal modes and spectral submanifolds: existence, uniqueness and use in model reduction, Nonlinear dynamics 86 (2016) 1493–1534.arxiv.org/abs/1602.00560
  65. [65]M. Cenedese, J. Axàs, B. Bäuerlein, K. Avila, G. Haller, Data-driven modeling and prediction of non-linearizable dynamics via spectral submanifolds, Nature Communications 13 (2022) 872.
  66. [66]E. Lorenz, Predictability: a problem partly solved, Seminar on Predictability, 4-8 September 1995 1 (1995) 1–18.
  67. [67]I. Fatkullin, E. Vanden-Eijnden, A computational strategy for multiscale systems with applications to lorenz 96 model, Journal of Computational Physics 200 (2) (2004) 605–638.DOI
  68. [68]S. Liu, Q. Tang, X.-z. Tang, A parallel cut-cell algorithm for the free-boundary Grad–Shafranov problem, SIAM Journal on Scientific Computing 43 (6) (2021) B1198–B1225.arxiv.org/abs/2012.06015

Paper details

vlnc:W_6q9z2wkt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents