betaFeedback

WorksComputer Science

Investigating the Surrogate Modeling Capabilities of Continuous Time Echo State Networks

Abstract

Continuous Time Echo State Networks (CTESNs) are a promising yet under-explored surrogate modeling technique for dynamical systems, particularly those governed by stiff Ordinary Differential Equations (ODEs). A key determinant of the generalization accuracy of a CTESN surrogate is the method of projecting the reservoir state to the output. This paper shows that of the two common projection methods (linear and nonlinear), the surrogates developed via the nonlinear projection consistently outperform those developed via the linear method. CTESN surrogates are developed for several challenging benchmark cases governed by stiff ODEs, and for each case, the performance of the linear and nonlinear projections is compared. The results of this paper demonstrate the applicability of CTESNs to a variety of problems while serving as a reference for important algorithmic and hyper-parameter choices for CTESNs

Introduction

Modeling dynamic systems using scientific machine learning (SciML) techniques is a rapidly growing field with advanced ML architectures being applied to model complex problems across a diverse range of applications. Some examples are rapid design optimization [1], real-time health monitoring [2], turbulent flow modeling [3], and materials discovery [4]. Many of these applications utilize a ” surrogate model ” that makes real-time predictions of the system behavior in place of full-order models that would be too slow or expensive for the application.

Recently, Echo State Networks (ESNs) have seen an increase in popularity for modeling highly nonlinear and chaotic phenomena in domains such as optimal control planning [5], chaotic time series prediction [6, 7], signals analysis [8] and even turbulent fluid flow [9]. These applications leverage the ability of ESNs to capture highly nonlinear transient behavior accurately, as well as the extremely low cost of training them, with the empirical success of ESNs on a wide range of approximation tasks discussed and explained in several works [10, 11]. One of the biggest limitations, however, for using the standard ESN implementation in the surrogate modeling of nonlinear dynamical systems, is that the available training data may not be uniformly sampled in time. A particular example of this is the numerical solution of stiff systems of Ordinary Differential Equations (ODEs) from ODE solvers. An ODE system is said to be stiff if, for any initial conditions and in certain intervals, the solving numerical method is forced to use a timestep which is very small compared to the smoothness of the exact solution [12]. Numerical ODE solvers overcome the instability due to stiffness by having a variable time step size during the solve, leading to uneven sampling of the solution in time.

There have been several attempts to apply reduced-order modeling for stiff ODEs. Ji et al. [13] used Physics Informed Neural Networks [14], Kim et al. [15] used Neural ODEs [16] and Goswami et al. [17] used Deep Operator Networks [18] to solve several stiff systems such as the ROBER [19] and POLLU [20] problems. However, the methodologies applied require assumptions and scalings that may not generalize, or require lots of training data and compute resources to train deep neural networks. Work by Anantharaman et al. [21] also showed the failure of popular architectures such as Long Short Term Memory (LSTM) and standard ESNs in modeling stiff systems.

To use the attractive properties of ESNs (i.e capacity to model highly nonlinear signals and ease of training) for modeling stiff systems, a variant of ESNs called Continuous Time Echo State Networks or CTESNs has been proposed [21] to address this issue. CTESNs have been successfully employed in various applications from accelerating predictions of power system dynamics [22] to accelerating solutions of pharmacology models [23].

In recent literature, two ways of using CTESNs for surrogate modeling have emerged; the Linear Projection CTESN (LPCTESN) [21] and the Nonlinear Projection CTESN (NLPCTESN) [22, 23] and have been applied to a range of problems. However, there is currently a lack of work critically examining the accuracy of both methods and how they compare to one another. Further, both projection methods use a Radial Basis Function (RBF) for interpolation, which also comes with several algorithmic choices that need to be explored. This study aims to fill this gap by investigating the effects of these algorithmic choices on surrogates created to solve several stiff ODE systems such as Robertson’s equations, the Sliding Basepoint model of automobile collision, and the POLLU air pollution model. The findings of the study show that for the same hyper-parameter settings of the CTESN, the NLPCTESN outperforms the LPCTESN on all benchmarks shown. Further, it is shown that for the interpolating RBFs used within CTESNs, k-Nearest Neighbor (k-NN) polynomial augmented RBFs outperform standard RBFs in predictive accuracy.

This paper is divided as follows: Section 2 introduces the concept of ESN and CTESN, along with the projection methods LPCTESN and NLPCTESN described above discussed in detail. Section 3 demonstrates the application of the methods to several challenging stiff ODE problems, with a qualitative and quantitative discussion of the results. Section 4 summarizes the work with possible future directions to take.

Methods

Standard Echo State Networks

An Echo State Network [24], depicted in Figure 1, is a form of reservoir computing that is very similar to the more popular Recurrent Neural Network (RNN) in its architecture. They make predictions by following a recurrence relation (Equation (1)) for updating a latent space vector and using that vector to map to a given output. Unlike RNNs, the parameters of the RNN in the ”reservoir” are fixed and are not updated during training, and it is only the mapping from latent space to the output space that is learned. Depending on the implementation, this makes the training of ESNs very fast (sometimes as fast as a simple least squares fit) and computationally cheap.

Depiction of a standard Echo State Network

Figure 1. Depiction of a standard Echo State Network.

The governing equation for an echo state network reads:

rn+1=σ(Winxn+Wrn),(1)\mathbf{r}_{n+1} = \sigma(\mathbf{W}_{\mathrm{in}}\mathbf{x}_n + \mathbf{W}\mathbf{r}_n), \tag*{(1)}

where rn∈RNr\mathbf{r}_n \in\mathbb{R}^{N_r} is the latent state at timestep nn, σ\sigma is an activation function (most commonly tanh), Win∈RNr×Nx\mathbf{W}_{\mathrm{in}} \in\mathbb{R}^{N_r \times N_x} and W∈RNr×Nr\mathbf{W} \in\mathbb{R}^{N_r \times N_r} are the reservoir matrices which are fixed and randomly initialized, and xn\mathbf{x}_n is the system state at time nn. The matrix W\mathbf{W} is a sparse matrix and usually has 1% nonzero entries.

The output projection reads:

xn+1=Φ(rn+1),(2)\mathbf{x}_{n+1} = \Phi(\mathbf{r}_{n+1}), \tag*{(2)}

where Φ\Phi is decided by the projection method. The most popular projection method is the linear projection, resulting in

xn+1=Woutrn+1,(3)\mathbf{x}_{n+1} = \mathbf{W}_{\mathrm{out}}\mathbf{r}_{n+1}, \tag*{(3)}

where Wout∈RNx×Nr\mathbf{W}_{\mathrm{out}} \in\mathbb{R}^{N_x \times N_r} is the linear projection matrix that needs to be fitted to the training data. Like most machine learning algorithms, ESNs have a set of hyper-parameters that need to be tuned, and there are several works [25, 26] that can be used as guides to select them. A key hyper-parameter in ESNs is the spectral radius of W\mathbf{W}. In this study, the spectral radius is fixed to a value of (0.01) for all models created. Although a bit smaller than the conventional values used in standard ESNs, it was found in this study on CTESNs that using the slightly smaller value maximized predictive accuracy while also ensuring the stability of the reservoir (r\mathbf{r}) solution. To fit the trainable matrix, one only has to solve the ordinary linear least squares problem:

If X=[x1;x2;…;xN]X = [\mathbf{x}_1; \mathbf{x}_2; \ldots; \mathbf{x}_N] and R=[r1;r2;…;rN]R = [\mathbf{r}_1; \mathbf{r}_2; \ldots; \mathbf{r}_N] then

Wout=(RRT)−1RX.(4)\mathbf{W}_{\mathrm{out}} = (RR^{T})^{-1}RX. \tag*{(4)}

Continuous Time Echo State Networks (CTESN)

Continuous Time Echo State Networks are a variant of ESNs that model time as a continuous rather than discrete quantity. The model equation for a CTESN is given by:

r˙=σ(Winx+Wr),(5)\dot{\mathbf{r}} = \sigma(\mathbf{W}_{\mathrm{in}}\mathbf{x} + \mathbf{W}\mathbf{r}), \tag*{(5)}

with all variables having the same definition as in the previous section. The projection equation reads:

x=Φ(r).(6)\mathbf{x} = \Phi(\mathbf{r}). \tag*{(6)}

As per the literature, there are two ways of modeling the relation between the latent space and the outputs [27]; the linear method (called Linear Projection CTESN or LPCTESN), described by:

x=Woutr.(7)\mathbf{x} = \mathbf{W}_{\mathrm{out}}\mathbf{r}. \tag*{(7)}

The second method is the nonlinear method (called NonLinear Projection CTESN or NLPCTESN), where the projection Φ\Phi is a nonlinear mapping. Many possible functions can be used, but the literature on NLPCTESNs [22, 27] currently uses standard Radial Basis Functions (RBFs) to write the projection:

x=RBF⁡(r).(8)\mathbf{x} = \operatorname{RBF}(\mathbf{r}). \tag*{(8)}

Surrogate Modeling via CTESNs

To create and use a surrogate model via the CTESN method, a few steps are followed. First, a Design of Experiments (DoE) space is created and NN sample combinations of query parameters are drawn; call this set P={p1,p2,…,pN}P=\{p_1,p_2,\ldots,p_N\}. Each pip_i represents a set of conditions at which the ODE is solved and the surrogate is expected to capture the change in the solution due to changes in the value of pip_i. Examples include the initial conditions (see § 3.1.2 and § 3.2), rate constants of the ODE (see § 3.1.1 and § 3.3), etc. The ODE is solved numerically at each pi∈Pp_i \in P, to return the solution set Y={y1,y2,…,yN}Y=\{y_1,y_2,\ldots,y_N\}, yi∈RNx×Ntsy_i \in\mathbb{R}^{N_x \times N_{ts}}. A single parameter combination p∗∈Pp^* \in P is drawn at random, and the reservoir ODE (given by Equation (5)) is solved using a numerical ODE solver. This returns a r(t)∈RNr×Nts\mathbf{r}(t) \in\mathbb{R}^{N_r \times N_{ts}} where NtsN_{ts} is the number of timesteps in the solution at p∗p^*. Then, depending on whether the method follows a linear or nonlinear projection method, either Algorithm 1 or Algorithm 2 is followed to fit and query the surrogate.

Algorithm 1 Linear Projection CTESN Surrogate Fitting

for yiy_i in Y\mathbf{Y} do

Fit Wouti\mathbf{W}^{i}_{out} from yi=Wouti⋅r\mathbf{y}_i=\mathbf{W}^{i}_{out}\cdot\mathbf{r} using Equation (4)

end for

Fit RBF mapping Wouti=RBF(pi)\mathbf{W}^{i}_{out}=\mathbf{RBF}(p_i), ∀  (Wouti,pi)\forall\;(\mathbf{W}^{i}_{out},p_i) pairs

To query a new parameter ptestp_{test}:

Step 1: Wouttest=RBF(ptest)\mathbf{W}^{test}_{out}=\mathbf{RBF}(p_{test})

Step 2: ytest(t)=Wouttest⋅r\mathbf{y}_{test}(t)=\mathbf{W}^{test}_{out}\cdot\mathbf{r}

Algorithm 2 NonLinear Projection CTESN Surrogate Fitting

for yiy_i in Y\mathbf{Y} do

Fit RBF1i\mathbf{RBF}^{i}_{1} as per yi=RBF1i(r(t),WRBF1i)\mathbf{y}_i=\mathbf{RBF}^{i}_{1}(\mathbf{r}(t),\mathbf{W}^{i}_{RBF_{1}})

end for

Fit RBF2\mathbf{RBF}_{2} as per WRBF1i=RBF2(pi)\mathbf{W}^{i}_{RBF_{1}}=\mathbf{RBF}_{2}(p_i), ∀  (WRBF1i,pi)\forall\;(\mathbf{W}^{i}_{RBF_{1}},p_i) pairs

To query a new parameter ptestp_{test}:

Step 1: WRBF1test=RBF2(ptest)\mathbf{W}^{test}_{RBF_{1}}=\mathbf{RBF}_{2}(p_{test})

Step 2: ytest(t)=RBF1(WRBF1test,r(t))\mathbf{y}_{test}(t)=\mathbf{RBF}_{1}(\mathbf{W}^{test}_{RBF_{1}},\mathbf{r}(t))

Algorithms [1] and [2] both use an interpolating RBF in their procedure, and in this article, it is demonstrated that polynomial augmented k-nearest neighbor (k-NN) RBFs deliver superior results in terms of generalization to new test problems for CTESN-based surrogate models, compared to standard RBFs. The reader is encouraged to read Appendix Section A for a detailed explanation of k-NN polynomial augmented RBFs.

Applications and Results

CTESN surrogates are created for three problems. First Robertson’s equations are solved, parametrizing the rates of reaction and initial condition separately. Then, the Sliding Basepoint system is modeled, comparing explicitly the difference due to the projection method, and parametrizing the initial conditions of the problem. Finally, the POLLU system is solved, again parametrizing the initial conditions of the problem and comparing the results obtained using differing projection methods. Unless mentioned otherwise, the training data was sampled from the DoE space randomly.

In all results in this section, the MAE is computed as

MAE=1Ntest∑j=1Ntest∑i=1Ntsj∣yj,predi−yj,truei∣Ntsj,(9)\mathrm{MAE}=\frac{1}{N_{test}}\sum_{j=1}^{N_{test}}\frac{\sum_{i=1}^{N^{j}_{ts}}\left|y^{i}_{j,pred}-y^{i}_{j,true}\right|}{N^{j}_{ts}}, \tag*{(9)}

where NtestN_{test} is the number of test cases, yj,prediy^{i}_{j,pred} and yj,trueiy^{i}_{j,true} are the prediction and true solution respectively for the i’th time step of the j’th test case.

Robertson’s Equation

The first demonstrated application is the surrogate modeling of the Robertson Equations [19]. They are written as:

y˙1=−r1⋅y1+r3⋅y2⋅y3,y˙2=r1⋅y1−r3⋅y2⋅y3−r2⋅y22,y˙3=r2⋅y22,(10)\begin{aligned} \dot{y}_1 &= -r_1\cdot y_1+r_3\cdot y_2\cdot y_3,\\ \dot{y}_2 &= r_1\cdot y_1-r_3\cdot y_2\cdot y_3-r_2\cdot y_2^2,\\ \dot{y}_3 &= r_2\cdot y_2^2, \tag*{(10)} \end{aligned}

where y1y_1, y2y_2 and y3y_3 represent the concentration of the reactant species and r1r_1, r2r_2 and r3r_3 represent the fixed rates of reaction. The system is usually subject to initial conditions of [y1,y2,y3]=[1,0,0][y_1,y_2,y_3] = [1,0,0]. They are a system of ODEs that describe a standard rate process and are often used to benchmark numerical ODE solvers due to the stiffness of the system. More specifically, the long-time integration of Robertson’s equations is known to be a challenging problem for numerical ODE solvers, and the system hence serves as a good test for surrogate models. In this work models are created by parametrizing the system in two ways; first, the rates of reaction are parametrized. This has been the focus of several other papers written on CTESNs [21, 27]. Second, the initial conditions of the system are parametrized.

Parametrizing Rates of Reaction

In this section, the Design of Experiment (DoE) space for the rates is given as:

r1=[0.032,0.048],r2=[2.4,3.6]×E7,r3=[0.8,1.2]×E4.(11)\begin{aligned} r_1 &= [0.032,0.048],\\ r_2 &= [2.4,3.6]\times E7,\\ r_3 &= [0.8,1.2]\times E4. \tag*{(11)} \end{aligned}

The focus is limited to presenting the results of the prediction of the variable y2y_2. This variable has a sharp transient that occurs at a time scale much smaller than the other two, and hence causes the system to be stiff.

The average MAE for y2y_2, averaged across several test cases listed in Table 1, is shown in Table 2 sorted in descending order of generalization MAE. Predictions for y2y_2 for a particular test parameter are shown in Figure 2. For the same hyper-parameters, it can be observed that the absence of either the augmenting polynomial or the k-NN interpolation (i.e using all collocation points to predict the solution in the RBF) significantly increases the error of prediction. It can also be seen that the nonlinear projection performs better on average than the linear projection.

Parameter Setr1/0.04r_1/0.04r2/3⋅E7r_2/3\cdot E7r3/1⋅E4r_3/1\cdot E4
P10.951.050.95
P20.91.10.9
P31.10.91.1
P41.030.991.04

Table 1. Test parameter values for rate parametrization of Robertson’s ODE system

Trial No.NrN_rNdataN_{data}NneighN_{neigh}Polynomial AugmentationProjection MethodAvg. MAE (×10−7\times10^{-7})
150504NLinear17.6
25050AllYLinear1.40
3505004YLinear1.29
450504YLinear0.76
550504YNonlinear0.32

Table 2. Average Mean Absolute Error (MAE) in y2y_2, averaged across all test cases.

Trial 1

(a) Trial 1

Trial 2

(b) Trial 2

Trial 3

(c) Trial 3

Trial 4

(d) Trial 4

Trial 5

(e) Trial 5

Figure 2. Figures show the time history for y2y_2 for a test parameter set (P2 from Table 1). Each trial mentioned below is referenced from Table 2. The NLPCTESN prediction is the best out of all models. (a) Trial 1 (b) Trial 2 (c) Trial 3 (d) Trial 4 (e) Trial 5

In the next section, the initial condition of Robertson’s ODE is parametrized, and a similar error analysis is done.

Parametrizing Initial Conditions

The initial condition of the system is parameterized. This problem can be challenging for lower initial values of y1(0)y_1(0) as it leads to a smaller and sharper transient in y2y_2, which becomes more difficult to capture accurately by the surrogate. The DoE space of the initial condition is given as

y1(0)=[0.5,1],y2(0)=0,y3(0)=1−y1(0).(12)\begin{aligned} y_1(0) &= [0.5,1],\\ y_2(0) &= 0,\\ y_3(0) &= 1-y_1(0). \tag*{(12)} \end{aligned}

The condition for y3y_3 is decided on the basis that the sum of all quantities should always equal 1. Once more, the focus is on comparing the predicted results in y2y_2.

Table 4 tabulates the average MAE across all test cases for the problem, listed in Table 3, sorted in descending order. Figure 3 shows the comparison for a test parameter, across 5 different configurations of the CTESN surrogate model. A similar trend of hyper-parameter performance as seen in Table 2 is noted, in that the absence of the k-NN interpolation increases the error of the prediction. For the same hyper-parameters, the nonlinear projection once again achieves lower generalization MAE than the linear projection.

Parameter Sety1y_1y2y_2y3y_3
P10.600.4
P20.700.3
P30.8500.15
P40.900.1

Table 3. Test parameter values for initial condition parametrization of Robertson’s ODE system

Trial No.NrN_rNdataN_{data}NneighN_{neigh}Projection MethodAvg. MAE (×10−7\times10^{-7})
150050AllLinear158
25050AllLinear28.6
3500504Linear25.2
450504Linear6.23
550504Nonlinear0.99

Table 4. Average Mean Absolute Error (MAE) in y2y_2, averaged across all test cases when parametrizing the initial condition.

Trial 1

(a) Trial 1

Trial 2

(b) Trial 2

Trial 3

(c) Trial 3

Trial 4

(d) Trial 4

Trial 5

(e) Trial 5

Figure 3. Figures show time history for y2y_2 for a test parameter set (P4 from Table 3). Each trial mentioned below is referenced from Table 4. The NLPCTESN performs the best out of all models. (a) Trial 1 (b) Trial 2 (c) Trial 3 (d) Trial 4 (e) Trial 5

Sliding Basepoint Model

Finally, the discussed methods are applied to the surrogate modeling of a realistic crash safety design problem. A system of ODEs proposed by Horváth et al. [28] that approximately models an automobile collision is solved via a created surrogate model. This system, called the Sliding Basepoint model, has parameters that were fitted on realistic crash data and are assumed to accurately model a collision problem. Figure 4 depicts the system at its initial state and at a later time. The system is given as:

Sliding Basepoint system for collision modeling

Figure 4. Sliding Basepoint system for collision modeling.

x˙1=v1,v˙1=1m1F1(Fs,c1,v1),x˙2=v2,v˙2=1m2F2(−Fs,c2,v2),k˙=Dk⋅P,c˙1=Dc⋅P.(13)\begin{aligned} \dot{x}_1 &= v_1,\\ \dot{v}_1 &= \frac{1}{m_1}F_1(F_s,c_1,v_1),\\ \dot{x}_2 &= v_2,\\ \dot{v}_2 &= \frac{1}{m_2}F_2(-F_s,c_2,v_2),\\ \dot{k} &= D_k \cdot P,\\ \dot{c}_1 &= D_c \cdot P. \tag*{(13)} \end{aligned}

where m1,m2,F1,F2,x1,x2,v1,v2m_1,m_2,F_1,F_2,x_1,x_2,v_1,v_2 refer to the masses, total forces on, positions and velocities of the bodies respectively. m1m_1 in reality reflects the mass of the chassis of the car, x1x_1 behaves similar to the deformation of the bumper. FsF_s represents the spring force:

Fs=k⋅(x2−x1),(14)F_s = k \cdot(x_2 - x_1), \tag*{(14)}

between the two masses and P represents the power of dissipation

P=∣m1⋅v˙12∣.(15)P = \left|m_1 \cdot\dot{v}_1^2\right|. \tag*{(15)}

Finally, the forces are computed based on best-fit models described in the paper:

F1(Fs,c1,v1)=Fs−c1⋅∣v1∣⋅sign⁡(v1),F_1(F_s,c_1,v_1) = F_s - c_1 \cdot\lvert v_1\rvert\cdot\operatorname{sign}(v_1),
F2(Fs,c2,v2)={0,if Fs<c2 or ∣v2∣<vfFs−c2⋅sign⁡(v2),otherwise(16)F_2(F_s,c_2,v_2) = \begin{cases} 0, & \text{if } F_s < c_2 \text{ or } \lvert v_2\rvert< v_f \\ F_s - c_2 \cdot\operatorname{sign}(v_2), & \text{otherwise} \end{cases} \tag*{(16)}

The values of the fixed parameters are given in Table 6. The reader is referred to the paper [28] for further details. The system is subject to initial conditions [x1,v1,x2,v2,k,c1]=[0,v0,0,0,k0,0][x_1,v_1,x_2,v_2,k,c_1] = [0,v_0,0,0,k_0,0].

m1m_1 (kg)m2m_2 (kg)DkD_kDcD_cc2c_2 (N)vfv_f (m/s)
191624.93.380.4591450000.1

Table 6. Parameter values taken from [28] with their SI units.

The mechanics of crash and impact problems are known to have sharp transients and highly oscillatory behaviors associated with them, making their numerical solutions slow and costly to compute for a wide range of parameters. Hence, CTESNs will be a good surrogate modeling tool for the problem.

In this work, the initial conditions of the problem (v0v_0 and k0k_0) are parametrized to simulate different impact velocities and directions (the spring constant of the bumper can be assumed to be different in different directions). The state space for the problem is the vector [x1,v1,x2,v2,k,c1][x_1, v_1, x_2, v_2, k, c_1].

The DoE space is the range:

v0=[5,25] (m/s),k0=[1,10]×106 kg/s2,(17)\begin{aligned} v_0 &= [5, 25]\ \text{(m/s)},\\ k_0 &= [1, 10] \times10^6\ \text{kg/s}^2, \tag*{(17)} \end{aligned}

chosen to represent a wide range of speeds and stiffness constants. The DoE space was sampled using a space-filling sampling strategy [29] and the surrogate models were trained on 100 data points, and data was generated using a stiff ODE solver using solver parameters given in [28].

Table 7 shows the average MAE for the state variable v2v_2 for the linear and nonlinear projection methods, all other hyper-parameters being kept the same, for test parameters listed in Table 6. v2v_2 was chosen to demonstrate the accuracy of the surrogates because, as visible in Figure 5, it has sharp nonlinear oscillatory transients starting from the moment of collision, which is difficult to capture for surrogate models. It can be seen from Table 7 and Figure 5 that the LPCTESN has a poorer performance on test cases compared to the NLPCTESN.

Figure 5. Solution v2v_2 for several tests with different test parameters, the average error of which is shown in Table 7. The left and right columns show results from the linear and nonlinear projections respectively. Table 5 defines the values P3-P5 mentioned below. (a) LPCTESN - P3 (b) NLPCTESN - P3 (c) LPCTESN - P4 (d) NLPCTESN - P4 (e) LPCTESN - P5 (f) NLPCTESN - P5

Parameter Setv0/25v_0/25k0/106k_0/10^6
P10.4964.46
P20.8848.45
P30.4241.43
P40.2833.75
P50.99.95

Table 5. Test parameter values for the collision modeling problem

Trial No.NxN_xNneighN_{neigh}Projection MethodAvg. MAE (v2v_2)
1504Linear1.02
2504Nonlinear0.51

Table 7. Average Mean Absolute Error (MAE) in v2v_2, averaged across all test cases in Table 5

Of particular interest is the speedup obtained by using the surrogate; using the surrogate leads to up to a 200x speedup in the prediction of the solution, the ODE solver taking roughly 0.02 seconds per solve of the ODE system. This is important because the transients in Figure 5 occur at the same time scale O(10−2 s)O(10^{-2}\,\mathrm{s}). With a 200x speedup, if the surrogate is deployed on board a vehicle, it can judge the severity of the collision much quicker by computing impact forces from the result of the system, and closed-loop control and safety measures can be deployed. In this case, it would effectively function as a digital twin for collision monitoring.

The POLLU Model

The CTESN approach is used to model the POLLU air pollution model developed at the Dutch National Institute of Public Health and Environmental Protection [20]. It consists of 20 species and 25 reactions, modeled by nonlinear ODEs of the form

dydt=P(t,y)−L(t,y)y(18)\frac{dy}{dt} = P(t,y) - L(t,y)y \tag*{(18)}

where y is the concentration vector (in ppm) of the reacting species, P is the production term and L is a diagonal matrix representing the loss term for every species in the system. Table 8 shows the production and loss rates for each species of the system. The values of the rate constants rr are given in table 11. The reader is referred to the paper [20] for a complete description of the reaction system.

Species (yy)Production Rate (PP)Loss Rate (LL)
1r2y2y4+r3y5y2+r9y11y2+r11y13+r12y10y2+r22y19+r25y20r_2y_2y_4 + r_3y_5y_2 + r_9y_{11}y_2 + r_{11}y_{13} + r_{12}y_{10}y_2 + r_{22}y_{19} + r_{25}y_{20}r1+r10y11+r14y6+r23y4+r24y19r_1 + r_{10}y_{11} + r_{14}y_6 + r_{23}y_4 + r_{24}y_{19}
2r1y1+r21y19r_1y_1 + r_{21}y_{19}r2y4+r3y5+r9y11+r12y10r_2y_4 + r_3y_5 + r_9y_{11} + r_{12}y_{10}
3r1y1+r17y5+r19y16+r22y19r_1y_1 + r_{17}y_5 + r_{19}y_{16} + r_{22}y_{19}r15r_{15}
4r15y3r_{15}y_3r2y2+r16+r17+r23y1r_2y_2 + r_{16} + r_{17} + r_{23}y_1
5r4y7+r4y7+r6y7y6+r7y9+r13y14+r20y17y6r_4y_7 + r_4y_7 + r_6y_7y_6 + r_7y_9 + r_{13}y_{14} + r_{20}y_{17}y_6r3y2r_3y_2
6r3y5y2+r18y16+r18y16r_3y_5y_2 + r_{18}y_{16} + r_{18}y_{16}r6y7+r8y9+r14y1+r20y17r_6y_7 + r_8y_9 + r_{14}y_1 + r_{20}y_{17}
7r13y14r_{13}y_{14}r4+r5+r6y6r_4 + r_5 + r_6y_6
8r4y7+r5y7+r6y7y6+r7y9r_4y_7 + r_5y_7 + r_6y_7y_6 + r_7y_90.00.0
900r7+r8y6r_7 + r_8y_6
10r7y9+r9y11y2r_7y_9 + r_9y_{11}y_2r12y2r_{12}y_2
11r8y9y6+r11y13r_8y_9y_6 + r_{11}y_{13}r9y2+r10y1r_9y_2 + r_{10}y_1
12r9y11y2r_9y_{11}y_20.00.0
13r10y11y1r_{10}y_{11}y_1r11r_{11}
14r12y10y2r_{12}y_{10}y_2r13r_{13}
15r14y1y6r_{14}y_1y_60.00.0
16r16y4r_{16}y_4r18+r19r_{18} + r_{19}
170.00.0r20y6r_{20}y_6
18r20y17y6r_{20}y_{17}y_60.00.0
19r23y1y4+r25y20r_{23}y_1y_4 + r_{25}y_{20}r21+r22+r24y1r_{21} + r_{22} + r_{24}y_1
20r24y19y1r_{24}y_{19}y_1r25r_{25}

Table 8. Species involved in the POLLU reaction system, with their production and loss rates, derived from [20].

The system is a common benchmark for stiff ODE solvers and represents a difficult problem to solve due to the large number of species and ODEs involved. When such systems have to be solved at many grid points, say in a computational mesh, they represent a very expensive computation and hence this example is an ideal application for testing surrogates of stiff ODEs. In this work, the initial conditions of the system are simultaneously parametrized, according to the following:

y2,0=[0.14,0.26],y4,0=[0.028,0.052],y7,0=[0.07,0.13].(19)\begin{aligned} y_{2,0} &= [0.14, 0.26],\\ y_{4,0} &= [0.028, 0.052],\\ y_{7,0} &= [0.07, 0.13]. \tag*{(19)} \end{aligned}

where y2,0y_{2,0}, y4,0y_{4,0} and y7,0y_{7,0} refer to the initial concentration of the respective species. The initial conditions for the rest of the species are default as per the paper [20].

The training data was sampled using 100 data points within this DoE space, selected randomly. The reservoir size Nr\mathbf{N}_r was set to 100, and the number of queried neighbors Nneigh\mathbf{N}_{neigh} by the k-NN RBF was set to 10. Table 10 shows the mean absolute errors computed across several test cases (listed in Table 9) for a few species in the reaction, for the linear and nonlinear projection methods. It can be observed that the nonlinear projection CTESN outperforms the linear projection CTESN when all other hyper-parameters are kept the same. Figure 6 shows the comparison of the prediction graphically; the LPCTESN prediction is much noisier than the NLPCTESN prediction at later times. This was also observed with Robertson’s equations, where the predictions at larger time scales by the LPCTESN tended to become noisier.

Parameter Sety2,0y_{2,0}y4,0y_{4,0}y7,0y_{7,0}
P10.1920.04640.114
P20.2580.040120.103
P30.1880.03320.0938
P40.2080.05080.115
P50.2280.04880.0714

Table 9. Test parameter values for the POLLU air pollution modeling problem

Projection MethodMean Absolute Error
y1(×10−5)y_1(\times10^{-5})y2(×10−5)y_2(\times10^{-5})y4(×10−4)y_4(\times10^{-4})y14(×10−8)y_{14}(\times10^{-8})y20(×10−6)y_{20}(\times10^{-6})
Linear75.176.37.6851.522.1
Nonlinear2.602.783.011.762.59

Table 10. Mean Absolute Error (MAE) for the POLLU problem, averaged across all test cases in Table 9.

Figure 6. Solutions y1y_1, y2y_2, y14y_{14} and y20y_{20} for a test parameter (P2 from Table 9). The left and right columns contain results from the linear and nonlinear projections respectively. The NLPCTESN outperforms the LPCTESN in this example. (a) LPCTESN - y1y_1 (b) NLPCTESN - y1y_1 (c) LPCTESN - y2y_2 (d) NLPCTESN - y2y_2 (e) LPCTESN - y14y_{14} (f) NLPCTESN - y14y_{14} (g) LPCTESN - y20y_{20} (h)

Discussion and Conclusion

From the numerical experiments conducted, it was observed that polynomial-augmented K-nearest neighbor RBFs outperform standard RBFs in terms of accuracy when used as part of the CTESN surrogate modeling algorithm. It was observed that the nonlinear projection (NLPCTESN) method consistently outperformed the linear projection (LPCTESN) method, achieving superior accuracy when the hyper-parameters are the same, on a variety of problems. The NLPCTESN method demonstrated accuracy across several problems with sharp transients, varying time scales, and long horizons of integration, whereas the LPCTESN method had a higher generalization error on all test examples. The surrogate was also shown to achieve a speedup of several orders of magnitude compared to an ODE solver of a realistic collision problem and could be used for several cases like design optimization and online collision severity monitoring.

There are several directions in which this work could be built upon. The CTESN is a black-box data-driven method, and future works need to investigate how well the model learns the physics of the problem, or apply physics-constrained modeling approaches to the outputs of the CTESN. The speedup of the surrogate model becomes very apparent when solving many instances of the ODE; this happens either when the ODE system is very large, or the small ODE system has to be solved repeatedly many times, such as in coupled ODE-PDE systems. Examples include chemically reacting flows similar to the POLLU system, in which a large stiff system of ODEs has to be solved at every compute node, or FEM-based crash solvers which require accurate modeling of sharp transients similar to those demonstrated in this work.

k-Nearest Neighbor Polynomial Augmented Radial Basis Function Interpolation

Radial basis function (RBF) interpolation is a high-order accurate method that uses radial basis functions to create interpolants of unstructured data, which can be in an arbitrary number of dimensions. The scalar form of the interpolant is

s(x)=∑i=0Nwiϕ(∥x−xi∥2)(20)s(x)=\sum_{i=0}^{N}w_i\phi(\lVert x-x_i\rVert_2) \tag*{(20)}

where xix_i represent the points at which the solution is known, wiw_i are the associated coefficients which are fitted and ϕ\phi is a kernel function. One of the most common kernel functions, also used in this work, is

ϕ(ϵr)=(ϵr)2log⁡(ϵr)(21)\phi(\epsilon r)=(\epsilon r)^2\log(\epsilon r) \tag*{(21)}

where ϵ\epsilon is the shape factor, which is an important hyperparameter. In this work, it is set to 1. However, it can often greatly affect the generalization capability of the RBF. Cao et al. [30] discussed how high-order polynomial augmented RBFs outperformed standalone RBFs and reduced the dependency of the RBF on the shape factor ϵ\epsilon, and several works [31, 32] have shown that adding high-order polynomials to the RBF greatly enhances the accuracy of the model. In this work, the polynomial augmented RBF takes the form:

f(x)=K(x,y)A+P(x)B(22)f(x)=\mathbf{K}(x,\mathbf{y})\mathbf{A}+\mathbf{P}(x)\mathbf{B} \tag*{(22)}

where A\mathbf{A} and B\mathbf{B} are fitted coefficient matrices, and K\mathbf{K} and P\mathbf{P} need to be constructed for a query point x\mathbf{x}, given data points y\mathbf{y}. If NdN_d, NON_O, NpN_p are determined by the number of data points, the output dimension, and the polynomial order respectively, we have A∈RNd×NO\mathbf{A}\in\mathbb{R}^{N_d\times N_O} and B∈RNp×NO\mathbf{B}\in\mathbb{R}^{N_p\times N_O}. K∈RNd\mathbf{K}\in\mathbb{R}^{N_d} can be constructed as

Ki(x)=ϕ(∥x−yi∥), i=1…Nd,(23)K_i(x)=\phi(\lVert x-y_i\rVert),\ i=1\ldots N_d, \tag*{(23)}

and P∈RNp\mathbf{P}\in\mathbb{R}^{N_p} can be constructed as

P(x)=1+x+…(24)\mathbf{P}(x)=\mathbf{1}+x+\ldots \tag*{(24)}

A further improvement to RBF interpolation is adding a kk-Nearest Neighbor constraint to the prediction process. This means that during prediction, only kk of the nearest collocation points (i.e. yiy_i) will contribute to the prediction at the test point. These NN points in practice are usually inferred using a decision tree. Intuitively, this method is useful when the interpolation spaces are large and the hyper-parameters of the RBF may not be optimally tuned, leading to collocation points at large distances from the test point and contributing to the interpolant evaluation, corrupting the solution.

POLLU Reaction rates

The rates of reaction of the POLLU system are shown in Table 11.

Reaction RateValueReaction RateValue
r1r_{1}0.350E+00.350\mathrm{E}+0r14r_{14}0.163E+50.163\mathrm{E}+5
r2r_{2}0.266E+20.266\mathrm{E}+2r15r_{15}0.480E+70.480\mathrm{E}+7
r3r_{3}0.120E+50.120\mathrm{E}+5r16r_{16}0.350E−30.350\mathrm{E}-3
r4r_{4}0.860E−30.860\mathrm{E}-3r17r_{17}0.175E−10.175\mathrm{E}-1
r5r_{5}0.820E−30.820\mathrm{E}-3r18r_{18}0.100E+90.100\mathrm{E}+9
r6r_{6}0.150E+50.150\mathrm{E}+5r19r_{19}0.444E+120.444\mathrm{E}+12
r7r_{7}0.130E−30.130\mathrm{E}-3r20r_{20}0.124E+40.124\mathrm{E}+4
r8r_{8}0.240E+50.240\mathrm{E}+5r21r_{21}0.210E+10.210\mathrm{E}+1
r9r_{9}0.165E+50.165\mathrm{E}+5r22r_{22}0.578E+10.578\mathrm{E}+1
r10r_{10}0.900E+40.900\mathrm{E}+4r23r_{23}0.474E−10.474\mathrm{E}-1
r11r_{11}0.220E−10.220\mathrm{E}-1r24r_{24}0.178E+40.178\mathrm{E}+4
r12r_{12}0.120E+50.120\mathrm{E}+5r25r_{25}0.312E+10.312\mathrm{E}+1
r13r_{13}0.188E+10.188\mathrm{E}+1

Table 11. Reaction rates for the POLLU problem

References

  1. [1]Renato G Nascimento, Felipe AC Viana, Matteo Corbetta, and Chetan S Kulkarni. A framework for li-ion battery prognosis based on hybrid bayesian physics-informed neural networks. Scientific Reports, 13(1):13856, 2023.
  2. [2]Arman Malekloo, Ekin Ozer, Mohammad AlHamydeh, and Mark Girolami. Machine learning and structural health monitoring overview with emerging technology and high-dimensional data source highlights. Structural Health Monitoring, 21(4):1906–1955, 2022.
  3. [3]Mustafa Z Yousif, Linqi Yu, Sergio Hoyas, Ricardo Vinuesa, and HeeChang Lim. A deep-learning approach for reconstructing 3d turbulent flows from 2d observation data. Scientific Reports, 13(1):2529, 2023.
  4. [4]Yue Liu, Tianlu Zhao, Wangwei Ju, and Siqi Shi. Materials discovery and design using machine learning. Journal of Materiomics, 3(3):159–177, 2017.DOI
  5. [5]Jianwen Liang, Zhen Ding, Qinghua Han, Hao Wu, and Jinbao Ji. Online learning compensation control of an electro-hydraulic shaking table using echo state networks. Engineering Applications of Artificial Intelligence, 123:106274, 2023.
  6. [6]Xin Li, Fengrong Bi, Xiao Yang, and Xiaoyang Bi. An echo state network with improved topology for time series prediction. IEEE Sensors Journal, 22(6):5869–5878, 2022.DOI
  7. [7]Meiling Xu, Min Han, Tie Qiu, and Hongfei Lin. Hybrid regularized echo state network for multivariate chaotic time series prediction. IEEE Transactions on Cybernetics, 49(6):2305–2315, 2018.DOI
  8. [8]André LO Duarte and Marcio Eisencraft. Denoising of discrete-time chaotic signals using echo state networks. Signal Processing, 214:109252, 2024.DOI
  9. [9]Mohammad Sharifi Ghazijahani, Florian Heyder, Jörg Schumacher, and Christian Cierpka. On the benefits and limitations of echo state networks for turbulent flow prediction. Measurement Science and Technology, 34(1):014002, 2022.DOI
  10. [10]Lukas Gonon, Lyudmila Grigoryeva, and Juan-Pablo Ortega. Approximation bounds for random neural networks and reservoir systems. The Annals of Applied Probability, 33(1):28–69, 2023.arxiv.org/abs/2002.05933
  11. [11]Allen Hart, James Hook, and Jonathan Dawes. Embedding and approximation theorems for echo state networks. Neural Networks, 128:234–247, 2020.DOI
  12. [12]John Denholm Lambert et al. Numerical Methods for Ordinary Differential Systems, volume 146. Wiley New York, 1991.
  13. [13]Weiqi Ji, Weilun Qiu, Zhiyu Shi, Shaowu Pan, and Sili Deng. Stiff-pinn: Physics-informed neural network for stiff chemical kinetics. The Journal of Physical Chemistry A, 125(36):8098–8106, 2021.DOI
  14. [14]Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informed neural networks: Where we are and what’s next. Journal of Scientific Computing, 92(3):88, 2022.arxiv.org/abs/2201.05624
  15. [15]Suyong Kim, Weiqi Ji, Sili Deng, Yingbo Ma, and Christopher Rackauckas. Stiff neural ordinary differential equations. Chaos: An Interdisciplinary Journal of Nonlinear Science, 31(9), 2021.DOI
  16. [16]Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31, 2018.arxiv.org/abs/1806.07366
  17. [17]Somdatta Goswami, Ameya D Jagtap, Hessam Babaee, Bryan T Susi, and George Em Karniadakis. Learning stiff chemical kinetics using extended deep neural operators. Computer Methods in Applied Mechanics and Engineering, 419:116674, 2024.arxiv.org/abs/2302.12645
  18. [18]Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, 2021.DOI
  19. [19]Matthias K Gobbert. Robertson’s example for stiff differential equations. Arizona State University, Technical report, 1996.
  20. [20]Jan G Verwer. Gauss–seidel iteration for stiff odes from chemical kinetics. SIAM Journal on Scientific Computing, 15(5):1243–1250, 1994.
  21. [21]Ranjan Anantharaman, Yingbo Ma, Shashi Gowda, Chris Laughman, Viral Shah, Alan Edelman, and Chris Rackauckas. Accelerating simulation of stiff nonlinear systems using continuous-time echo state networks. arXiv preprint arXiv:2010.04004, 2020.arxiv.org/abs/2010.04004
  22. [22]Ciaran Roberts, José Daniel Lara, Rodrigo Henriquez-Auba, Matthew Bossart, Ranjan Anantharaman, Chris Rackauckas, Bri-Mathias Hodge, and Duncan S Callaway. Continuous-time echo state networks for predicting power system dynamics. Electric Power Systems Research, 212:108562, 2022.
  23. [23]Ranjan Anantharaman, Anas Abdelrehim, Anand Jain, Avik Pal, Danny Sharp, Alan Edelman, Chris Rackauckas, et al. Stably accelerating stiff quantitative systems pharmacology models: Continuous-time echo state networks as implicit machine learning. IFAC-PapersOnLine, 55(23):1–6, 2022.DOI
  24. [24]Herbert Jaeger. Echo state network. Scholarpedia, 2(9):2330, 2007.
  25. [25]Johannes Viehweg, Karl Worthmann, and Patrick Mäder. Parameterizing echo state networks for multi-step time series prediction. Neurocomputing, 522:214–228, 2023.DOI
  26. [26]Mantas Lukoševičius. A practical guide to applying echo state networks. In Neural Networks: Tricks of the Trade: Second Edition, pages 659–686. Springer, 2012.
  27. [27]Chris Rackauckas, Maja Gwozdz, Anand Jain, Yingbo Ma, Francesco Martinuzzi, Utkarsh Rajput, Elliot Saba, Viral B Shah, Ranjan Anantharaman, Alan Edelman, et al. Composing modeling and simulation with machine learning in julia. In 2022 Annual Modeling and Simulation Conference (ANNSIM), pages 1–17. IEEE, 2022.arxiv.org/abs/2105.05946
  28. [28]András Horváth, Miklós F Hatwágner, and István Á Harmati. Searching for a nonlinear ode model of vehicle crash with genetic optimization. In 2012 7th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI), pages 131–136. IEEE, 2012.DOI
  29. [29]Max D. Morris and Toby J. Mitchell. Exploratory designs for computational experiments. Journal of Statistical Planning and Inference, 43(3):381–402, 1995. ISSN 0378-3758. doi: https://doi.org/10.1016/0378-3758(94)00035-T.
  30. [30]Dingding Cao, Xinxiang Li, and Huiqing Zhu. A polynomial-augmented rbf collocation method using fictitious centres for solving the cahn–hilliard equation. Engineering Analysis with Boundary Elements, 137:41–55, 2022.DOI
  31. [31]Małgorzata A Jankowska, Andreas Karageorghis, and Ching-Shyang Chen. Improved kansa rbf method for the solution of nonlinear boundary value problems. Engineering Analysis with Boundary Elements, 87:173–183, 2018.DOI
  32. [32]Guangming Yao, Ching-Shyang Chen, and Hui Zheng. A modified method of approximate particular solutions for solving linear and nonlinear pdes. Numerical Methods for Partial Differential Equations, 33(6):1839–1858, 2017.DOI

Paper details

vlnc:W_fgte96mf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents