betaFeedback

WorksComputer Science

On the Application of Data-Driven Deep Neural Networks in Linear and Nonlinear Structural Dynamics

Abstract

The use of deep neural network (DNN) models as surrogates for linear and nonlinear structural dynamical systems is explored. The goal is to develop DNN based surrogates to predict structural response, i.e., displacements and accelerations, for given input (harmonic) excitations. In particular, the focus is on the development of efficient network architectures using fully-connected, sparsely-connected, and convolutional network layers, and on the corresponding training strategies that can provide a balance between the overall network complexity and prediction accuracy in the target dataspaces. For linear dynamics, sparsity patterns of the weight matrix in the network layers are used to construct convolutional DNNs with sparse layers. For nonlinear dynamics, it is shown that sparsity in network layers is lost, and efficient DNNs architectures with fully-connected and convolutional network layers are explored. A transfer learning strategy is also introduced to successfully train the proposed DNNs, and various loading factors that influence the network architectures are studied. It is shown that the proposed DNNs can be used as effective and accurate surrogates for predicting linear and nonlinear dynamical responses under harmonic loadings.

Introduction

In recent years, deep neural network (DNN) based models have received considerable attention due to the rapid development in computational hardware and accessibility of big data [1, 2, 3, 4]. With these developments, DNN models are now widely used in applications ranging from image classification [5, 6], speech recognition [7, 8], machine translation systems [9, 10], natural language processing [11, 12], among others. In parallel, there is also an increasing interest in applying DNNs in structural analysis [13, 14] and design [15], e.g., utilization of DNN as surrogates for structural static response [16, 17], failure probability prediction [18, 19], damage detection [20, 21, 22], parameter estimation of hysteretic structural systems [23] and structural topology optimization [24, 25, 26], among others.

In structural dynamics applications, which is the focus of this paper, early attempts of using neural network models can be traced back to the work by Masri et al. [27] where a feedforward neural network (FFNN) was trained to predict the restoring force given structural displacement and velocity at specified discrete-time instants for single-degree-of-freedom (SDOF) systems. In their study, an FFNN architecture with two hidden layers was chosen and network training was performed via a back-propagation algorithm [28]. With the advancements in learning algorithms for training deep network architectures and the availability of graphical processing units (GPUs) for data processing on large datasets [2], there is a renewed interest in the application of neural networks in structural dynamics applications. For instance, Wu et al. [29] used neural networks to predict the response of dynamical systems under earthquake loading using FFNN and convolutional neural network (CNN) models. Zhang et al. [30] applied long short-term memory (LSTM) recurrent networks to predict the structural response of a nonlinear multiple-degree-of-freedom (MDOF) system under seismic excitation. Stoffel et al. [31, 32] carried out a comparative study on the performance of different types of neural networks for approximating the deflection of an inelastic plate under an out-of-plane impulse load. Moreover, Yu et al. [33] proposed a residual recurrent neural network architecture to predict the structural responses of linear and nonlinear systems under sinusoidal or earthquake excitations. In the framework of so-called physics-informed DNNs, Zhang et al. [34] proposed to use a combination of both the standard mean squared error (MSE) of the output data and a physics loss term to train a CNN model taking ground motion acceleration as input and structural response variables as output. The physics loss term consists of an MSE of velocity computed via CNN prediction and finite difference based on the predicted displacement, and the MSE of a constraint that forces CNN outputs to satisfy the underlying governing equations. The authors later [35] used a similar loss function to train several self-tailored LSTM networks for predicting displacement, velocity, and restoring force of nonlinear structures considering a hysteresis model. In another study, Eshkevari et al. [36] designed a recurrent neural network to predict the response quantities including displacement, velocity, internal force, and acceleration for nonlinear systems subjected to ground motion acceleration.

Previous studies have shown the potential of DNNs in structural dynamics applications and many neural networks and training strategies have been presented to solve various problems. However, this is still a nascent field, and many issues have to be addressed before these networks can be successfully used in applications. In particular, in the context of SDOF and MDOF dynamical systems, two related issues must be addressed: a) design of effective network architectures, and b) construction and specification of dataspaces in which the network models are valid. For instance, in many cases, the networks are designed on an ad-hoc basis with many network layers and there is a lack of a systematic exploration of network architectures. With this architecture design strategy, efficient architectures with fewer training parameters may be overlooked. This is an important issue, as it is possible to choose a complex network architecture for a simple target system, which even with regularization is more likely to lead to overfitting [37]. Moreover, there is an increased computational overhead with large networks, as storage, training, and evaluation of large networks is expensive due to the involved floating-point operations. To address this issue, network architectures should be determined in accordance with a balance between model complexity and prediction accuracy. More importantly, when neural network models are used as data-driven surrogates, their validity is only guaranteed within the dataspaces in which they are designed and trained. For instance, a larger target design dataspace might require a deeper network as compared to a smaller dataspace. However, the characterization of the underlying modeling dataspaces is often missing, and therefore, the range of applicability of the trained networks is not clear.

In this exploratory study, DNN models for linear and nonlinear structural dynamics analysis are systematically investigated. The focus is on the construction of efficient network architectures using fully connected, sparsely connected, and convolutional network layers, and on the corresponding training strategies that can provide a balance between the overall network complexity and prediction accuracy. Considering a linear combination of harmonic functions, the input dataspace for a DNN model construction and training is explicitly expressed within which the trained DNN model can be applied. In particular, for linear structural dynamic responses, sparsity patterns of the fully connected layers in the network models are discovered. These sparsity patterns are then used to construct sparsely connected layers to reduce the trainable model parameters, and to compensate for the sparsity parameter-efficient convolutional layers are introduced. For nonlinear dynamics, it is shown that sparsity in network layers is lost, and efficient DNNs architectures with fully connected and convolutional network layers are explored. A transfer learning strategy is also presented to train the proposed DNNs, and various factors that influence the network architectures such as the duration, frequency range, number of frequencies in the load, and nonlinearity of the structure are studied. The rest of the paper is organized as follows: § 2 describes the structural dynamics problems considered in this study. § 3 presents the network architecture and training strategies. In § 4, the proposed network architectures and training methodology are applied to linear and nonlinear SDOF dynamical systems considering harmonic loading with different parameter ranges. In § 5, the application of DNNs is extended to linear and nonlinear MDOF systems. Finally, conclusions and important remarks are given in § 6.

Dynamical Systems Description

In this study, neural networks (NNs) are used to predict the response of a single degree of freedom (SDOF) and multiple degrees of freedom (MDOF) dynamical systems subjected to harmonic loadings that are randomly sampled from prescribed dataspaces. Here, appropriate dataspaces are chosen according to the available hardware and computing resources, as larger dataspaces require more memory and training time to train the networks to reach a satisfactory performance. These dynamical systems and the corresponding dataspaces are described in the following sections.

SDOF system

Linear case

For a linear SDOF system, the equation of motion reads

mu¨+cu˙+ku=p(t)(1)m\ddot{u} + c\dot{u} + ku = p(t) \tag*{(1)}

where mm, kk and cc are mass, stiffness and damping coefficient, respectively; u=u(t)u = u(t) is displacement dependent on time tt and p(t)p(t) is the applied load. The following parameters are used to model the SDOF system: mass m=13.5m = 13.5 kg, fundamental period of system Tn=0.5T_n = 0.5 sec and natural frequency of the system ωn=2π/Tn=12.6\omega_n = 2\pi/T_n = 12.6 rad/sec, and the damping ratio is taken as ξ=0.02\xi= 0.02. Thus, the stiffness is k=mωn2=2131.8k = m\omega_n^2 = 2131.8 kg/sec2^2 and the damping coefficient is c=2mωnξ=6.8c = 2m\omega_n\xi= 6.8 kg/sec. The initial conditions are u(0)=0u(0) = 0 and u˙(0)=0\dot{u}(0) = 0.

The input to this system is the loading p(t)p(t), and the response quantities of interest are the displacement u(t)u(t) and acceleration u¨(t)\ddot{u}(t) within a prescribed time window t∈[0,T]t \in[0,T]. The harmonic loading p(t)p(t) is modeled as a linear combination of harmonic components and is given by

p(t)=∑i=1Npaisin⁡(ωit+ϕi)(2)p(t) = \sum_{i=1}^{N_p} a_i \sin(\omega_i t + \phi_i) \tag*{(2)}

where NpN_p is the number of harmonic terms considered; aia_i, ωi\omega_i and ϕi\phi_i are the amplitudes, frequencies, and phase angles in the harmonic terms (i=1,2,…,Npi = 1, 2, \ldots, N_p), respectively, that are modeled as uniformly distributed random variables with probability densities given by ai∼U(0,10)a_i \sim\mathcal{U}(0,10), ωi∼U(0,6π)\omega_i \sim\mathcal{U}(0,6\pi) and ϕi∼U(0,2π)\phi_i \sim\mathcal{U}(0,2\pi). Thus, the underlying input dataspace can be described by Dlsdof={ai∼U(0,10),ϕi∼U(0,2π),and ωi∼U(0,6π) with i=1,2,…,Np}\mathcal{D}_{\mathrm{lsdof}} = \{a_i \sim\mathcal{U}(0,10), \phi_i \sim\mathcal{U}(0,2\pi), \text{and } \omega_i \sim\mathcal{U}(0,6\pi) \text{ with } i = 1,2,\ldots,N_p\}.

The response quantities at uniformly spaced discrete times (tkt_k) are of interest, where tk=kΔtt_k = k\Delta t (k=0,1,2…,nsk = 0, 1, 2 \ldots,n_s) with Δt=T/ns\Delta t = T/n_s and nsn_s is the total number of time steps. Responses are represented by n×1n \times1 vectors as y=[u(t0),u(t1),…,u(tns)]\boldsymbol{y} = [u(t_0), u(t_1), \ldots, u(t_{n_s})] or y=[u¨(t0),u¨(t1),…,u¨(tns)]\boldsymbol{y} = [\ddot{u}(t_0), \ddot{u}(t_1), \ldots, \ddot{u}(t_{n_s})], where n=ns+1n = n_s + 1. For a given input p(t)∈Dlsdofp(t) \in\mathcal{D}_{\mathrm{lsdof}}, the response y\mathbf{y} of the dynamical system in Eq. (1) can be computed using analytical solutions provided in Appendix § 8.§ 1.

Nonlinear case

For a nonlinear SDOF system with a cubic spring, the equation of motion reads

mu¨+cu˙+ku+bu3=p(t)(3)m\ddot{u} + c\dot{u} + ku + bu^3 = p(t) \tag*{(3)}

where mm, cc, kk are taken the same values as those in § 2.1.1. The coefficient in the cubic term bb is chosen from the set {0.25k,0.5k,0.75k,k}\{0.25k, 0.5k, 0.75k, k\}.

The underlying input dataspace for the harmonic loading p(t)p(t) in Eq. (2) is considered to be

Dnlsdof={ai∼U(0,20), φi∼U(0,2π), and ωi∼U(0,6π) with i=1,2,⋯ ,Np}.\mathcal{D}_{\mathrm{nlsdof}} = \{a_i \sim\mathcal{U}(0,20),\ \varphi_i \sim\mathcal{U}(0,2\pi),\ \text{and }\omega_i \sim\mathcal{U}(0,6\pi)\ \text{with } i = 1,2,\cdots,N_p\}.

The response quantities of interest in this case are still displacement or acceleration at discrete times and the response y\mathbf{y} of the dynamical system in Eq. (3) is computed using Newmark-β\beta method [38].

MDOF system

Linear case

For the linear elastic MDOF case, a dynamical system with six degrees of freedom is considered. This system is used to model a six-story shear frame with Rayleigh damping and is governed by

Mu¨+Cu˙+Ku=p(t)(4)\mathbf{M}\ddot{\mathbf{u}} + \mathbf{C}\dot{\mathbf{u}} + \mathbf{K}\mathbf{u} = \mathbf{p}(t) \tag*{(4)}

where u={ui(t)}i=16\mathbf{u} = \{u_i(t)\}_{i=1}^{6} represents displacement vector and u˙\dot{\mathbf{u}} and u¨\ddot{\mathbf{u}} are the 1st-order and 2nd-order time derivative of u\mathbf{u} representing velocity and acceleration vectors, respectively, and

M=[m1000000m1000000m2000000m2000000m3000000m3]\mathbf{M} = \begin{bmatrix} m_{1} & 0 & 0 & 0 & 0 & 0 \\ 0 & m_{1} & 0 & 0 & 0 & 0 \\ 0 & 0 & m_{2} & 0 & 0 & 0 \\ 0 & 0 & 0 & m_{2} & 0 & 0 \\ 0 & 0 & 0 & 0 & m_{3} & 0 \\ 0 & 0 & 0 & 0 & 0 & m_{3} \end{bmatrix}
K=EIH3[48−240000−2448−240000−2448−240000−2448−240000−2448−240000−2424](5)\mathbf{K} = \frac{EI}{H^{3}} \begin{bmatrix} 48 & -24 & 0 & 0 & 0 & 0 \\ -24 & 48 & -24 & 0 & 0 & 0 \\ 0 & -24 & 48 & -24 & 0 & 0 \\ 0 & 0 & -24 & 48 & -24 & 0 \\ 0 & 0 & 0 & -24 & 48 & -24 \\ 0 & 0 & 0 & 0 & -24 & 24 \end{bmatrix} \tag*{(5)}
C=α0M+α1K\mathbf{C} = \alpha_{0}\mathbf{M} + \alpha_{1}\mathbf{K}
α0=2ξωn1ωn2ωn1+ωn2andα1=2ξωn1+ωn2\alpha_{0} = \frac{2\xi\omega_{n1}\omega_{n2}}{\omega_{n1}+\omega_{n2}} \quad\text{and}\quad \alpha_{1} = \frac{2\xi}{\omega_{n1}+\omega_{n2}}
p(t)=−ag(t)M[111111]T\mathbf{p}(t) = -a_{g}(t)\mathbf{M} \begin{bmatrix} 1 & 1 & 1 & 1 & 1 & 1 \end{bmatrix}^{T}

where mim_i (i=1,2,3i = 1,2,3) are the story mass; EE, II and HH are Young’s modulus, moment of inertia and story height, respectively; ξ\xi is the damping ratio for the system, and ωn1\omega_{n1} and ωn2\omega_{n2} are the first and second natural frequencies of the system, respectively. The values of these model parameters are summarized in Table 1.

In this case, the load is applied in terms of harmonic ground motion acceleration, ag(t)a_g(t), and is modeled as the sum of sinusoidal components as

ag(t)=∑i=1Npaisin⁡(ωit+ϕi)(6)a_g(t) = \sum_{i=1}^{N_p} a_i \sin(\omega_i t + \phi_i) \tag*{(6)}

where aia_i, ωi\omega_i and ϕi\phi_i are amplitudes, frequencies, and phases of the harmonic terms that follow uniform distributions, i.e., ai∼U(0,4)a_i \sim\mathcal{U}(0,4), ωi∼U(0,20π)\omega_i \sim\mathcal{U}(0,20\pi) and ϕi∼U(0,2π)\phi_i \sim\mathcal{U}(0,2\pi), respectively. The range of the harmonic frequencies is chosen such that the first two natural frequencies are covered. Thus, the underlying input dataspace can be described by Dlmdof={ai∼U(0,4),ϕi∼U(0,2π),\mathcal{D}_{\mathrm{lmdof}} = \{a_i \sim\mathcal{U}(0,4), \phi_i \sim\mathcal{U}(0,2\pi), and ωi∼U(0,20π)\omega_i \sim\mathcal{U}(0,20\pi) with i=1,2,⋯ ,Np}i = 1,2,\cdots,N_p\}.

In this case, the displacement and acceleration responses at the 6th6^{\mathrm{th}} story at uniformly spaced discrete times are of interest, i.e. y=[u6(t0),u6(t1),…,u6(tns)]\mathbf{y} = [u_6(t_0), u_6(t_1), \ldots, u_6(t_{n_s})] or y=[u¨6(t0),u¨6(t1),…,u¨6(tns)]\mathbf{y} = [\ddot{u}_6(t_0), \ddot{u}_6(t_1), \ldots, \ddot{u}_6(t_{n_s})], which can be computed by using analytical solutions in Appendix A.2.

Nonlinear case

A nonlinear MDOF system with cubic springs is considered, whose response is governed by

Mu¨+Cu˙+Fint(u)=p(t)(7)\mathbf{M}\ddot{\mathbf{u}} + \mathbf{C}\dot{\mathbf{u}} + \mathbf{F}_{\mathrm{int}}(\mathbf{u}) = \mathbf{p}(t) \tag*{(7)}

where the mass matrix M\mathbf{M} and damping matrix C\mathbf{C} keep the same as those in § 2.2.1. The internal force is given by

Fint(u)=[(ku1+bu13)+k(u1−u2)+b(u1−u2)3k(u2−u1)+b(u2−u1)3+k(u2−u3)+b(u2−u3)3k(u3−u2)+b(u3−u2)3+k(u3−u4)+b(u3−u4)3k(u4−u3)+b(u4−u3)3+k(u4−u5)+b(u4−u5)3k(u5−u4)+b(u5−u4)3+k(u5−u6)+b(u5−u6)3k(u6−u5)+b(u6−u5)3](8)\mathbf{F}_{\mathrm{int}}(\mathbf{u}) = \begin{bmatrix} (ku_1 + bu_1^3) + k(u_1-u_2) + b(u_1-u_2)^3 \\ k(u_2-u_1) + b(u_2-u_1)^3 + k(u_2-u_3) + b(u_2-u_3)^3 \\ k(u_3-u_2) + b(u_3-u_2)^3 + k(u_3-u_4) + b(u_3-u_4)^3 \\ k(u_4-u_3) + b(u_4-u_3)^3 + k(u_4-u_5) + b(u_4-u_5)^3 \\ k(u_5-u_4) + b(u_5-u_4)^3 + k(u_5-u_6) + b(u_5-u_6)^3 \\ k(u_6-u_5) + b(u_6-u_5)^3 \end{bmatrix} \tag*{(8)}

where the stiffness coefficient is k=24EI/H3k = 24EI/H^3 and the value of the cubic coefficient bb is chosen from the set {0.25k,0.5k,0.75k,k}\{0.25k, 0.5k, 0.75k, k\}. It is remarked that the nonlinear system in Eq. (7) will degenerate to the linear system in Eq. (4) if the cubic coefficient is chosen as b=0b = 0.

In this case, the load p(t)\mathbf{p}(t) is again applied in terms of harmonic ground motion acceleration ag(t)a_g(t).

The underlying input dataspace for the ground motion acceleration ag(t)a_g(t) in Eq. (6) is considered to be Dnlmdof={ai∼U(0,4),φi∼U(0,2π), and ωi∼U(0,10π) with i=1,2,…,Np}\mathcal{D}_{\mathrm{nlmdof}} = \{a_i \sim\mathcal{U}(0,4), \varphi_i \sim\mathcal{U}(0,2\pi), \text{ and } \omega_i \sim\mathcal{U}(0,10\pi) \text{ with } i = 1,2,\ldots,N_p\}. The same response quantities in § 2.2.2 are of interest and are computed using Newmark-β\beta method [38].

Neural Network Design and Training

In this section, NNs are designed and trained such that they can be used as surrogates to the dynamical systems considered in Section 2. Neural networks use compositions of relatively simple mathematical functions to represent complex mappings. In particular, in many NNs these mathematical operations are carried out in a layer-by-layer fashion [2], where an output of one layer is taken as an input to the next layer. The design of NNs involves determining the architecture of each layer and the number of such data processing layers in the network while training a NN involves determining the network parameters (Θ\boldsymbol{\Theta}) that are used to describe these computational layers.

For the considered dynamical systems, the input of the NN is the loading history given by an n×1n \times1 vector, i.e. x=[p(t0),p(t1),…,p(tns)]\mathbf{x} = [p(t_0), p(t_1), \ldots, p(t_{n_s})] or x=[ag(t0),ag(t1),…,ag(tns)]\mathbf{x} = [a_g(t_0), a_g(t_1), \ldots, a_g(t_{n_s})], which is sampled from the underlying dataspace D\mathcal{D}, where D\mathcal{D} is taken to be Dlsdof\mathcal{D}_{\mathrm{lsdof}} or Dnlsdof\mathcal{D}_{\mathrm{nlsdof}} for linear or nonlinear SDOF problems, and Dlmdof\mathcal{D}_{\mathrm{lmdof}} or Dnlmdof\mathcal{D}_{\mathrm{nlmdof}} for linear or nonlinear MDOF problems. The output of the NN is the response of the system that includes displacement and acceleration time history, e.g., y=[u(0),u(1),…,u(tns)]\mathbf{y} = [u(0), u(1), \ldots, u(t_{n_s})] for the time series of displacement of SDOF. In essence, NN represents a mapping fnnf_{nn} parameterized by Θ\boldsymbol{\Theta} that maps the input to the output, i.e., fnn ⁣:x↦yf_{nn}\colon\mathbf{x} \mapsto\mathbf{y} where x,y∈Rn\mathbf{x}, \mathbf{y} \in\mathbb{R}^{n}. Thus, the task of a network is to predict y\mathbf{y} from x\mathbf{x} by outputting y^=fnn(x;Θ)\hat{\mathbf{y}} = f_{nn}(\mathbf{x}; \boldsymbol{\Theta}) and the goal of machine learning is to learn the parameters Θ\boldsymbol{\Theta}.

From a probabilistic regression viewpoint [39], the predictive conditional PDF p(y∣x)p(\mathbf{y}|\mathbf{x}) reads

p(y∣x)=N(y∣fnn(x;Θ),σ2I)(9)p(\mathbf{y}|\mathbf{x}) = \mathcal{N}(\mathbf{y}|f_{nn}(\mathbf{x};\boldsymbol{\Theta}),\sigma^2\mathbf{I}) \tag*{(9)}

In addition, by assuming that all training data pairs (xi,yi)(\mathbf{x}_i,\mathbf{y}_i) (i=1,2,…,Ntrain)(i = 1, 2, \ldots, N_{\mathrm{train}}) are i.i.d., the joint PDF of all training data points read

p(y1,y2,⋯ ,yNtrain∣x1,x2,⋯ ,xNtrain)=∏i=1Ntrainp(yi∣xi)(10)p(y_1,y_2,\cdots,y_{N_{\mathrm{train}}}\mid x_1,x_2,\cdots,x_{N_{\mathrm{train}}})=\prod_{i=1}^{N_{\mathrm{train}}}p(y_i\mid x_i) \tag*{(10)}

The joint PDF in Eq. (10) can be considered as a measure for quantifying how well the proposed NN model fits the observed training data points. Thus, the value of network parameters Θ\boldsymbol{\Theta} can be obtained by solving the following maximum likelihood estimation (MLE) optimization problem

Θ∗=argmax⁡Θln⁡p(y1,y2,⋯ ,yNtrain∣x1,x2,⋯ ,xNtrain)(11)\boldsymbol{\Theta}^{*}=\underset{\boldsymbol{\Theta}}{\operatorname{argmax}}\ln p(y_1,y_2,\cdots,y_{N_{\mathrm{train}}}\mid x_1,x_2,\cdots,x_{N_{\mathrm{train}}}) \tag*{(11)}

Finally, the learned conditional distribution p(y∣x)p(\boldsymbol{y}\mid\boldsymbol{x}) is N(y∣fnn(x;Θ∗),σ2I)\mathcal{N}(\boldsymbol{y}\mid f_{nn}(\boldsymbol{x};\boldsymbol{\Theta}^{*}),\sigma^2\boldsymbol{I}) and the most probable NN mapping is given by y^=fnn(x;Θ∗)\hat{\boldsymbol{y}}=f_{nn}(\boldsymbol{x};\boldsymbol{\Theta}^{*}).

Fully-connected (FC) neural network

The neural network architectures used to model dynamical systems in this study are shown in Figure 1a. The network starts with an input layer x\boldsymbol{x} of size n×1n\times1 representing the external loading time sequence. The output layer y\boldsymbol{y}, i.e., displacement or acceleration, has the same size as the input layer. For a standard (lthl^{th}) FC layer (Figure 1a), the weighted sum operation is applied to the input vector x[l−1]\boldsymbol{x}^{[l-1]} from the (l−1)th(l-1)^{th} layer using a nH[l]×nH[l−1]n_H^{[l]}\times n_H^{[l-1]} weight matrix W[l]\boldsymbol{W}^{[l]} and nH[l]×1n_H^{[l]}\times1 bias vector b[l]\boldsymbol{b}^{[l]}, where nH[l−1]n_H^{[l-1]} and nH[l]n_H^{[l]} represent the size of the (l−1)th(l-1)^{th} and lthl^{th} layers, respectively. This operation gives

h1[l]=W[l]x[l−1]+b[l](12)\boldsymbol{h}_1^{[l]}=\boldsymbol{W}^{[l]}\boldsymbol{x}^{[l-1]}+\boldsymbol{b}^{[l]} \tag*{(12)}

where h1[l]\boldsymbol{h}_1^{[l]} is a nH[l]×1n_H^{[l]}\times1 vector. After the weighted sum in Eq. (12), a linear (identity) activation or a nonlinear ReLU activation is applied depending on whether the FC layer is the last layer of the network or not. Thus, the output vector of the lthl^{\text{th}} layer reads

x[l]={h1[l]if layer l is the last layerΦ(h1[l])otherwise(13)\mathbf{x}^{[l]} = \begin{cases} \mathbf{h}_{1}^{[l]} & \text{if layer } l \text{ is the last layer} \\ \Phi\left(\mathbf{h}_{1}^{[l]}\right) & \text{otherwise} \end{cases} \tag*{(13)}

where x[l]\mathbf{x}^{[l]} is a nH[l]×1n_{H}^{[l]} \times1 vector and Φ(■)\Phi(\blacksquare) represents an element-wise ReLU activation function, i.e.,

Φ(x)={xif x>00otherwise(14)\Phi(x) = \begin{cases} x & \text{if } x > 0 \\ 0 & \text{otherwise} \end{cases} \tag*{(14)}

Sparse convolutional neural networks

In some cases, the trained FC layers in § 3.1 exhibit a sparsity pattern, i.e., there are many small weights in the layers. Although these small weights help to improve the model performance, they also lead to increased computational cost in terms of both network evaluation and parameter optimization. To make use of the underlying sparsity pattern while keeping the desired network accuracy the convolutional layer enhanced sparse networks are proposed in this study. The overall idea of these sparse networks is to use the sparsity pattern to construct sparse FC layers that are then used together with convolutional layers. The added convolutional layers provide additional capacity that compensates for the sparsity of the FC layer. As the convolutions layers provide parameter sharing, they do not significantly increase the number of trainable parameters.

In Figure 1b, the sparse network architecture contains, between the input and output layers, convolutional (CONV) layers, batch normalization (BN) layers, and sparsely connected (SC) layers. BN layers are added before each CONV layer, except the first CONV layer which is preceded by the input layer that has already been normalized (§ 3.3.1). In the cases where higher representational capacity is required, more batch-convolutional blocks (BN-CONV) that comprise a batch normalization and a convolutional layer are added. In general, increasing the number of BN-CONV layers will increase the overall representational capacity of the network. However, this will also lead to a deeper NN model with more training parameters (θ\boldsymbol{\theta}), and thus may require more sophisticated training strategies. The computational operations in each of these layers are described in the following sections.

Sparsely connected (SC) layer

To construct an SC layer, the NN model with FC layers is first trained. Next, a sparsity analysis on the trained weight matrices W[l]\mathbf{W}^{[l]} (l→l \to index of hidden layers) is carried out with a predefined threshold value such that the weights that are smaller than the threshold are taken as zero to determine the sparsity pattern of weight matrices. The zero weights in the resulting sparse weight matrices are then fixed and are not updated during the training process. As an illustration, Figure 2 shows a weight matrix of one FC layer where the intensity of color denotes the relative magnitudes of the weights. With a pre-defined threshold, only the weights that are larger than the threshold are kept for training, and this results in a sparse representation shown in Figure 2.

Illustration of sparsity pattern

Figure 2. Illustration of sparsity pattern.

Convolutional (CONV) layer

In a CONV layer, padding, convolution, and activation are the main operations that are used to extract data features and add nonlinearities to the architecture. Suppose the lthl^{\mathrm{th}} layer is a CONV layer, padding, convolution, and activation are applied sequentially to the data from the previous layer x[l−1]\mathbf{x}^{[l-1]} of dimension nH[l−1]×nC[l−1]n_{H}^{[l-1]} \times n_{C}^{[l-1]}. For padding with dimension p[l]p^{[l]}, a volume of dimension p[l]×nC[l−1]p^{[l]} \times n_{C}^{[l-1]} containing only zeros is concatenated on two ends of the volume x[l−1]\mathbf{x}^{[l-1]} and the output from this operation is h1[l]=[0;x[l−1];0]\mathbf{h}_1^{[l]} = [\mathbf{0}; \mathbf{x}^{[l-1]}; \mathbf{0}]. The size of the volume h1[l]\mathbf{h}_1^{[l]} is then (2p[l]+nH[l−1])×nC[l−1](2p^{[l]} + n_H^{[l-1]}) \times n_C^{[l-1]}. After padding, convolution operation is applied to volume h1[l]\mathbf{h}_1^{[l]} by using nC[l]n_C^{[l]} filters. The size of each filter is f[l]×nC[l−1]f^{[l]} \times n_C^{[l-1]} and the filter stride used is denoted by s[l]s^{[l]}. In each filter, there are weights Wi[l]\mathbf{W}_i^{[l]} of size f[l]×nC[l−1]f^{[l]} \times n_C^{[l-1]} and scalar biases bi[l]b_i^{[l]} (i=1,2,⋯ ,nC[l]i = 1, 2, \cdots, n_C^{[l]}). Using the iith filter, the convolution operation outputs a vector:

vi=h1[l]∗Wi[l]=[h1[l][1:f[l]]⋆Wi[l]+bi[l]h1[l][s[l]+1:s[l]+f[l]]⋆Wi[l]+bi[l]h1[l][2s[l]+1:2s[l]+f[l]]⋆Wi[l]+bi[l]⋮h1[l][2p[l]+nH[l−1]−f[l]+1:2p[l]+nH[l−1]]⋆Wi[l]+bi[l]](15)\mathbf{v}_i = \mathbf{h}_1^{[l]} * \mathbf{W}_i^{[l]} = \begin{bmatrix} \mathbf{h}_1^{[l]}[1:f^{[l]}] \star\mathbf{W}_i^{[l]} + b_i^{[l]} \\ \mathbf{h}_1^{[l]}[s^{[l]}+1:s^{[l]}+f^{[l]}] \star\mathbf{W}_i^{[l]} + b_i^{[l]} \\ \mathbf{h}_1^{[l]}[2s^{[l]}+1:2s^{[l]}+f^{[l]}] \star\mathbf{W}_i^{[l]} + b_i^{[l]} \\ \vdots\\ \mathbf{h}_1^{[l]}[2p^{[l]}+n_H^{[l-1]}-f^{[l]}+1:2p^{[l]}+n_H^{[l-1]}] \star\mathbf{W}_i^{[l]} + b_i^{[l]} \end{bmatrix} \tag*{(15)}

i=1,2,⋯ ,nC[l]i = 1, 2, \cdots, n_C^{[l]}

with dimension

(2p[l]+nH[l−1]−f[l]s[l]+1)×1(16)\left(\frac{2p^{[l]} + n_H^{[l-1]} - f^{[l]}}{s^{[l]}} + 1\right) \times1 \tag*{(16)}

and for two volumes a\mathbf{a} and b\mathbf{b} of the same size n1×n2n_1 \times n_2, the operation ⋆\star is given by

a⋆b≜∑i=1n1∑j=1n2a[i,j]b[i,j](17)\mathbf{a} \star\mathbf{b} \triangleq\sum_{i=1}^{n_1}\sum_{j=1}^{n_2} a[i,j]b[i,j] \tag*{(17)}

In Eq. (15), h1[l][p:q]\mathbf{h}_1^{[l]}[p:q] means that the volume h1[l]\mathbf{h}_1^{[l]} is sliced along the height from pthp^{\mathrm{th}} to qthq^{\mathrm{th}} rows.

It is noted that the padding size p[l]p^{[l]} and filter stride s[l]s^{[l]} are often chosen such that the size in Eq. (16) is an integer.

Finally, the vectors vi\mathbf{v}_i (i=1,2,⋯ ,nC[l]i = 1, 2, \cdots, n_C^{[l]}) are stacked together to output a new volume

h2[l]=[v1v2⋯vnC[l]⋮⋮⋮]nH[l]×nC[l](18)h_2^{[l]} = \begin{bmatrix} \mathbf{v}_1 & \mathbf{v}_2 & \cdots& \mathbf{v}_{n_C^{[l]}} \\ \vdots& \vdots& & \vdots \end{bmatrix}_{n_H^{[l]} \times n_C^{[l]}} \tag*{(18)}

where

nH[l]=2p[l]+nH[l−1]−f[l]s[l]+1(19)n_H^{[l]} = \frac{2p^{[l]} + n_H^{[l-1]} - f^{[l]}}{s^{[l]}} + 1 \tag*{(19)}

In this study, it is required that the output volume h2[l]h_2^{[l]} has the same height as x[l−1]x^{[l-1]} such that the convolutional layer does not change the height of the input volume. Hence, the padding size is chosen to be

p[l]=12[(nH[l−1]−1)s[l]+f[l]−nH[l−1]](20)p^{[l]} = \frac{1}{2}\left[\left(n_H^{[l-1]} - 1\right)s^{[l]} + f^{[l]} - n_H^{[l-1]}\right] \tag*{(20)}

With this padding, nH[l]=nH[l−1]n_H^{[l]} = n_H^{[l-1]} and the size of the volume h2[l]h_2^{[l]} becomes nH[l−1]×nC[l]n_H^{[l-1]} \times n_C^{[l]}.

To add nonlinearities in the layer, the rectified linear unit (ReLU) [1] is used as an activation function and is applied to each element in the volume h2[l]h_2^{[l]}. As a result, the output volume of this convolutional layer is

x[l]=Φ(h2[l])(21)x^{[l]} = \Phi\left(h_2^{[l]}\right) \tag*{(21)}

where the activation (Φ\Phi) is an element-wise operation and does not change the size of the volume h2[l]h_2^{[l]}. Depending on the location of the CONV layer, linear or ReLU activation is applied. In particular, linear activation is used for the last CONV layer whose functionality is mainly for resizing the data vector to nH[0]×1n_H^{[0]} \times1, while the ReLU activation function is applied to all the other middle CONV layers.

Batch normalization (BN) layer

The BN layer is designed to address the so-called internal covariate drift issue, which refers to a phenomenon that in the training process the distribution of each layer’s input x[l−1]x^{[l-1]} continue to change, as previous CONV or FC layer’s weights W[l−1]W^{[l-1]} and biases b[l−1]b^{[l-1]} get updated. This phenomenon is found to significantly slow down the training [40]. With the BN layer, each input is normalized independently based on the statistics of the samples in the current mini-batch (explanation of mini-batch can be found in § 3.3.3) [40]. This normalization helps to maintain a stable training process.

Consider that the lthl^{\mathrm{th}} layer in the architecture is a BN layer, the input volume x[l−1]x^{[l-1]} is normalized by its expectation and variance. The normalization operation is applied to each element of volume x[l−1]x^{[l-1]} and outputs a new volume x[l]x^{[l]} with the same size as x[l−1]x^{[l-1]}, i.e., nH[l]=nH[l−1]n_{H}^{[l]} = n_{H}^{[l-1]} and nC[l]=nC[l−1]n_{C}^{[l]} = n_{C}^{[l-1]}. The calculation of x[l]x^{[l]} given x[l−1]x^{[l-1]} is

x[i,j][l]=γ[i,j][l]x[i,j][l−1]−E[x[i,j][l−1]]Var⁡[x[i,j][l−1]]+ϵBN+β[i,j][l](22)x_{[i,j]}^{[l]} = \gamma_{[i,j]}^{[l]} \frac{x_{[i,j]}^{[l-1]} - E\left[x_{[i,j]}^{[l-1]}\right]}{\sqrt{\operatorname{Var}\left[x_{[i,j]}^{[l-1]}\right] + \epsilon_{BN}}} + \beta_{[i,j]}^{[l]} \tag*{(22)}

with i=1,2,⋯ ,nH[l−1]i = 1,2,\cdots,n_{H}^{[l-1]} and j=1,2,⋯ ,nC[l−1]j = 1,2,\cdots,n_{C}^{[l-1]}

where x[i,j][l]x_{[i,j]}^{[l]} denotes the element of volume x[l]x^{[l]} with index [i,j][i,j]; γ[l]\gamma^{[l]} and β[l]\beta^{[l]} are matrices of size nH[l−1]×nC[l−1]n_{H}^{[l-1]} \times n_{C}^{[l-1]} that contains all the scaling and shifting parameters; ϵBN\epsilon_{BN} is a hyperparameter used to avoid numerical issues when the variance is close to zero. The shifting and scaling parameters ensure that the transformation in Eq. (22) can represent an identity [40], which means no operation may be done in the BN layer. Assuming there are NsN_s samples in the current minibatch and a collection of these data for the (l−1)th(l-1)^{th} layer, i.e., x[l−1]\mathbf{x}^{[l-1]}, DN={x1[l−1],x2[l−1],…,xNs[l−1]}D_N = \{\mathbf{x}_1^{[l-1]}, \mathbf{x}_2^{[l-1]}, \ldots, \mathbf{x}_{N_s}^{[l-1]}\}, the statistics in Eq. (22) are estimated by

E[x[i,j][l−1]]≈1Ns∑k=1Nsxk[i,j][l−1]Var⁡[x[i,j][l−1]]≈1Ns∑k=1Ns(xk[i,j][l−1]−E[x[i,j][l−1]])2(23)\begin{aligned} E\left[\mathbf{x}^{[l-1]}_{[i,j]}\right] &\approx\frac{1}{N_s}\sum_{k=1}^{N_s}\mathbf{x}^{[l-1]}_{k[i,j]} \\ \operatorname{Var}\left[\mathbf{x}^{[l-1]}_{[i,j]}\right] &\approx\frac{1}{N_s}\sum_{k=1}^{N_s}\left(\mathbf{x}^{[l-1]}_{k[i,j]}-E\left[\mathbf{x}^{[l-1]}_{[i,j]}\right]\right)^2 \tag*{(23)} \end{aligned}

where the square operation is applied in an elementwise fashion in Eq. (23). It should be noted that in the training stage where a mini-batch stochastic gradient algorithm is used, the statistical quantities in Eq. (23) are computed based on a mini-batch training dataset at the current iteration. However, in the testing stage, the statistics in Eq. (23) are computed based on the complete training dataset.

Neural networks training

The neural network models considered in this study are parameterized by parameters Θ={W,b,γ,β}\boldsymbol{\Theta} = \{\mathbf{W},\mathbf{b},\boldsymbol{\gamma},\boldsymbol{\beta}\}, where W\mathbf{W} and b\mathbf{b} are weight matrices and bias vectors from all the CONV and SC layers; γ\boldsymbol{\gamma} and β\boldsymbol{\beta} are scaling and shifting parameters from all the BN layers in the network. The neural networks training refers to the optimization process of finding the optimum Θ∗\boldsymbol{\Theta}^{*} in Eq. (11).

Data preprocessing

Although theoretically not necessary, data (input and output pairs) preprocessing has become a routine step in deep learning, as it can speed up learning and lead to better model performance. There are multiple approaches for data preprocessing [41, 42], and the commonly used normalization strategy is chosen in this study. For each component η\eta of the input as well as output, all the samples in the training dataset (ηi,i=1,…,Ntrain)(\eta_i, i = 1, \ldots, N_{\mathrm{train}}) are normalized by

η~i=ηi−μηVη+ϵD,i=1,2,…,Ntrain(24)\tilde{\eta}_i = \frac{\eta_i-\mu_\eta}{\sqrt{V_\eta+\epsilon_D}}, \qquad i = 1,2,\ldots,N_{\mathrm{train}} \tag*{(24)}

with

μη=1Ntrain∑i=1NtrainηiandVη=1Ntrain∑i=1Ntrain(ηi−μη)2(25)\mu_\eta= \frac{1}{N_{\mathrm{train}}}\sum_{i=1}^{N_{\mathrm{train}}}\eta_i \qquad\text{and} \qquad V_\eta= \frac{1}{N_{\mathrm{train}}}\sum_{i=1}^{N_{\mathrm{train}}}(\eta_i-\mu_\eta)^2 \tag*{(25)}

where ϵD\epsilon_D is added to avoid numerical issues when the value of variance is close to zero. This happens to the first component of output, as all samples follow the same initial condition, i.e., u(t0)=0u(t_0)=0. It should be noted that the testing dataset is normalized by the same statistics of the training dataset, i.e., μη\mu_\eta and VηV_\eta in Eq. (25), before using them as inputs to a network model. The final output is then obtained by carrying out the inverse of the transformation in Eq. (24) in a component-wise manner.

Loss function and error metric

As has been discussed at the beginning of § 3, from a probabilistic viewpoint, the network parameters Θ\boldsymbol{\Theta} can be determined by solving the maximum likelihood estimation (MLE) optimization problem in Eq. (11). Replacing the input/output pairs (xi,yi)(\boldsymbol{x}_i,\boldsymbol{y}_i) with the normalized one (x~i,y~i)(\tilde{\boldsymbol{x}}_i,\tilde{\boldsymbol{y}}_i) in Eqns. (9)-(11), the training task can be completed equivalently by finding optimal Θ\boldsymbol{\Theta} in Rnp\mathbb{R}^{n_p} to minimize the negative log-likelihood function, where npn_p is the number of model parameters, i.e.,

L1(Θ)=−∑i=1Ntrainln⁡p(y~i∣x~i,Θ)=12σ12∑i=1Ntrain∥fnn(x~i;Θ)−y~i∥22+nNtrain2ln⁡2πσ2(26)L_1(\boldsymbol{\Theta})=-\sum_{i=1}^{N_{\mathrm{train}}}\ln p(\tilde{\boldsymbol{y}}_i|\tilde{\boldsymbol{x}}_i,\boldsymbol{\Theta})=\frac{1}{2\sigma_1^2}\sum_{i=1}^{N_{\mathrm{train}}}\left\|f_{nn}(\tilde{\boldsymbol{x}}_i;\boldsymbol{\Theta})-\tilde{\boldsymbol{y}}_i\right\|_2^2+\frac{nN_{\mathrm{train}}}{2}\ln2\pi\sigma^2 \tag*{(26)}

where ∥■∥2\|\blacksquare\|_2 is the Euclidean norm of a vector. Thus, the training task can be further simplified by finding optimal Θ\boldsymbol{\Theta} in Rnp\mathbb{R}^{n_p} to minimize a squared error that reads

L2(Θ)=1Ntrain∑i=1Ntrain∥fnn(x~i;Θ)−y~i∥22(27)L_2(\boldsymbol{\Theta}) = \frac{1}{N_{\mathrm{train}}} \sum_{i=1}^{N_{\mathrm{train}}} \left\| f_{nn}(\tilde{\mathbf{x}}_i;\boldsymbol{\Theta}) - \tilde{\mathbf{y}}_i \right\|_2^2 \tag*{(27)}

Finally, when regularization is also considered, loss function in the learning process in this study is given by

L(Θ)=L2(Θ)+λ(∥W∥F2+∥b∥22)(28)\mathcal{L}(\boldsymbol{\Theta}) = L_2(\boldsymbol{\Theta}) + \lambda\left(\left\|\mathbf{W}\right\|_F^2 + \left\|\mathbf{b}\right\|_2^2\right) \tag*{(28)}

where the first term in the loss function represents the application of likelihood principle over the training dataset (DtrainD_{\mathrm{train}}) and the second term is used to regularize the loss function, in which λ\lambda is a scalar regularization hyperparameter and ∥■∥F\left\|\blacksquare\right\|_F is the Frobenius matrix norm. The second term in the loss function is needed since, without this regularization term, the ML algorithm will always prefer to train a network model that minimizes the observation error over the training dataset regardless of the complexity of the trained model and chances are high that such models do not have good performance over the testing dataset, which is usually referred to as overfitting. By adding a regularization term in the loss function, the optimization algorithm will try to find a balance between the complexity of the trained model and the training error, which helps prevent overfitting issues. Apart from adding L2L^2 norm of parameters in the loss function, there are also other regularization options such as adding L1L^1 norm of parameters, early-stopping, random dropout, and model averaging [2].

Next, the unknown parameters in NN models are learned by solving an optimization problem, i.e.,

Θ∗=argmin⁡Θ∈Rnp L(Θ)(29)\boldsymbol{\Theta}^* = \underset{\boldsymbol{\Theta}\in\mathbb{R}^{n_p}}{\operatorname{argmin}}\ \mathcal{L}(\boldsymbol{\Theta}) \tag*{(29)}

After training, for arbitrary observed input data point x~\tilde{\mathbf{x}}, the predictive PDF of the output data y~\tilde{\mathbf{y}} reads

p(y~∣x~,Θ)=N(y~∣fnn(x~;Θ∗),σ2I)(30)p(\tilde{\mathbf{y}} \mid\tilde{\mathbf{x}}, \boldsymbol{\Theta}) = \mathcal{N}(\tilde{\mathbf{y}} \mid f_{nn}(\tilde{\mathbf{x}}; \boldsymbol{\Theta}^{*}), \sigma^{2} I) \tag*{(30)}

Therefore, from Eq. (30) the mean value of y~\tilde{\mathbf{y}} given by the NN model is y^=fnn(x~;Θ∗)\hat{\mathbf{y}} = f_{nn}(\tilde{\mathbf{x}}; \boldsymbol{\Theta}^{*}).

Finally, the performance (P\mathcal{P}) of a NN model is measured in terms of the mean squared error of the model on the testing dataset using Eq. (27), i.e., Pmse≡L2(Θ)\mathcal{P}_{\mathrm{mse}} \equiv L_{2}(\boldsymbol{\Theta}), with training data replaced with testing data. Note that a low value of Pmse\mathcal{P}_{\mathrm{mse}} indicates a good model performance, and vice versa.

Another measure for the model prediction performance is through relative error metric

εr=∥y−y^∥2∥y∥2×100%(31)\varepsilon_{r} = \frac{\lVert\mathbf{y} - \hat{\mathbf{y}} \rVert_{2}}{\lVert\mathbf{y} \rVert_{2}} \times100\% \tag*{(31)}

Stochastic gradient descent-based Adam optimizer

To solve the optimization problem in Eq. (29), stochastic gradient-descent-based learning algorithms are commonly employed [1], since calculating the true gradient over the entire training dataspace can slow down the training process and lead to prohibitively high memory requirement [2]. In particular, mini-batch stochastic gradient descent is used, where the training dataset DtrainD_{\mathrm{train}} is divided into nbn_b mini-batches, i.e., D1D_1, D2D_2, ..., DnbD_{n_b}. The number of samples in each batch sb=Ntrain/nbs_b = N_{\mathrm{train}}/n_b is called the batch size. All mini-batch datasets are used one by one to compute a stochastic gradient for updating model parameters, and one cycle over all mini-batches is called an epoch. Hence, one epoch consists of nbn_b optimization iterations. Since the gradient computed from the data in one mini-batch might be away from the true gradient over the entire training dataspace and might be very different from batch to batch, it can lead to an oscillating convergence behavior and slow down the training process. To address this issue, Adam optimizer considers moving statistics of the stochastic gradient, where at the kthk^{\text{th}} optimization step in one epoch [43], the update process of an arbitrary trainable variable θ∈Θ\theta\in\boldsymbol{\Theta} is given by

θk=θk−1−αkmk(32)\theta_k = \theta_{k-1} - \alpha_k m_k \tag*{(32)}

where

mk=11−(β1)k[β1mk−1+(1−β1)gk],αk=αvk+ϵl(33)m_k = \frac{1}{1-(\beta_1)^k}\left[\beta_1m_{k-1}+(1-\beta_1)g_k\right], \quad\alpha_k = \frac{\alpha}{\sqrt{v_k}+\epsilon_l} \tag*{(33)}

with vk=11−(β2)k[β2vk−1+(1−β2)(gk)2]v_k = \frac{1}{1-(\beta_2)^k}\left[\beta_2v_{k-1}+(1-\beta_2)(g_k)^2\right] and gk=∂L~Dk(θ)∂θ∣θ=θkg_k = \left.\frac{\partial\widetilde{\mathcal{L}}_{D_k}(\theta)}{\partial\theta}\right|_{\theta=\theta_k}

in which β1\beta_1 and β2\beta_2 represent decay rates for the first and second-order moment estimates of the stochastic gradient with default values being β1=0.9\beta_1 = 0.9 and β2=0.999\beta_2 = 0.999 [43]; mkm_k and vkv_k are the first and second order moment estimates of stochastic gradient and are initialized by m0=v0=0m_0 = v_0 = 0; ϵl=10−8\epsilon_l = 10^{-8} is a small constant for numerical stability purposes; αk\alpha_k is the learning rate at the current step and α\alpha is a hyperparameter for controlling the learning rate. Here, the normalized factor 11−(β1)k\frac{1}{1-(\beta_1)^k} and 11−(β2)k\frac{1}{1-(\beta_2)^k} are used to correct the moment estimates at the beginning of optimization and have little effect as kk becomes large.

Model parameter initialization

All the model parameters have to be initialized to start the model training process, i.e., parameter optimization, see Eq. (32). Unless transfer learning (§ 3.3.5) is used where the values of some model parameters are taken from other trained models, the initialization of the parameters is carried out as follows: (a) scaling and shifting parameters in all BN layers, γ\gamma and β\beta, are initialized to ones and zeros, respectively; (b) all the biases are initialized to zeros; (c) for all weights in CONV and FC layers, Xavier normal initialization method [44] is used, i.e., for any weight from a given neuron, it is randomly initialized from a Gaussian distribution N(0,2Nin)\mathcal{N}\left(0,\frac{2}{N_{\mathrm{in}}}\right), where NinN_{\mathrm{in}} is the number of inputs for the this neuron. It should be noted that in a CONV layer, NinN_{\mathrm{in}} is equal to the filter size. With Xavier method, the variance of NN output is close to the variance of its input, which helps to prevent gradient vanishing/exploding issues at the start of training [44].

Transfer learning

a) From FC to SC layer

To construct a sparse neural network model described in § 3.2, a sparsity analysis is carried out on the trained fully connected model in § 3.1. Therefore, a fully connected model is first trained using the initialization strategy in § 3.3.4 and following the training procedures described above. After a sparsity analysis on the weighting matrices with a pre-selected threshold, the remaining non-zero weights and biases are taken as the initialization of the corresponding parameters in the SC layers of the sparse network model, see Figure 1b. Starting from the basic neural network architecture with two CONV layers and transferred SC layers, the other model parameters in BN and CONV layers are initialized with the same strategy described in § 3.3.4.

b) From shallow to deep CONV layer enriched SC networks

Starting with a basic architecture shown in Figure 3, more BN-CONV layers are added, as needed for a given problem. It is important to note that in the considered architecture, an SC layer has a significantly greater number of parameters as compared to a CONV layer and has a larger representational capacity. Therefore, it does not lead to a significant increase of trainable parameters by adding more BN-CONV layers. It should be remarked that although with Xavier initialization the gradient vanishing/exploding issues can be mitigated, the training performance can be still unstable, i.e., a deeper architecture may have larger training/testing error than a shallower architecture after the same number of training epochs. This is because even though a deeper network has more capacity, it is difficult to train a deeper network via a simple training strategy where all the parameters are simultaneously trained. To improve the training efficiency of deeper networks, the strategy of transfer learning [2] is utilized. The proposed transfer learning procedure for this study is detailed below.

Basic architecture of the CONV enriched sparse NNs

Figure 3. Basic architecture of the CONV enriched sparse NNs.

In the proposed transfer learning strategy, the training starts with a basic NN architecture shown in Figure 3. The BN-CONV layers in the NN architecture are initialized by using the initialization procedure explained in § 3.3.4, together with the transferred SC layer(s). After training, the values of all learned parameters are stored for future use. Suppose that more BN-CONV layer(s) are needed for higher representational capacity, then one CONV-BN layer is added between BN layer 1 and CONV 2, and the new architecture is shown in Figure 4. Using transfer learning, the new CONV layer 1-1 and BN layer 1-1 are trained in two steps. In the first step, all the trainable parameters from layers except the added CONV layer 1-1 and BN layer 1-1 are initialized by the ones stored in the previously trained architecture in Figure 3. The other trainable parameters are initialized in a standard way using Xavier initialization. In the first part of the training process, the parameters in all layers except the new layers are frozen and only the new layer parameters are trained. Thus, in this step, the previous knowledge is transferred to the new deeper architecture and this makes the training task easier by reducing the number of trainable parameters. In the second part of the training, all the layers are freed to continue training from where they stop in the first part of the training. This step is used to search the optimal solutions near the values obtained from the first step training such that better solutions are not missed. When more BN-CONV layers are used, they can be added one by one and trained by using the two-step strategy presented above.

Deep CONV enriched sparse NNs with added BN-CONV layers

Figure 4. Deep CONV enriched sparse NNs with added BN-CONV layers.

In this manner, deeper architectures can effectively be trained to have smaller training errors. The efficacy of this training strategy is illustrated via numerical examples in the next section.

Numerical examples for SDOF systems

All the network training tasks are performed using TensorFlow on a superserver with dual twelve-core Intel Xeon© CPU @ 2.2GHz, 128GB RAM, and an Nvidia Geforce© GTX 1080 Ti GPU.

Dataspace

Five different input dataspaces are considered for the SDOF systems and are shown in Table 2.

Among them, Case-1 to Case-4 are considered for linear SDOF problem (§ 2.1.1), while Case-5 is considered for nonlinear SDOF problem in § 2.1.2. For MDOF systems, the input dataspaces for linear (§ 2.2.1) and nonlinear (§ 2.2.2) springs are considered in Case-6 and Case-7, respectively, in Table 2. Latin-hypercube sampling [45] is used to generate the input samples (x\boldsymbol{x}) from the distributions of loading parameters aia_i, ωi\omega_i and ϕi\phi_i (i=1,2,⋯ ,Npi = 1, 2, \cdots, N_p), as it samples the entire dataspace more uniformly compared to random sampling. Output samples are then generated by solving the linear dynamical systems in Eq. (1) using analytical solutions in Appendix § 8.§ 1 or nonlinear systems in Eq. (3) using the Newmark-β\beta method with a uniform time step size of 0.001 sec, which is tested to be able to give satisfactory accuracy [38]. The dataset that consists of input and output samples is divided into training (NtrainN_{\mathrm{train}}), validation (NvalN_{\mathrm{val}}) and testing (NtestN_{\mathrm{test}}) datasets (Table 2), where validation dataset is used for checking overfitting and determining NN hyperparameters (§ 4.3.2) and testing dataset is used to test the performance of trained NN model for the unseen data, i.e., the prediction/generalization capacity of the model.

Hyperparameter settings

For NN models that employ only FC layers, i.e., nonlinear SDOF and MDOF dynamical systems, in § 4.5 and § 5.2, as well as the SC layers in the sparse NN models, the number of neurons in each hidden layer is chosen to be the same as nn (the size of input/output). For the neural networks that exhibit sparsity pattern, the basic neural network architecture, as shown in Figure 3, comprises four computational layers: one CONV layer with nC[1]n_C^{[1]} filters, one CONV layer with 1 filter, two BN layers and transferred SC layers. More (BN-CONV) layers are added to this architecture to construct deeper networks, as needed. For all CONV layers, the filter size f[l]f^{[l]} is fixed to be 2 and the stride s[l]s^{[l]} is 1. The number of filters nC[l]n_C^{[l]} and the number of BN-CONV layers nln_l are two hyperparameters that are determined for each specific problem. In each BN layer, the hyperparameter ϵBN\epsilon_{BN} is fixed to be 10−810^{-8} for numerical stability.

For all the NN models training, the penalty factor in the loss function λ=10−4\lambda= 10^{-4} is chosen for regularization using grid search from the set {10−6, 10−5, 10−4, 10−3, 10−2, 10−1}\{10^{-6},\ 10^{-5},\ 10^{-4},\ 10^{-3},\ 10^{-2},\ 10^{-1}\}. For numerical stability purposes, the hyperparameter ϵD\epsilon_D is fixed to be 10−810^{-8} in the training data normalization. For Adam optimizer, an optimal hyperparameter α=10−3\alpha= 10^{-3} is chosen based on a grid search from the set {10−5, 10−4, 10−3, 10−2, 10−1}\{10^{-5},\ 10^{-4},\ 10^{-3},\ 10^{-2},\ 10^{-1}\}. All other hyperparameters in Adam optimizer are using recommended values listed in § 3.3.3. When training the NN architecture, the batch size is fixed to 1024 and the number of epochs is fixed to 300 for all cases.

Linear SDOF system with a short loading duration

In this subsection, a short loading duration of T=5T = 5 sec is considered, and this constitutes two cases – Case-1 and Case-2 – in Table 2. The two cases are used for investigating the influence of the number of harmonic frequencies in the loading on the model performance.

Case-1: 5 sec loading with 5 harmonic terms

The number of harmonic terms in the loading is Np=5N_p = 5. The training dataset contains 2182^{18} (= 262144) samples, while the validation dataset composes of 5000 samples. The testing dataset includes 5000 samples. In this case, a single dense layer is first trained to predict displacement or acceleration. The total number of trainable parameters is 10302. The histories of training and validation loss are shown in Figure 5. The corresponding training, validation, and testing mean squared errors are given in Table 3. Second, a sparsity analysis is carried out on the trained FC models, where a threshold value chosen as the 5% of the maximum absolute value of trained weight matrix W\mathbf{W} is used to filter out weights. The sparsity patterns of the FC layers after threshold cutoff are shown in Figure 6. Based on these sparsity patterns, the SC layer is constructed such that all the upper-triangular entries of the weight matrix in the SC layer are frozen to zeros and are not trained.

DisplacementAcceleration
Training error3.0759×10−53.0759 \times10^{-5}3.1964×10−53.1964 \times10^{-5}
Validation error3.1658×10−53.1658 \times10^{-5}3.2958×10−53.2958 \times10^{-5}
Testing error3.3516×10−53.3516 \times10^{-5}3.3524×10−53.3524 \times10^{-5}

Table 3. Training, validation and testing mean squared errors of the trained dense networks for linear SDOF system (Case-1)

Training and validation loss histories for displacement

(a) (a) Displacement

Training and validation loss histories for acceleration

(b) (b) Acceleration

Figure 5. Training and validation loss histories during training process of the dense FC model in Case-1: (a) Displacement; (b) Acceleration.

Sparsity pattern of the weight matrix for displacement

(a) (a) Displacement (#non-zero weights = 3979)

Sparsity pattern of the weight matrix for acceleration

(b) (b) Acceleration (#non-zero weights = 3985)

Figure 6. Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-1: (a) Displacement; (b) Acceleration.

Next, the optimal number (nln_l) of CONV layers and the optimal number of filters (nC[l]n_C^{[l]}) in each CONV layer are investigated. In particular, the optimal number of filters (nC[l]n_C^{[l]}) is selected from the set {1, 4, 8, 16, 32} by training two NN architectures with nl=1n_l = 1 and nl=2n_l = 2 for the displacement and acceleration responses, and the training and testing errors are given in Figure 7 and Figure 8. It can be seen that the errors are reducing as nC[l]n_C^{[l]} increases and the improvement slows down when nC[l]≥16n_C^{[l]} \ge16. Therefore, nC[l]=16n_C^{[l]} = 16 is chosen in all NN architectures in this study. To investigate the optimal nln_l, more CONV layers are continuously added to the model. However, if random parameter initialization is used for deeper architectures, the performance of the trained model can be unstable, see Figure 9(a) where error increases when the number of CONV layers increases. To address this unstable training performance, transfer learning described in § 3.3.5 is used, and accordingly, the training and testing errors for the same architectures are shown in Figure 9(b). With transfer learning, the overall error decreases when more CONV layers are added, and stable training performance is achieved. Thus, transfer learning will be considered in training NN architectures in all the following numerical cases. With transfer learning, the performance of displacement and acceleration models with different numbers of CONV layers is examined in more detail, see Figure 10 where nl=6n_l = 6 leads to an acceptable performance. Detailed information of the deep sparse NN architectures with nl=6n_l = 6 is summarized in Table 4. Compared to the baseline single dense layer network with 10302 trainable parameters, this sparse network contains only 9169 trainable parameters (11.0% reduction). However, the displacement and acceleration testing errors of the sparse architectures (nl=6n_l = 6) are 6.3472×10−66.3472 \times10^{-6} and 8.1845×10−68.1845 \times10^{-6}, respectively, which are smaller than that of the FC layers (3.3516×10−53.3516 \times10^{-5} and 3.3524×10−53.3524 \times10^{-5}). Some predictions of sparse and baseline network models are compared against the analytical solutions in Figure 11, which again shows the competitive prediction accuracy of the sparse model as compared to the baseline dense network model.

Layer type# of trainable variablesActivationLayer type# of trainable variablesActivation
Layer 1Convolution48ReLULayer 8BN502None
Layer 2BN404NoneLayer 9Convolution528ReLU
Layer 3Convolution528ReLULayer 10BN502None
Layer 4BN404NoneLayer 11Convolution528ReLU
Layer 5Convolution528ReLULayer 12BN502None
Layer 6BN404NoneLayer 13Convolution17None
Layer 7Convolution528ReLULayer 14SC5252None
Total # of trainable parameters: 9169
Reduced percentage of trainable parameters compared to dense model: 11.00%

Table 4. Summary of parameters in the sparse network architecture (nl=6n_l = 6) for Case-1 and Case-2

Parametric studies of the number of filters in CONV layers for displacement, with panels (a) $n_l=1$ and (b) $n_l=2$

Figure 7. Parametric studies of the number of filters (nc[l])(n_c^{[l]}) in CONV layers in Case-1 for displacement.

Parametric studies of the number of filters in CONV layers for acceleration, with panels (a) $n_l=1$ and (b) $n_l=2$

Figure 8. Parametric studies of the number of filters (nc[l])(n_c^{[l]}) in CONV layers in Case-1 for acceleration.

Comparison of training strategies using random initialization and transfer learning, with training and testing error bar charts

Figure 9. Comparison of different training strategies on multiple networks of displacement model (Case-1).

Parametric studies of displacement and acceleration for different numbers of CONV layers

Figure 10. Parametric studies on the optimal number (nln_l) of CONV layers in the sparse networks in Case-1: (a) Displacement; (b) Accelerations.

Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-1

Figure 11. Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-1: Left – displacement; Right – acceleration.

Case-2: 5 sec loading with 25 harmonic terms

With other settings fixed to be the same as in Case-1, the number of harmonic terms is increased to Np=25N_p = 25 for Case-2 (Table 2). Following the same procedures described in § 4.3.1, the histories of training and validation errors for the single FC layer model are shown in Figure 12. The sparsity pattern is obtained using the same threshold criterion as in Case-1 and a similar pattern is achieved, see Figure 13. Hence, the SC layer is again constructed with all the upper-triangular entries of the weight matrix in the SC layer frozen to zeros (non-trainable). The transferred SC layer is then combined with nln_l CONV layers (nC[l]=16n_C^{[l]} = 16) and the training and testing errors for different nln_l values are shown in Figure 14a and (b) for displacement and acceleration, respectively. Again, nl=6n_l = 6 is chosen and results in the same NNs as in Table 4. The displacement and acceleration testing errors of sparse NNs (nl=6n_l = 6) are 6.4031×10−66.4031 \times10^{-6} and 6.2719×10−66.2719 \times10^{-6}, respectively, and are smaller than that of the FC model, i.e., 3.1252×10−53.1252 \times10^{-5} and 3.3527×10−53.3527 \times10^{-5}, respectively. Some predictions of sparse and baseline dense network models on testing samples are plotted in Figure 15, which shows similar prediction accuracy of the two models.

Training and validation loss histories for displacement

(a) (a) Displacement

Training and validation loss histories for acceleration

(b) (b) Acceleration

Figure 12. Training and validation loss histories during training process of the FC dense model in Case-2: (a) Displacement; (b) Acceleration.

Sparsity pattern of the weight matrix for displacement

(a) (a) Displacement (#non-zero weights = 3976)

Sparsity pattern of the weight matrix for acceleration

(b) (b) Acceleration (#non-zero weights = 3985)

Figure 13. Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-2: (a) Displacement; (b) Acceleration.

Performance comparison plots for three samples, showing target, dense network, and sparse network displacement and acceleration

Figure 15. Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-2: Left – displacement; Right – acceleration.

Linear SDOF system with a long loading duration

In this section, the performance of the proposed sparse NNs is investigated for a longer loading time duration, i.e., T=20T = 20 sec. With the time step size Δt=0.1\Delta t = 0.1 sec, this results in 201×1201 \times1 input and output vectors, see Case-3 and Case-4 in Table 2.

Case-3: 20 sec loading with 5 harmonic terms

In Case-3 the applied load contains 5 harmonic terms. Following similar procedures in § 4.3.2, a one-FC layer model is first trained with the results – training and validation loss – shown in Figure 16. Next, the sparsity patterns of the trained dense networks are obtained using the same criterion as Case-1 and are shown in Figure 17, which is different from Case-1 and Case-2 due to the changed time step size and loading duration. Based on that, an SC layer with a bandwidth of wb=100w_b = 100 in the lower-triangular part is constructed. With the transferred non-zero weights, the SC layer is then combined with CONV layers as before, and the results for different numbers (nln_l) of added CONV layers are shown in Figure 18a and (b), respectively, for displacement and acceleration predictions. According to the results, nl=9n_l = 9 is chosen as they lead to desirable performance. The details of the final sparse network model are given in Table 5. Compared to the dense models, there is a 42.47% decrease in the number of trainable parameters in the sparse model.

Layer type# of trainable variablesActivationLayer type# of trainable variablesActivation
Layer 1Convolution48ReLULayer 11Convolution
Layer 2BN804NoneLayer 12BN
Layer 3Convolution528ReLULayer 13Convolution
Layer 4BN804NoneLayer 14BN
Layer 5Convolution528ReLULayer 15Convolution
Layer 6BN804NoneLayer 16BN
Layer 7Convolution528ReLULayer 17Convolution
Layer 8BN804NoneLayer 18BN
Layer 9Convolution528ReLULayer 19Convolution
Layer 10BN804NoneLayer 20SC
Total # of trainable parameters: 23359
Reduced percentage of trainable parameters compared to dense model: 42.47%

Table 5. Summary of parameters in the sparse network architecture (nl=9n_l = 9) for Case-3, Case-4, and Case-6

Training and validation loss histories during training process of the FC dense model in Case-3: (a) Displacement; (b) Acceleration.

Figure 16. Training and validation loss histories during training process of the FC dense model in Case-3: (a) Displacement; (b) Acceleration.

Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-3: (a) Displacement; (b) Acceleration.

Figure 17. Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-3: (a) Displacement; (b). Acceleration.

In the meantime, the testing errors of the sparse models are 0.0023 (for displacement) and 0.0025 (for acceleration), which are lower than that of the dense model, i.e., 0.0035 for displacement and 0.0047 for acceleration. Finally, some predictions on the testing samples of sparse and dense models are shown in Figure 19.

Performance comparison plots for three samples, showing displacement on the left and acceleration on the right, with Target, Dense network, and Sparse network curves

Figure 19. Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-3: Left – displacement; Right – acceleration.

Case-4: 20 sec loading with 25 harmonic terms

Compared to Case-3, the number of harmonic terms, in this case, is increased to Np=25N_p = 25 in Case-4. With one FC layer, the dense models for the displacement and acceleration are first trained, see the results in Figure 20, and the trained layers are then used to obtain their sparsity patterns, see Figure 21. Due to the same sparsity pattern as compared to Case-3, the same structure of the SC layer is constructed but with different transferred learned weights from the FC layer. The training results of the transferred SC layer integrated with a different number (nln_l) of CONV layers are given in Figure 22, where it can be seen that nl=9n_l = 9 leads to good accuracy. With nl=9n_l = 9, the resulted sparse NNs is the same as Case-3 of which the details are given in Table 5. This sparse NN leads to an over 42.47% decrease in the number of trainable parameters compared to the dense model. Meanwhile, the testing errors of the sparse model are 0.0024 (displacement) and 0.0025 (acceleration), which are smaller than that of the dense model, i.e. 0.0063 (displacement) and 0.0072 (acceleration). Some prediction samples from the testing dataset by the two models are presented in Figure 23.

Training and validation loss histories for displacement

(a) (a) Displacement

Training and validation loss histories for acceleration

(b) (b) Acceleration

Figure 20. Training and validation loss histories during training process of the FC dense model in Case-4: (a) Displacement; (b) Acceleration.

Sparsity pattern of the weight matrix for displacement

(a) (a) Displacement (#non-zero weights = 8042)

Sparsity pattern of the weight matrix for acceleration

(b) (b) Acceleration (#non-zero weights = 8097)

Figure 21. Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-4: (a) Displacement; (b) Acceleration.

Parametric studies on the optimal number of CONV layers in the sparse networks in Case-4: (a) Displacement; (b) Acceleration

Figure 22. Parametric studies on the optimal number (nln_l) of CONV layers in the sparse networks in Case-4: (a) Displacement; (b) Acceleration.

Performance comparison plots for three random samples

Figure 23. Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-4: Left – displacement; Right – acceleration.

Nonlinear SDOF system – Case-5

Considering long loading duration (20 sec) and 25 harmonic terms in the load, Case-5 is used to study the appropriate NN model for a nonlinear dynamical system. For the nonlinear case considering different values of cubic term coefficient bb, a single FC layer is not enough for accurate prediction. Therefore, a hyperparameter study on the number of FC hidden layers (chosen from the set {1,2,…,7}\{1,2,\ldots,7\}) is carried out and the results are shown in Table 6, which shows the number of FC layers needed to reach the accuracy requirement that the mean relative errors of testing samples for predicting displacement and acceleration are smaller than 10%. Moreover, it is found that the number of FC layers needed to reach a good prediction accuracy increases as the system nonlinearity increases. Several prediction samples of displacement and acceleration by the dense models with an appropriate number of FC layers are shown in Figure 24, Figure 25, Figure 26, and Figure 27, which again confirm the good accuracy of the proposed dense models.

Cubic term coefficient bbNumber of needed FC layersDisplacementAcceleration
0.25k0.25k27.56%7.97%
0.5k0.5k27.74%8.88%
0.75k0.75k47.44%7.92%
kk58.63%8.76%

Table 6. Number of needed FC layers in the dense networks for nonlinear SDOF system (Case-5) to reach acceptable accuracy (mean relative error of testing samples smaller than 10%) and the corresponding mean relative errors

Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient $b = 0.25k$: (a) Sample 1, (b) Sample 2, (c) Sample 3; left displacement and right acceleration

Figure 24. Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient b=0.25kb = 0.25k: Left – displacement; Right – acceleration.

Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient $b = 0.5k$

Figure 25. Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient b=0.5kb = 0.5k: Left – displacement; Right – acceleration.

Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient $b = 0.75k$: three samples, each showing displacement on the left and acceleration on the right

Figure 26. Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient b=0.75kb = 0.75k: Left – displacement; Right – acceleration.

Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient $b = k$

Figure 27. Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient b=kb = k: Left – displacement; Right – acceleration.

The sparsity patterns of the trained dense models are examined with the same criterion in § 4.3.1 and no sparsity patterns are observed for both displacement and acceleration. As a result, no SC layers can be constructed. To leverage the lightweight property of CONV layers to reduce the size of the dense model, the first hidden FC layer is replaced with CONV layers. In the meantime, other layers in the trained dense model are transferred. It should be noted that the strategy of replacing multiple FC layers is not adopted here because it would result in much deeper CONV layers, and thus make the training more expensive. For illustration purposes, the replacement strategy is applied to the most challenging case where the cubic term coefficient b=kb = k. The number (nln_l) of added CONV layers is investigated by a parametric study with transfer learning. With nl=5n_l = 5, the mean relative errors of testing samples are 8.38% for displacement, and 8.65% for acceleration, which are close to that of the dense NN models, i.e., 8.63% for displacement and 8.76% for acceleration. However, the number of trainable parameters in the CONV enriched NNs models is 17.94% less than the original dense NNs models (see Table 7). The comparison of the predictions by the two types of models for several random testing samples is shown in Figure 28, where good prediction accuracy can be observed for both models.

Layer type# of trainable variablesActivationLayer type# of trainable variablesActivation
Layer 1Convolution48ReLULayer 9Convolution528ReLU
Layer 2BN804NoneLayer 10BN804None
Layer 3Convolution528ReLULayer 11Convolution17None
Layer 4BN804NoneLayer 12FC40602ReLU
Layer 5Convolution528ReLULayer 13FC40602ReLU
Layer 6BN804NoneLayer 14FC40602ReLU
Layer 7Convolution528ReLULayer 15FC40602None
Layer 8BN804None
Total # of trainable parameters: 166595
Percentage reduction in trainable parameters compared to dense model: 17.94%

Table 7. Summary of parameters in the CONV layers enriched network architecture (nl=5n_l = 5) for Case-5 and Case-7

Image not extracted

(a) Sample 1

Image not extracted

(b) Sample 2

Image not extracted

(c) Sample 3

Figure 28. Performance comparison of the CONV enriched NN model and the dense NN model on three random samples from testing dataset in Case-5 with cubic coefficient b=kb = k: Left – displacement; Right – acceleration.

Numerical examples for MDOF systems

A six-story multi-degree-of-freedom frame with Rayleigh damping is considered and the model details are given in § 2.2. The two cases, which correspond to linear and cubic nonlinear springs, are considered in this study. In this problem, the input is the ground motion acceleration, and the output is the displacement or acceleration of the sixth degree-of-freedom with observation time step size being 0.1 sec. Detailed input dataset space is given in Table 2. The same hyperparameter settings given in § 4.2 are used.

Case-6: Linear MDOF

The procedures for the CONV enriched sparse NNs training in § 4.3 are also employed in this section. First, a series of FC layer-based dense models with the number of hidden layers from the set {1,2,3,4,5,6,7,8}\{1,2,3,4,5,6,7,8\} are trained. With the harmonic loading frequencies drawn from the range [0,20π][0,20\pi], however, increasing the hidden layers is not to be able to improve the accuracy, and all the trained NNs result in over 10% mean relative errors for both training and testing datasets. It is conjectured that due to the large dataspace, both the NN model complexity and the training datasets size need to be increased substantially to achieve successful training, and this puts a much higher demand on the computational hardware and requires much more computational effort to train.

To solve the MDOF problem with the available computational resources in a reasonable time window, the frequency range [0,20π][0,20\pi] is divided into three parts, i.e., [0,10π][0,10\pi], [10π,15π][10\pi,15\pi] and [15π,20π][15\pi,20\pi]. For each of the three dataspaces, a one-layer dense network is trained for the displacement or acceleration and its sparsity pattern is obtained using the same criterion in § 4. The resulted sparsity patterns are shown in Figure 29, Figure 30, and Figure 31, respectively, for the three frequency ranges. The mean squared testing errors given in Table 8 and the mean relative errors given in Table 9 for the dense model on displacement and acceleration predictions show that with the data range division strategy the model accuracy becomes acceptable (less than 10% mean relative errors). Next, SC layers are constructed based on the sparsity patterns, which are similar to Case-2 and Case-4 in § 4.3.2 and § 4.4.2. After that, the optimal number (nln_l) of CONV layers to be added to SC layers is investigated and the training results with different nln_l’s are plotted in Figure 32, Figure 33, and Figure 34. From the results, nl=9n_l = 9 is chosen for the sparse NNs model, which gives adequate accuracy. The mean squared errors and mean relative errors of sparse NN approximation for displacement and acceleration are given in Table 10 and Table 11. By comparing Table 8 with Table 10 and Table 9 with Table 11, it can be concluded that the proposed sparse NN architecture can be a good candidate for a parsimonious model with competitive accuracy. For different loading frequency ranges, predictions of displacement and acceleration on random loading samples from the testing dataset are compared against the analytical solutions in Figure 35, Figure 36, and Figure 37, where fairly good accuracy can be observed. It is noted that the combined three NNs trained for the three subranges [0,10π][0,10\pi], [10π,15π][10\pi,15\pi] and [15π,20π][15\pi,20\pi] cannot fully fulfill the original task where loading frequencies are drawn randomly from the single range [0,2π][0,2\pi]. This is because the samples that contain frequencies across multiple subranges cannot be predicted using the three NNs.

Loading frequency rangeDisplacementAcceleration
[0,10π][0,10\pi]0.00490.0051
[10π,15π][10\pi,15\pi]0.00200.0094
[15π,20π][15\pi,20\pi]0.00840.0183

Table 8. Mean squared testing errors of the trained dense networks for linear MDOF system (Case-6) with different loading frequency ranges

Loading frequency rangeDisplacementAcceleration
[0,10π][0,10\pi]1.27%1.30%
[10π,15π][10\pi,15\pi]0.98%6.68%
[15π,20π][15\pi,20\pi]6.21%8.96%

Table 9. Mean relative errors over testing dataset of the trained dense networks for linear MDOF system (Case-6) with different loading frequency ranges

Loading frequency rangeDisplacementAcceleration
[0,10π][0,10\pi]0.00510.0064
[10π,15π][10\pi,15\pi]0.00220.0098
[15π,20π][15\pi,20\pi]0.00690.0159

Table 10. Mean squared testing errors of the trained sparse network (nl=9n_l = 9) for MDOF system (Case-6) with different loading frequency ranges

Loading frequency rangeDisplacementAcceleration
[0,10π][0,10\pi]3.45%3.53%
[10π,15π][10\pi,15\pi]1.33%6.84%
[15π,20π][15\pi,20\pi]5.49%7.68%

Table 11. Mean relative errors over testing dataset of the trained sparse network (nl=9n_l = 9) for MDOF system (Case-6) with different loading frequency ranges

Image not extracted

(a) (a) Displacement (#non-zero weights = 5523)

Image not extracted

(b) (b) Acceleration (#non-zero weights = 5661)

Figure 29. Sparsity patterns of the weight matrix of (single) FC layer in the dense models for linear MDOF (Case-6) in the loading frequency range [0,10π][0,10\pi]: (a) Displacement; (b) Acceleration.

Image not extracted

Figure 30. (a) Displacement (#non-zero weights = 3227) (b) Acceleration (#non-zero weights = 3367). Sparsity patterns of the weight matrix of (single) FC layer in the dense models for linear MDOF (Case-6) in the loading frequency range [10,15π][10,15\pi]: (a) Displacement; (b) Acceleration.

Image not extracted

Figure 31. (a) Displacement (#non-zero weights = 2030) (b) Acceleration (#non-zero weights = 2247). Sparsity patterns of the weight matrix of (single) FC layer in the dense models for linear MDOF (Case-6) in the loading frequency range [15π,20π][15\pi,20\pi]: (a) Displacement; (b) Acceleration.

Image not extracted

Figure 32. Parametric studies on the optimal number (nln_l) of CONV layers in the sparse networks for linear MDOF (Case-6) in the loading frequency range [0,10π][0,10\pi]: (a) Displacement; (b) Acceleration.

Image not extracted

Figure 33. Parametric studies on the optimal number (nln_l) of CONV layers in the sparse networks for linear MDOF (Case-6) in the loading frequency range [10π,15π][10\pi,15\pi]: (a) Displacement; (b) Acceleration.

Image not extracted

(a) Displacement

Image not extracted

(b) Acceleration

Figure 34. Parametric studies on the optimal number (nln_l) of CONV layers in the sparse networks for linear MDOF (Case-6) in the loading frequency range [15π,20π][15\pi, 20\pi]: (a) Displacement; (b) Acceleration.

Image not extracted

Figure 35. Performance comparison of the dense and sparse network models on three random samples from testing dataset for linear MDOF (Case-6) in the loading frequency range [0,10π][0,10\pi]: Left – displacement; Right – acceleration.

Image not extracted

Figure 36. Performance comparison of the dense and sparse network models on three random samples from testing dataset for linear MDOF (Case-6) in the loading frequency range [10π,15π][10\pi, 15\pi]: Left – displacement; Right – acceleration.

Image not extracted

Figure 37. Performance comparison of the dense and sparse network models on three random samples from testing dataset for linear MDOF (Case-6) in the loading frequency range [15π,20π][15\pi,20\pi]: Left – displacement; Right – acceleration.

Case-7: Nonlinear MDOF

Since nonlinear spring is considered in this case, a dense network with a single FC layer is unable to predict the responses accurately. Therefore, the hyperparameter study of dense networks in § 4.5 is conducted in this section as well. The number of FC layers needed to reach an acceptable accuracy, i.e., mean relative testing error smaller than 10%, is shown in Table 12 and can be seen to increase with the nonlinearity in the system. Figure 38, Figure 39, Figure 40, and Figure 41 shows several prediction samples of displacement and acceleration for the dynamical system with different nonlinearities and these results confirm the good accuracy of the presented network models.

Cubic term coefficient bbNumber of needed FC layersDisplacementAcceleration
0.25k0.25k26.11%6.31%
0.5k0.5k27.04%7.72%
0.75k0.75k46.41%6.88%
kk56.86%7.00%

Table 12. Number of needed FC layers of the trained dense networks for nonlinear MDOF system (Case-7) to reach the acceptable accuracy (mean relative error of testing samples smaller than 10%) and the corresponding mean relative errors

Image not extracted

Figure 38. Prediction by the dense network of three random samples from testing dataset in Case-7 with cubic coefficient b=0.25kb = 0.25k: Left – displacement; Right – acceleration.

Image not extracted

Figure 39. (a) Sample 1; (b) Sample 2; (c) Sample 3. Prediction by the dense network of three random samples from testing dataset in Case-7 with cubic coefficient b=0.5kb = 0.5k: Left – displacement; Right – acceleration.

Image not extracted

Figure 40. Prediction by the dense network of three random samples from testing dataset in Case-7 with cubic coefficient b=0.75kb = 0.75k: Left – displacement; Right – acceleration.

Image not extracted

Figure 41. Prediction by the dense network of three random samples from testing dataset in Case-7 with cubic coefficient b=kb=k: Left – displacement; Right – acceleration.

By examining the sparsity with the criterion in § 4.3.1, it is found that there is no sparsity in the weight matrices in the dense network models, and therefore no sparse models can be constructed. Again, the same replacement strategy with lightweight CONV layers in § 4.5 is adopted for the MDOF case with the largest nonlinearity, i.e., b=kb = k. Transfer learning is used to investigate the number (nln_l) of CONV layers needed to reach a similar accuracy level as with the original dense layer. With nl=5n_l = 5 CONV layers, the mean relative errors of testing samples are 6.34% for displacement and 6.92% for acceleration, which are close to those from the original dense models, i.e., 6.86% for displacement and 7.00% for acceleration. Moreover, there is again a 17.94% decrease in the number of trainable parameters in the CONV enriched NN model as compared to the original dense model. Several predicted samples from the two NN models are shown in Figure 42, which confirms the competitive performance of the two NN architectures.

Image not extracted

Figure 42. Performance comparison of the CONV enriched NN model and the dense NN model on three random samples from testing dataset in Case-7 with cubic coefficient b=kb=k: Left – displacement; Right – acceleration.

Conclusions

In this study, the use of deep neural network models as surrogates for linear and nonlinear structural dynamical systems is systematically examined. It is shown that for the considered linear SDOF and MDOF systems the fully connected neural network models exhibit sparsity patterns, i.e., some of the network weight parameters are much smaller than others. Moreover, the sparsity patterns can be different for different problem settings and may depend on the loading duration and the number of harmonic terms considered. This sparsity pattern can be utilized to prune a network architecture, which leads to lightweight and more memory-efficient network models. The performance of the sparse networks is further enhanced by adding parameter-efficient convolutional layers. The addition of the convolutional layers does not significantly increase the total number of trainable parameters due to their parameter sharing properties. As a result, the convolutional layer enhanced sparse network models are proposed for linear structural dynamics. The proposed sparse network models have competitive performance and are computationally efficient as compared to the fully connected NNs. Moreover, for training deep NN models, the transfer learning strategy proposed in this study is shown to have a stable training performance, i.e., loss function value continues to decrease as more convolutional layers are added.

For nonlinear dynamical response with cubic nonlinearities, it is shown that more fully connected network layers are needed as the magnitude of the cubic coefficient increases, i.e., a deeper network model is needed to learn a more complex system, as expected. Moreover, in this case, the sparsity patterns in the fully connected networks are not present. To take advantage of the lightweight property of CONV layers, several CONV layers are used to replace the first fully connected layer in the dense model and trained via transfer learning while other layers in the trained dense model are transferred. Compared to the fully connected NN model, the CONV enriched NN models have less trainable parameters while having similar testing performances. Thus, these models can be used as an alternative to fully connected NNs. Finally, in this study, only linear and nonlinear elastic systems are considered under harmonic excitations. Future work will explore the use of similar machine learning techniques for inelastic structural dynamical response prediction under more complex loading scenarios, e.g., seismic excitations and impact/blast loadings.

ACKNOWLEDGMENTS

The presented work is supported in part by the USNational Science Foundation through grant CMMI-1762277. Any opinions, findings, conclusions, and recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the sponsors.

Analytical solutions of linear SDOF and Mdof sYstems

Linear SDOF system

For linear SDOF systems in Eq. (1), the analytical solution of displacement can be expressed as

u(t)=ug(t)+up(t)(34)u(t) = u_g(t) + u_p(t) \tag*{(34)}

where the general solution ug(t)u_g(t) and particular solution up(t)u_p(t) are calculated as

ug(t)=e−ρt(Acos⁡ζt+Bsin⁡ζt)withu_g(t) = e^{-\rho t}(A\cos\zeta t + B\sin\zeta t) \quad\text{with}
ρ=c2mandζ=4mk−c22m\rho= \frac{c}{2m} \quad\text{and} \quad\zeta= \frac{\sqrt{4mk-c^2}}{2m}
A=−∑j=1Npajk−2ζωjωncos⁡ϕj+(1−(ωjωn)2)sin⁡ϕj[1−(ωjωn)2]2+(2ζωjωn)2(35)A = -\sum_{j=1}^{N_p}\frac{a_j}{k}\frac{-2\zeta\frac{\omega_j}{\omega_n}\cos\phi_j+\left(1-\left(\frac{\omega_j}{\omega_n}\right)^2\right)\sin\phi_j}{\left[1-\left(\frac{\omega_j}{\omega_n}\right)^2\right]^2+\left(2\zeta\frac{\omega_j}{\omega_n}\right)^2} \tag*{(35)}
B=1ζ[ρA−∑j=1Npajωjk(1−(ωjωn)2)cos⁡ϕj+2ζωjωnsin⁡ϕj[1−(ωjωn)2]2+(2ζωjωn)2]B = \frac{1}{\zeta}\left[\rho A-\sum_{j=1}^{N_p}\frac{a_j\omega_j}{k}\frac{\left(1-\left(\frac{\omega_j}{\omega_n}\right)^2\right)\cos\phi_j+2\zeta\frac{\omega_j}{\omega_n}\sin\phi_j}{\left[1-\left(\frac{\omega_j}{\omega_n}\right)^2\right]^2+\left(2\zeta\frac{\omega_j}{\omega_n}\right)^2}\right]

and

up(t)=∑j=1Np(ujp1+ujp2)withujp1=Cjsin⁡ωjt+Djcos⁡ωjtandujp2=Ejsin⁡ωjt+Fjcos⁡ωjt(36)\begin{aligned} u_p(t) &= \sum_{j=1}^{N_p}\left(u_j^{p1}+u_j^{p2}\right) \quad\text{with}\\ u_j^{p1} &= C_j\sin\omega_jt+D_j\cos\omega_jt \quad\text{and} \quad u_j^{p2}=E_j\sin\omega_jt+F_j\cos\omega_jt \tag*{(36)} \end{aligned}
Cj=ajcos⁡ϕjk1−(ωjωn)2[1−(ωjωn)2]2+(2ζωjωn)2C_j = \frac{a_j\cos\phi_j}{k}\frac{1-\left(\frac{\omega_j}{\omega_n}\right)^2}{\left[1-\left(\frac{\omega_j}{\omega_n}\right)^2\right]^2+\left(2\zeta\frac{\omega_j}{\omega_n}\right)^2}
Dj=ajcos⁡ϕjk−2ξωjωn[1−(ωjωn)2]2+(2ξωjωn)2D_j=\frac{a_j\cos\phi_j}{k}\frac{-2\xi\frac{\omega_j}{\omega_n}}{\left[1-\left(\frac{\omega_j}{\omega_n}\right)^2\right]^2+\left(2\xi\frac{\omega_j}{\omega_n}\right)^2}
Ej=ajsin⁡ϕjk2ξωjωn[1−(ωjωn)2]2+(2ξωjωn)2E_j=\frac{a_j\sin\phi_j}{k}\frac{2\xi\frac{\omega_j}{\omega_n}}{\left[1-\left(\frac{\omega_j}{\omega_n}\right)^2\right]^2+\left(2\xi\frac{\omega_j}{\omega_n}\right)^2}
Fj=ajsin⁡ϕjk1−(ωjωn)2[1−(ωjωn)2]2+(2ξωjωn)2F_j=\frac{a_j\sin\phi_j}{k}\frac{1-\left(\frac{\omega_j}{\omega_n}\right)^2}{\left[1-\left(\frac{\omega_j}{\omega_n}\right)^2\right]^2+\left(2\xi\frac{\omega_j}{\omega_n}\right)^2}

The acceleration can be computed by taking the second order derivatives of u(t)u(t) in Eq. (A 1), i.e.

u¨(t)=u¨g(t)+u¨p(t)(37)\ddot{u}(t)=\ddot{u}_g(t)+\ddot{u}_p(t) \tag*{(37)}

where

u¨g(t)=ϱ2e−ϱt(Acos⁡ςt+Bsin⁡ςt)−2ϱe−ϱt(−Aςsin⁡ςt+Bςcos⁡ςt)+e−ϱt(−Aς2cos⁡ςt−Bς2sin⁡ςt)(38)\ddot{u}_g(t)=\varrho^2e^{-\varrho t}(A\cos\varsigma t+B\sin\varsigma t)-2\varrho e^{-\varrho t}(-A\varsigma\sin\varsigma t+B\varsigma\cos\varsigma t) +e^{-\varrho t}(-A\varsigma^2\cos\varsigma t-B\varsigma^2\sin\varsigma t) \tag*{(38)}

and

u¨p(t)=∑j=1Np(u¨jp1+u¨jp2)withu¨jp1=−Cjωj2sin⁡(ωjt)−Djωj2cos⁡(ωjt)u¨jp2=−Ejωj2sin⁡(ωjt)−Fjωj2cos⁡(ωjt)(39)\begin{aligned} \ddot{u}_p(t)&=\sum_{j=1}^{N_p}\left(\ddot{u}_j^{p1}+\ddot{u}_j^{p2}\right)\quad\text{with}\\ \ddot{u}_j^{p1}&=-C_j\omega_j^2\sin(\omega_jt)-D_j\omega_j^2\cos(\omega_jt)\\ \ddot{u}_j^{p2}&=-E_j\omega_j^2\sin(\omega_jt)-F_j\omega_j^2\cos(\omega_jt) \tag*{(39)} \end{aligned}

Linear MDOF system

For linear MDOF system in Eq. (4), the analytical solution of displacement given by modal decomposition reads

u(t)=ϕ~q(t)withϕ~=[ϕ~1,ϕ~2,ϕ~3,ϕ~4,ϕ~5,ϕ~6]and\mathbf{u}(t)=\tilde{\boldsymbol{\phi}}\mathbf{q}(t) \quad\text{with} \quad\tilde{\boldsymbol{\phi}}=[\tilde{\boldsymbol{\phi}}_1,\tilde{\boldsymbol{\phi}}_2,\tilde{\boldsymbol{\phi}}_3,\tilde{\boldsymbol{\phi}}_4,\tilde{\boldsymbol{\phi}}_5,\tilde{\boldsymbol{\phi}}_6] \quad\text{and}
q(t)=[q1(t)q2(t)q3(t)q4(t)q5(t)q6(t)]andϕ~i≔1ϕiTMϕiϕi, i=1,2,…6(40)\mathbf{q}(t)=\begin{bmatrix}q_1(t)\\q_2(t)\\q_3(t)\\q_4(t)\\q_5(t)\\q_6(t)\end{bmatrix} \quad\text{and} \quad \tilde{\boldsymbol{\phi}}_i \coloneqq\frac{1}{\sqrt{\boldsymbol{\phi}_i^{T}\mathbf{M}\boldsymbol{\phi}_i}}\boldsymbol{\phi}_i,\ i=1,2,\ldots6 \tag*{(40)}

where (ωni2,ϕi)(\omega_{ni}^{2},\boldsymbol{\phi}_i) are eigenvalue-eigenvector pairs of the matrix M−1K\mathbf{M}^{-1}\mathbf{K} and

qi(t)=qig(t)+qip(t),i=1,2,…6(41)q_i(t)=q_i^{g}(t)+q_i^{p}(t),i=1,2,\ldots6 \tag*{(41)}

with the general solution

qig(t)=e−ϱit(Gicos⁡ςit+Hisin⁡ςit)withq_i^{g}(t)=e^{-\varrho_i t}(G_i\cos\varsigma_i t+H_i\sin\varsigma_i t) \quad\text{with}
Gi=−∑j=1Npγiajωni2(−2ζiωjωnicos⁡ϕj+(1−(ωjωni)2)sin⁡ϕj[1−(ωjωni)2]2+(2ζiωjωni)2)Hi=1ςi[ϱiAi−∑j=1Npγiajωjωni2((1−(ωjωni)2)cos⁡ϕj+2ζiωjωnisin⁡ϕj[1−(ωjωni)2]2+(2ζiωjωni)2)](42)\begin{aligned} G_i={}&-\sum_{j=1}^{N_p}\frac{\gamma_i a_j}{\omega_{ni}^{2}} \left( \frac{-2\zeta_i\frac{\omega_j}{\omega_{ni}}\cos\phi_j+ \left(1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2\right)\sin\phi_j} {\left[1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2\right]^2+ \left(2\zeta_i\frac{\omega_j}{\omega_{ni}}\right)^2} \right)\\ H_i={}&\frac{1}{\varsigma_i}\left[ \varrho_i A_i-\sum_{j=1}^{N_p}\frac{\gamma_i a_j\omega_j}{\omega_{ni}^{2}} \left( \frac{\left(1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2\right)\cos\phi_j+ 2\zeta_i\frac{\omega_j}{\omega_{ni}}\sin\phi_j} {\left[1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2\right]^2+ \left(2\zeta_i\frac{\omega_j}{\omega_{ni}}\right)^2} \right) \right] \tag*{(42)} \end{aligned}
ϱi=α0+α1ωni22andςi=4ωni2−(α0+α1ωni2)22\varrho_i=\frac{\alpha_0+\alpha_1\omega_{ni}^{2}}{2} \quad\text{and} \quad \varsigma_i=\frac{\sqrt{4\omega_{ni}^{2}-(\alpha_0+\alpha_1\omega_{ni}^{2})^2}}{2}
γi=−ϕ~iTdiag⁡(M)andζi=α0+α1ωni22ωni\gamma_i=-\tilde{\boldsymbol{\phi}}_i^{T}\operatorname{diag}(\mathbf{M}) \quad\text{and} \quad \zeta_i=\frac{\alpha_0+\alpha_1\omega_{ni}^{2}}{2\omega_{ni}}

and particular solution

qip(t)=∑j=1Np(qip1j+qip2j)with(43)q_i^{p}(t)=\sum_{j=1}^{N_p}\left(q_i^{p1j}+q_i^{p2j}\right) \quad\text{with} \tag*{(43)}
qip1j(t)=Iisin⁡(ωjt)+Jicos⁡(ωjt)q_i^{p1j}(t) = I_i \sin(\omega_j t) + J_i \cos(\omega_j t)
qip2j(t)=Lisin⁡(ωjt)+Micos⁡(ωjt)q_i^{p2j}(t) = L_i \sin(\omega_j t) + M_i \cos(\omega_j t)
Ii=γiajcos⁡ϕjωni21−(ωjωni)2[1−(ωjωni)2]2+(2ζiωjωni)2I_i = \frac{\gamma_i a_j \cos\phi_j}{\omega_{ni}^{2}} \frac{1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2}{\left[1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2\right]^2+\left(2\zeta_i\frac{\omega_j}{\omega_{ni}}\right)^2}
Ji=γiajcos⁡ϕjωni2−2ζiωjωni[1−(ωjωni)2]2+(2ζiωjωni)2J_i = \frac{\gamma_i a_j \cos\phi_j}{\omega_{ni}^{2}} \frac{-2\zeta_i\frac{\omega_j}{\omega_{ni}}}{\left[1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2\right]^2+\left(2\zeta_i\frac{\omega_j}{\omega_{ni}}\right)^2}
Li=γiajsin⁡ϕjωni22ζiωjωni[1−(ωjωni)2]2+(2ζiωjωni)2L_i = \frac{\gamma_i a_j \sin\phi_j}{\omega_{ni}^{2}} \frac{2\zeta_i\frac{\omega_j}{\omega_{ni}}}{\left[1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2\right]^2+\left(2\zeta_i\frac{\omega_j}{\omega_{ni}}\right)^2}
Mi=γiajsin⁡ϕjωni21−(ωjωni)2[1−(ωjωni)2]2+(2ζiωjωni)2M_i = \frac{\gamma_i a_j \sin\phi_j}{\omega_{ni}^{2}} \frac{1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2}{\left[1-\left(\frac{\omega_j}{\omega_{ni}}\right)^2\right]^2+\left(2\zeta_i\frac{\omega_j}{\omega_{ni}}\right)^2}

Taking the second order derivative of u(t)\mathbf{u}(t) in Eq. (A 7), the analytical solution of acceleration reads

u¨(t)=ϕ~q¨(t)\ddot{\mathbf{u}}(t)=\tilde{\boldsymbol{\phi}}\ddot{\mathbf{q}}(t)
q¨(t)=[q¨1(t)q¨2(t)q¨3(t)q¨4(t)q¨5(t)q¨6(t)](44)\ddot{\mathbf{q}}(t)= \begin{bmatrix} \ddot{q}_1(t)\\ \ddot{q}_2(t)\\ \ddot{q}_3(t)\\ \ddot{q}_4(t)\\ \ddot{q}_5(t)\\ \ddot{q}_6(t) \end{bmatrix} \tag*{(44)}

where

q¨i(t)=q¨ig(t)+q¨ip(t),i=1,2,…,6,with(45)\ddot{q}_i(t)=\ddot{q}_i^g(t)+\ddot{q}_i^p(t), \qquad i=1,2,\ldots,6, \qquad\text{with} \tag*{(45)}
q¨ig(t)=ρi2e−ρit(Gicos⁡ξit+Hisin⁡ξit)−2ρie−ρit(−Giξisin⁡ξit+Hiξicos⁡ξit)+e−ρit(−Giξi2cos⁡ξit−Hiξi2sin⁡ξit)\ddot{q}_i^g(t) = \rho_i^2 e^{-\rho_i t}(G_i \cos\xi_i t + H_i \sin\xi_i t) - 2\rho_i e^{-\rho_i t}(-G_i \xi_i \sin\xi_i t + H_i \xi_i \cos\xi_i t) \\ + e^{-\rho_i t}(-G_i \xi_i^2 \cos\xi_i t - H_i \xi_i^2 \sin\xi_i t)
q¨ip(t)=∑j=1Np(q¨ip1j+q¨ip2j)\ddot{q}_i^p(t) = \sum_{j=1}^{N_p} \left(\ddot{q}_i^{p1j} + \ddot{q}_i^{p2j}\right)
q¨ip1j=−Iiωj2sin⁡(ωjt)−Jiωj2cos⁡(ωjt)\ddot{q}_i^{p1j} = -I_i\omega_j^2\sin(\omega_j t) - J_i\omega_j^2\cos(\omega_j t)
q¨ip2j=−Liωj2sin⁡(ωjt)−Miωj2cos⁡(ωjt)\ddot{q}_i^{p2j} = -L_i\omega_j^2\sin(\omega_j t) - M_i\omega_j^2\cos(\omega_j t)
Displacement mean squared error versus number of layers

(a) Displacement

Acceleration mean squared error versus number of layers

(b) Acceleration

Figure 14. Parametric studies on the optimal number (nln_l) of CONV layers in the sparse networks in Case-2: (a) Displacement; (b) Acceleration.

Displacement bar chart

(a) (a) Displacement

Acceleration bar chart

(b) (b) Acceleration

Figure 18. Parametric studies on the optimal number (nln_l) of CONV layers in the sparse networks in Case-3: (a). Displacement; (b) Acceleration.

References

  1. [1]C. Aggarwal, Neural Networks and Deep Learning. A Textbook, in, Cham: Springer International Publishing, 2018.
  2. [2]I. Goodfellow, Y. Bengio, A. Courville, Deep learning, MIT press, 2016.
  3. [3]M. Längkvist, L. Karlsson, A. Loutfi, A review of unsupervised feature learning and deep learning for time-series modeling, Pattern Recognition Letters, 42 (2014) 11-24.DOI
  4. [4]J. Schmidhuber, Deep learning in neural networks: An overview, Neural networks, 61 (2015) 85-117.
  5. [5]A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, 2012, pp. 1097-1105.
  6. [6]M.D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, in: European conference on computer vision, Springer, 2014, pp. 818-833.arxiv.org/abs/1311.2901
  7. [7]O. Abdel-Hamid, A. Mohamed, H. Jiang, L. Deng, G. Penn, D. Yu, Convolutional Neural Networks for Speech Recognition, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22 (2014) 1533-1545.
  8. [8]W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang, A. Stolcke, The Microsoft 2017 Conversational Speech Recognition System, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5934-5938.DOI
  9. [9]Y. Wu, M. Schuster, Z. Chen, Q.V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, Google's neural machine translation system: Bridging the gap between human and machine translation, arXiv preprint arXiv:1609.08144, (2016).arxiv.org/abs/1609.08144
  10. [10]J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, (2018).arxiv.org/abs/1810.04805
  11. [11]Y. Kim, Convolutional neural networks for sentence classification, arXiv preprint arXiv:1408.5882, (2014).arxiv.org/abs/1408.5882
  12. [12]A. Conneau, H. Schwenk, L. Barrault, Y. Lecun, Very deep convolutional networks for natural language processing, arXiv preprint arXiv:1606.01781, 2 (2016).
  13. [13]H. Sun, H.V. Burton, H. Huang, Machine learning applications for building structural design and performance assessment: State-of-the-art review, Journal of Building Engineering, 33 (2021) 101816.
  14. [14]J. Xue, Z. Xiang, G. Ou, Predicting single freestanding transmission tower time history response during complex wind input through a convolutional neural network based surrogate model, Engineering Structures, 233 (2021) 111859.
  15. [15]S. Mukherjee, D. Lu, B. Raghavan, P. Breitkopf, S. Dutta, M. Xiao, W. Zhang, Accelerating Large-scale Topology Optimization: State-of-the-Art and Challenges, Archives of Computational Methods in Engineering, (2021).DOI
  16. [16]X. Liu, F. Tao, W. Yu, A neural network enhanced system for learning nonlinear constitutive law and failure initiation criterion of composites using indirectly measurable data, Composite Structures, 252 (2020) 112658.DOI
  17. [17]A. Koeppe, F. Bamer, B. Markert, An intelligent nonlinear meta element for elastoplastic continua: deep learning using a new Time-distributed Residual U-Net architecture, Computer Methods in Applied Mechanics and Engineering, 366 (2020) 113088.DOI
  18. [18]Z. Xiang, J. Chen, Y. Bao, H. Li, An active learning method combining deep neural network and weighted sampling for structural reliability analysis, Mechanical Systems and Signal Processing, 140 (2020) 106684.
  19. [19]M. Li, Z. Wang, Deep learning for high-dimensional reliability analysis, Mechanical Systems and Signal Processing, 139 (2020) 106399.DOI
  20. [20]K. Morfidis, K. Kostinakis, Approaches to the rapid seismic damage prediction of r/c buildings using artificial neural networks, Engineering Structures, 165 (2018) 120-141.DOI
  21. [21]S. Naito, H. Tomozawa, Y. Mori, T. Nagata, N. Momma, H. Nakamura, H. Fujiwara, G. Shoji, Building-damage detection method based on machine learning utilizing aerial photographs of the Kumamoto earthquake, Earthquake Spectra, 36 (2020) 1166-1187.DOI
  22. [22]G. Fan, J. Li, H. Hao, Y. Xin, Data driven structural dynamic response reconstruction using segment based generative adversarial networks, Engineering Structures, 234 (2021) 111970.
  23. [23]T.A. Horton, I. Hajirasouliha, B. Davison, Z. Ozdemir, Accurate prediction of cyclic hysteresis behaviour of RBS connections using Deep Learning Neural Networks, Engineering Structures, 247 (2021) 113156.
  24. [24]N.A. Kallioras, G. Kazakis, N.D. Lagaros, Accelerated topology optimization by means of deep learning, Structural and Multidisciplinary Optimization, 62 (2020) 1185-1212.DOI
  25. [25]D.W. Abueidda, S. Koric, N.A. Sobh, Topology optimization of 2D structures with nonlinearities using deep learning, Computers & Structures, 237 (2020) 106283.DOI
  26. [26]L.C. Nguyen, H. Nguyen-Xuan, Deep learning for computational structural optimization, ISA Transactions, 103 (2020) 177-191.DOI
  27. [27]S. Masri, M. Nakamura, A. Chassiakos, T. Caughey, Neural network approach to detection of changes in structural parameters, Journal of engineering mechanics, 122 (1996) 350-360.DOI
  28. [28]D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning representations by back-propagating errors, nature, 323 (1986) 533-536.
  29. [29]R.-T. Wu, M.R. Jahanshahi, Deep convolutional neural network for structural dynamic response estimation and system identification, Journal of Engineering Mechanics, 145 (2019) 04018125.
  30. [30]R. Zhang, Z. Chen, S. Chen, J. Zheng, O. Büyüköztürk, H. Sun, Deep long short-term memory networks for nonlinear structural seismic response prediction, Computers & Structures, 220 (2019) 55-68.
  31. [31]M. Stoffel, R. Gulakala, F. Bamer, B. Markert, Artificial neural networks in structural dynamics: A new modular radial basis function approach vs. convolutional and feedforward topologies, Computer Methods in Applied Mechanics and Engineering, 364 (2020) 112989.DOI
  32. [32]M. Stoffel, F. Bamer, B. Markert, Deep convolutional neural networks in structural dynamics under consideration of viscoplastic material behaviour, Mechanics Research Communications, 108 (2020) 103565.DOI
  33. [33]Y. Yu, H. Yao, Y. Liu, Structural dynamics simulation using a novel physics-guided machine learning method, Engineering Applications of Artificial Intelligence, 96 (2020) 103947.
  34. [34]R. Zhang, Y. Liu, H. Sun, Physics-guided convolutional neural network (PhyCNN) for data-driven seismic response modeling, Engineering Structures, 215 (2020) 110704.arxiv.org/abs/1909.08118
  35. [35]R. Zhang, Y. Liu, H. Sun, Physics-Informed Multi-LSTM Networks for Metamodeling of Nonlinear Structures, arXiv preprint arXiv:2002.10253, (2020).DOI
  36. [36]S. Sadeghi Eshkevari, M. Takáč, S.N. Pakzad, M. Jahani, DynNet: Physics-based neural architecture design for nonlinear structural response modeling and prediction, Engineering Structures, 229 (2021) 111582.
  37. [37]C.M. Bishop, Pattern recognition, Machine learning, 128 (2006).
  38. [38]T.J. Hughes, The finite element method: linear static and dynamic finite element analysis, Courier Corporation, 2012.
  39. [39]K.P. Murphy, Machine learning: a probabilistic perspective, MIT press, 2012.
  40. [40]S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:1502.03167, (2015).arxiv.org/abs/1502.03167
  41. [41]J. Grus, Data science from scratch: first principles with python, O'Reilly Media, 2019.
  42. [42]D. Tax, R. Duin, Feature scaling in support vector data descriptions, Learning from Imbalanced Datasets, (2000) 25-30.
  43. [43]D.P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, (2014).arxiv.org/abs/1412.6980
  44. [44]K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026-1034.arxiv.org/abs/1502.01852
  45. [45]D.P. Kroese, T. Taimre, Z.I. Botev, Handbook of monte carlo methods, John Wiley & Sons, 2013.

Paper details

vlnc:W_4zfp6cu2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents