WorksComputer Science
On the Application of Data-Driven Deep Neural Networks in Linear and Nonlinear Structural Dynamics
Abstract
The use of deep neural network (DNN) models as surrogates for linear and nonlinear structural dynamical systems is explored. The goal is to develop DNN based surrogates to predict structural response, i.e., displacements and accelerations, for given input (harmonic) excitations. In particular, the focus is on the development of efficient network architectures using fully-connected, sparsely-connected, and convolutional network layers, and on the corresponding training strategies that can provide a balance between the overall network complexity and prediction accuracy in the target dataspaces. For linear dynamics, sparsity patterns of the weight matrix in the network layers are used to construct convolutional DNNs with sparse layers. For nonlinear dynamics, it is shown that sparsity in network layers is lost, and efficient DNNs architectures with fully-connected and convolutional network layers are explored. A transfer learning strategy is also introduced to successfully train the proposed DNNs, and various loading factors that influence the network architectures are studied. It is shown that the proposed DNNs can be used as effective and accurate surrogates for predicting linear and nonlinear dynamical responses under harmonic loadings.
Introduction
In recent years, deep neural network (DNN) based models have received considerable attention due to the rapid development in computational hardware and accessibility of big data [1, 2, 3, 4]. With these developments, DNN models are now widely used in applications ranging from image classification [5, 6], speech recognition [7, 8], machine translation systems [9, 10], natural language processing [11, 12], among others. In parallel, there is also an increasing interest in applying DNNs in structural analysis [13, 14] and design [15], e.g., utilization of DNN as surrogates for structural static response [16, 17], failure probability prediction [18, 19], damage detection [20, 21, 22], parameter estimation of hysteretic structural systems [23] and structural topology optimization [24, 25, 26], among others.
In structural dynamics applications, which is the focus of this paper, early attempts of using neural network models can be traced back to the work by Masri et al. [27] where a feedforward neural network (FFNN) was trained to predict the restoring force given structural displacement and velocity at specified discrete-time instants for single-degree-of-freedom (SDOF) systems. In their study, an FFNN architecture with two hidden layers was chosen and network training was performed via a back-propagation algorithm [28]. With the advancements in learning algorithms for training deep network architectures and the availability of graphical processing units (GPUs) for data processing on large datasets [2], there is a renewed interest in the application of neural networks in structural dynamics applications. For instance, Wu et al. [29] used neural networks to predict the response of dynamical systems under earthquake loading using FFNN and convolutional neural network (CNN) models. Zhang et al. [30] applied long short-term memory (LSTM) recurrent networks to predict the structural response of a nonlinear multiple-degree-of-freedom (MDOF) system under seismic excitation. Stoffel et al. [31, 32] carried out a comparative study on the performance of different types of neural networks for approximating the deflection of an inelastic plate under an out-of-plane impulse load. Moreover, Yu et al. [33] proposed a residual recurrent neural network architecture to predict the structural responses of linear and nonlinear systems under sinusoidal or earthquake excitations. In the framework of so-called physics-informed DNNs, Zhang et al. [34] proposed to use a combination of both the standard mean squared error (MSE) of the output data and a physics loss term to train a CNN model taking ground motion acceleration as input and structural response variables as output. The physics loss term consists of an MSE of velocity computed via CNN prediction and finite difference based on the predicted displacement, and the MSE of a constraint that forces CNN outputs to satisfy the underlying governing equations. The authors later [35] used a similar loss function to train several self-tailored LSTM networks for predicting displacement, velocity, and restoring force of nonlinear structures considering a hysteresis model. In another study, Eshkevari et al. [36] designed a recurrent neural network to predict the response quantities including displacement, velocity, internal force, and acceleration for nonlinear systems subjected to ground motion acceleration.
Previous studies have shown the potential of DNNs in structural dynamics applications and many neural networks and training strategies have been presented to solve various problems. However, this is still a nascent field, and many issues have to be addressed before these networks can be successfully used in applications. In particular, in the context of SDOF and MDOF dynamical systems, two related issues must be addressed: a) design of effective network architectures, and b) construction and specification of dataspaces in which the network models are valid. For instance, in many cases, the networks are designed on an ad-hoc basis with many network layers and there is a lack of a systematic exploration of network architectures. With this architecture design strategy, efficient architectures with fewer training parameters may be overlooked. This is an important issue, as it is possible to choose a complex network architecture for a simple target system, which even with regularization is more likely to lead to overfitting [37]. Moreover, there is an increased computational overhead with large networks, as storage, training, and evaluation of large networks is expensive due to the involved floating-point operations. To address this issue, network architectures should be determined in accordance with a balance between model complexity and prediction accuracy. More importantly, when neural network models are used as data-driven surrogates, their validity is only guaranteed within the dataspaces in which they are designed and trained. For instance, a larger target design dataspace might require a deeper network as compared to a smaller dataspace. However, the characterization of the underlying modeling dataspaces is often missing, and therefore, the range of applicability of the trained networks is not clear.
In this exploratory study, DNN models for linear and nonlinear structural dynamics analysis are systematically investigated. The focus is on the construction of efficient network architectures using fully connected, sparsely connected, and convolutional network layers, and on the corresponding training strategies that can provide a balance between the overall network complexity and prediction accuracy. Considering a linear combination of harmonic functions, the input dataspace for a DNN model construction and training is explicitly expressed within which the trained DNN model can be applied. In particular, for linear structural dynamic responses, sparsity patterns of the fully connected layers in the network models are discovered. These sparsity patterns are then used to construct sparsely connected layers to reduce the trainable model parameters, and to compensate for the sparsity parameter-efficient convolutional layers are introduced. For nonlinear dynamics, it is shown that sparsity in network layers is lost, and efficient DNNs architectures with fully connected and convolutional network layers are explored. A transfer learning strategy is also presented to train the proposed DNNs, and various factors that influence the network architectures such as the duration, frequency range, number of frequencies in the load, and nonlinearity of the structure are studied. The rest of the paper is organized as follows: § 2 describes the structural dynamics problems considered in this study. § 3 presents the network architecture and training strategies. In § 4, the proposed network architectures and training methodology are applied to linear and nonlinear SDOF dynamical systems considering harmonic loading with different parameter ranges. In § 5, the application of DNNs is extended to linear and nonlinear MDOF systems. Finally, conclusions and important remarks are given in § 6.
Dynamical Systems Description
In this study, neural networks (NNs) are used to predict the response of a single degree of freedom (SDOF) and multiple degrees of freedom (MDOF) dynamical systems subjected to harmonic loadings that are randomly sampled from prescribed dataspaces. Here, appropriate dataspaces are chosen according to the available hardware and computing resources, as larger dataspaces require more memory and training time to train the networks to reach a satisfactory performance. These dynamical systems and the corresponding dataspaces are described in the following sections.
SDOF system
Linear case
For a linear SDOF system, the equation of motion reads
where , and are mass, stiffness and damping coefficient, respectively; is displacement dependent on time and is the applied load. The following parameters are used to model the SDOF system: mass kg, fundamental period of system sec and natural frequency of the system rad/sec, and the damping ratio is taken as . Thus, the stiffness is kg/sec and the damping coefficient is kg/sec. The initial conditions are and .
The input to this system is the loading , and the response quantities of interest are the displacement and acceleration within a prescribed time window . The harmonic loading is modeled as a linear combination of harmonic components and is given by
where is the number of harmonic terms considered; , and are the amplitudes, frequencies, and phase angles in the harmonic terms (), respectively, that are modeled as uniformly distributed random variables with probability densities given by , and . Thus, the underlying input dataspace can be described by .
The response quantities at uniformly spaced discrete times () are of interest, where () with and is the total number of time steps. Responses are represented by vectors as or , where . For a given input , the response of the dynamical system in Eq. (1) can be computed using analytical solutions provided in Appendix § 8.§ 1.
Nonlinear case
For a nonlinear SDOF system with a cubic spring, the equation of motion reads
where , , are taken the same values as those in § 2.1.1. The coefficient in the cubic term is chosen from the set .
The underlying input dataspace for the harmonic loading in Eq. (2) is considered to be
The response quantities of interest in this case are still displacement or acceleration at discrete times and the response of the dynamical system in Eq. (3) is computed using Newmark- method [38].
MDOF system
Linear case
For the linear elastic MDOF case, a dynamical system with six degrees of freedom is considered. This system is used to model a six-story shear frame with Rayleigh damping and is governed by
where represents displacement vector and and are the 1st-order and 2nd-order time derivative of representing velocity and acceleration vectors, respectively, and
where () are the story mass; , and are Young’s modulus, moment of inertia and story height, respectively; is the damping ratio for the system, and and are the first and second natural frequencies of the system, respectively. The values of these model parameters are summarized in Table 1.
In this case, the load is applied in terms of harmonic ground motion acceleration, , and is modeled as the sum of sinusoidal components as
where , and are amplitudes, frequencies, and phases of the harmonic terms that follow uniform distributions, i.e., , and , respectively. The range of the harmonic frequencies is chosen such that the first two natural frequencies are covered. Thus, the underlying input dataspace can be described by and with .
In this case, the displacement and acceleration responses at the story at uniformly spaced discrete times are of interest, i.e. or , which can be computed by using analytical solutions in Appendix A.2.
Nonlinear case
A nonlinear MDOF system with cubic springs is considered, whose response is governed by
where the mass matrix and damping matrix keep the same as those in § 2.2.1. The internal force is given by
where the stiffness coefficient is and the value of the cubic coefficient is chosen from the set . It is remarked that the nonlinear system in Eq. (7) will degenerate to the linear system in Eq. (4) if the cubic coefficient is chosen as .
In this case, the load is again applied in terms of harmonic ground motion acceleration .
The underlying input dataspace for the ground motion acceleration in Eq. (6) is considered to be . The same response quantities in § 2.2.2 are of interest and are computed using Newmark- method [38].
Neural Network Design and Training
In this section, NNs are designed and trained such that they can be used as surrogates to the dynamical systems considered in Section 2. Neural networks use compositions of relatively simple mathematical functions to represent complex mappings. In particular, in many NNs these mathematical operations are carried out in a layer-by-layer fashion [2], where an output of one layer is taken as an input to the next layer. The design of NNs involves determining the architecture of each layer and the number of such data processing layers in the network while training a NN involves determining the network parameters () that are used to describe these computational layers.
For the considered dynamical systems, the input of the NN is the loading history given by an vector, i.e. or , which is sampled from the underlying dataspace , where is taken to be or for linear or nonlinear SDOF problems, and or for linear or nonlinear MDOF problems. The output of the NN is the response of the system that includes displacement and acceleration time history, e.g., for the time series of displacement of SDOF. In essence, NN represents a mapping parameterized by that maps the input to the output, i.e., where . Thus, the task of a network is to predict from by outputting and the goal of machine learning is to learn the parameters .
From a probabilistic regression viewpoint [39], the predictive conditional PDF reads
In addition, by assuming that all training data pairs are i.i.d., the joint PDF of all training data points read
The joint PDF in Eq. (10) can be considered as a measure for quantifying how well the proposed NN model fits the observed training data points. Thus, the value of network parameters can be obtained by solving the following maximum likelihood estimation (MLE) optimization problem
Finally, the learned conditional distribution is and the most probable NN mapping is given by .
Fully-connected (FC) neural network
The neural network architectures used to model dynamical systems in this study are shown in Figure 1a. The network starts with an input layer of size representing the external loading time sequence. The output layer , i.e., displacement or acceleration, has the same size as the input layer. For a standard () FC layer (Figure 1a), the weighted sum operation is applied to the input vector from the layer using a weight matrix and bias vector , where and represent the size of the and layers, respectively. This operation gives
where is a vector. After the weighted sum in Eq. (12), a linear (identity) activation or a nonlinear ReLU activation is applied depending on whether the FC layer is the last layer of the network or not. Thus, the output vector of the layer reads
where is a vector and represents an element-wise ReLU activation function, i.e.,
Sparse convolutional neural networks
In some cases, the trained FC layers in § 3.1 exhibit a sparsity pattern, i.e., there are many small weights in the layers. Although these small weights help to improve the model performance, they also lead to increased computational cost in terms of both network evaluation and parameter optimization. To make use of the underlying sparsity pattern while keeping the desired network accuracy the convolutional layer enhanced sparse networks are proposed in this study. The overall idea of these sparse networks is to use the sparsity pattern to construct sparse FC layers that are then used together with convolutional layers. The added convolutional layers provide additional capacity that compensates for the sparsity of the FC layer. As the convolutions layers provide parameter sharing, they do not significantly increase the number of trainable parameters.
In Figure 1b, the sparse network architecture contains, between the input and output layers, convolutional (CONV) layers, batch normalization (BN) layers, and sparsely connected (SC) layers. BN layers are added before each CONV layer, except the first CONV layer which is preceded by the input layer that has already been normalized (§ 3.3.1). In the cases where higher representational capacity is required, more batch-convolutional blocks (BN-CONV) that comprise a batch normalization and a convolutional layer are added. In general, increasing the number of BN-CONV layers will increase the overall representational capacity of the network. However, this will also lead to a deeper NN model with more training parameters (), and thus may require more sophisticated training strategies. The computational operations in each of these layers are described in the following sections.
Sparsely connected (SC) layer
To construct an SC layer, the NN model with FC layers is first trained. Next, a sparsity analysis on the trained weight matrices ( index of hidden layers) is carried out with a predefined threshold value such that the weights that are smaller than the threshold are taken as zero to determine the sparsity pattern of weight matrices. The zero weights in the resulting sparse weight matrices are then fixed and are not updated during the training process. As an illustration, Figure 2 shows a weight matrix of one FC layer where the intensity of color denotes the relative magnitudes of the weights. With a pre-defined threshold, only the weights that are larger than the threshold are kept for training, and this results in a sparse representation shown in Figure 2.

Figure 2. Illustration of sparsity pattern.
Convolutional (CONV) layer
In a CONV layer, padding, convolution, and activation are the main operations that are used to extract data features and add nonlinearities to the architecture. Suppose the layer is a CONV layer, padding, convolution, and activation are applied sequentially to the data from the previous layer of dimension . For padding with dimension , a volume of dimension containing only zeros is concatenated on two ends of the volume and the output from this operation is . The size of the volume is then . After padding, convolution operation is applied to volume by using filters. The size of each filter is and the filter stride used is denoted by . In each filter, there are weights of size and scalar biases (). Using the th filter, the convolution operation outputs a vector:
with dimension
and for two volumes and of the same size , the operation is given by
In Eq. (15), means that the volume is sliced along the height from to rows.
It is noted that the padding size and filter stride are often chosen such that the size in Eq. (16) is an integer.
Finally, the vectors () are stacked together to output a new volume
where
In this study, it is required that the output volume has the same height as such that the convolutional layer does not change the height of the input volume. Hence, the padding size is chosen to be
With this padding, and the size of the volume becomes .
To add nonlinearities in the layer, the rectified linear unit (ReLU) [1] is used as an activation function and is applied to each element in the volume . As a result, the output volume of this convolutional layer is
where the activation () is an element-wise operation and does not change the size of the volume . Depending on the location of the CONV layer, linear or ReLU activation is applied. In particular, linear activation is used for the last CONV layer whose functionality is mainly for resizing the data vector to , while the ReLU activation function is applied to all the other middle CONV layers.
Batch normalization (BN) layer
The BN layer is designed to address the so-called internal covariate drift issue, which refers to a phenomenon that in the training process the distribution of each layer’s input continue to change, as previous CONV or FC layer’s weights and biases get updated. This phenomenon is found to significantly slow down the training [40]. With the BN layer, each input is normalized independently based on the statistics of the samples in the current mini-batch (explanation of mini-batch can be found in § 3.3.3) [40]. This normalization helps to maintain a stable training process.
Consider that the layer in the architecture is a BN layer, the input volume is normalized by its expectation and variance. The normalization operation is applied to each element of volume and outputs a new volume with the same size as , i.e., and . The calculation of given is
with and
where denotes the element of volume with index ; and are matrices of size that contains all the scaling and shifting parameters; is a hyperparameter used to avoid numerical issues when the variance is close to zero. The shifting and scaling parameters ensure that the transformation in Eq. (22) can represent an identity [40], which means no operation may be done in the BN layer. Assuming there are samples in the current minibatch and a collection of these data for the layer, i.e., , , the statistics in Eq. (22) are estimated by
where the square operation is applied in an elementwise fashion in Eq. (23). It should be noted that in the training stage where a mini-batch stochastic gradient algorithm is used, the statistical quantities in Eq. (23) are computed based on a mini-batch training dataset at the current iteration. However, in the testing stage, the statistics in Eq. (23) are computed based on the complete training dataset.
Neural networks training
The neural network models considered in this study are parameterized by parameters , where and are weight matrices and bias vectors from all the CONV and SC layers; and are scaling and shifting parameters from all the BN layers in the network. The neural networks training refers to the optimization process of finding the optimum in Eq. (11).
Data preprocessing
Although theoretically not necessary, data (input and output pairs) preprocessing has become a routine step in deep learning, as it can speed up learning and lead to better model performance. There are multiple approaches for data preprocessing [41, 42], and the commonly used normalization strategy is chosen in this study. For each component of the input as well as output, all the samples in the training dataset are normalized by
with
where is added to avoid numerical issues when the value of variance is close to zero. This happens to the first component of output, as all samples follow the same initial condition, i.e., . It should be noted that the testing dataset is normalized by the same statistics of the training dataset, i.e., and in Eq. (25), before using them as inputs to a network model. The final output is then obtained by carrying out the inverse of the transformation in Eq. (24) in a component-wise manner.
Loss function and error metric
As has been discussed at the beginning of § 3, from a probabilistic viewpoint, the network parameters can be determined by solving the maximum likelihood estimation (MLE) optimization problem in Eq. (11). Replacing the input/output pairs with the normalized one in Eqns. (9)-(11), the training task can be completed equivalently by finding optimal in to minimize the negative log-likelihood function, where is the number of model parameters, i.e.,
where is the Euclidean norm of a vector. Thus, the training task can be further simplified by finding optimal in to minimize a squared error that reads
Finally, when regularization is also considered, loss function in the learning process in this study is given by
where the first term in the loss function represents the application of likelihood principle over the training dataset () and the second term is used to regularize the loss function, in which is a scalar regularization hyperparameter and is the Frobenius matrix norm. The second term in the loss function is needed since, without this regularization term, the ML algorithm will always prefer to train a network model that minimizes the observation error over the training dataset regardless of the complexity of the trained model and chances are high that such models do not have good performance over the testing dataset, which is usually referred to as overfitting. By adding a regularization term in the loss function, the optimization algorithm will try to find a balance between the complexity of the trained model and the training error, which helps prevent overfitting issues. Apart from adding norm of parameters in the loss function, there are also other regularization options such as adding norm of parameters, early-stopping, random dropout, and model averaging [2].
Next, the unknown parameters in NN models are learned by solving an optimization problem, i.e.,
After training, for arbitrary observed input data point , the predictive PDF of the output data reads
Therefore, from Eq. (30) the mean value of given by the NN model is .
Finally, the performance () of a NN model is measured in terms of the mean squared error of the model on the testing dataset using Eq. (27), i.e., , with training data replaced with testing data. Note that a low value of indicates a good model performance, and vice versa.
Another measure for the model prediction performance is through relative error metric
Stochastic gradient descent-based Adam optimizer
To solve the optimization problem in Eq. (29), stochastic gradient-descent-based learning algorithms are commonly employed [1], since calculating the true gradient over the entire training dataspace can slow down the training process and lead to prohibitively high memory requirement [2]. In particular, mini-batch stochastic gradient descent is used, where the training dataset is divided into mini-batches, i.e., , , ..., . The number of samples in each batch is called the batch size. All mini-batch datasets are used one by one to compute a stochastic gradient for updating model parameters, and one cycle over all mini-batches is called an epoch. Hence, one epoch consists of optimization iterations. Since the gradient computed from the data in one mini-batch might be away from the true gradient over the entire training dataspace and might be very different from batch to batch, it can lead to an oscillating convergence behavior and slow down the training process. To address this issue, Adam optimizer considers moving statistics of the stochastic gradient, where at the optimization step in one epoch [43], the update process of an arbitrary trainable variable is given by
where
with and
in which and represent decay rates for the first and second-order moment estimates of the stochastic gradient with default values being and [43]; and are the first and second order moment estimates of stochastic gradient and are initialized by ; is a small constant for numerical stability purposes; is the learning rate at the current step and is a hyperparameter for controlling the learning rate. Here, the normalized factor and are used to correct the moment estimates at the beginning of optimization and have little effect as becomes large.
Model parameter initialization
All the model parameters have to be initialized to start the model training process, i.e., parameter optimization, see Eq. (32). Unless transfer learning (§ 3.3.5) is used where the values of some model parameters are taken from other trained models, the initialization of the parameters is carried out as follows: (a) scaling and shifting parameters in all BN layers, and , are initialized to ones and zeros, respectively; (b) all the biases are initialized to zeros; (c) for all weights in CONV and FC layers, Xavier normal initialization method [44] is used, i.e., for any weight from a given neuron, it is randomly initialized from a Gaussian distribution , where is the number of inputs for the this neuron. It should be noted that in a CONV layer, is equal to the filter size. With Xavier method, the variance of NN output is close to the variance of its input, which helps to prevent gradient vanishing/exploding issues at the start of training [44].
Transfer learning
a) From FC to SC layer
To construct a sparse neural network model described in § 3.2, a sparsity analysis is carried out on the trained fully connected model in § 3.1. Therefore, a fully connected model is first trained using the initialization strategy in § 3.3.4 and following the training procedures described above. After a sparsity analysis on the weighting matrices with a pre-selected threshold, the remaining non-zero weights and biases are taken as the initialization of the corresponding parameters in the SC layers of the sparse network model, see Figure 1b. Starting from the basic neural network architecture with two CONV layers and transferred SC layers, the other model parameters in BN and CONV layers are initialized with the same strategy described in § 3.3.4.
b) From shallow to deep CONV layer enriched SC networks
Starting with a basic architecture shown in Figure 3, more BN-CONV layers are added, as needed for a given problem. It is important to note that in the considered architecture, an SC layer has a significantly greater number of parameters as compared to a CONV layer and has a larger representational capacity. Therefore, it does not lead to a significant increase of trainable parameters by adding more BN-CONV layers. It should be remarked that although with Xavier initialization the gradient vanishing/exploding issues can be mitigated, the training performance can be still unstable, i.e., a deeper architecture may have larger training/testing error than a shallower architecture after the same number of training epochs. This is because even though a deeper network has more capacity, it is difficult to train a deeper network via a simple training strategy where all the parameters are simultaneously trained. To improve the training efficiency of deeper networks, the strategy of transfer learning [2] is utilized. The proposed transfer learning procedure for this study is detailed below.

Figure 3. Basic architecture of the CONV enriched sparse NNs.
In the proposed transfer learning strategy, the training starts with a basic NN architecture shown in Figure 3. The BN-CONV layers in the NN architecture are initialized by using the initialization procedure explained in § 3.3.4, together with the transferred SC layer(s). After training, the values of all learned parameters are stored for future use. Suppose that more BN-CONV layer(s) are needed for higher representational capacity, then one CONV-BN layer is added between BN layer 1 and CONV 2, and the new architecture is shown in Figure 4. Using transfer learning, the new CONV layer 1-1 and BN layer 1-1 are trained in two steps. In the first step, all the trainable parameters from layers except the added CONV layer 1-1 and BN layer 1-1 are initialized by the ones stored in the previously trained architecture in Figure 3. The other trainable parameters are initialized in a standard way using Xavier initialization. In the first part of the training process, the parameters in all layers except the new layers are frozen and only the new layer parameters are trained. Thus, in this step, the previous knowledge is transferred to the new deeper architecture and this makes the training task easier by reducing the number of trainable parameters. In the second part of the training, all the layers are freed to continue training from where they stop in the first part of the training. This step is used to search the optimal solutions near the values obtained from the first step training such that better solutions are not missed. When more BN-CONV layers are used, they can be added one by one and trained by using the two-step strategy presented above.

Figure 4. Deep CONV enriched sparse NNs with added BN-CONV layers.
In this manner, deeper architectures can effectively be trained to have smaller training errors. The efficacy of this training strategy is illustrated via numerical examples in the next section.
Numerical examples for SDOF systems
All the network training tasks are performed using TensorFlow on a superserver with dual twelve-core Intel Xeon© CPU @ 2.2GHz, 128GB RAM, and an Nvidia Geforce© GTX 1080 Ti GPU.
Dataspace
Five different input dataspaces are considered for the SDOF systems and are shown in Table 2.
Among them, Case-1 to Case-4 are considered for linear SDOF problem (§ 2.1.1), while Case-5 is considered for nonlinear SDOF problem in § 2.1.2. For MDOF systems, the input dataspaces for linear (§ 2.2.1) and nonlinear (§ 2.2.2) springs are considered in Case-6 and Case-7, respectively, in Table 2. Latin-hypercube sampling [45] is used to generate the input samples () from the distributions of loading parameters , and (), as it samples the entire dataspace more uniformly compared to random sampling. Output samples are then generated by solving the linear dynamical systems in Eq. (1) using analytical solutions in Appendix § 8.§ 1 or nonlinear systems in Eq. (3) using the Newmark- method with a uniform time step size of 0.001 sec, which is tested to be able to give satisfactory accuracy [38]. The dataset that consists of input and output samples is divided into training (), validation () and testing () datasets (Table 2), where validation dataset is used for checking overfitting and determining NN hyperparameters (§ 4.3.2) and testing dataset is used to test the performance of trained NN model for the unseen data, i.e., the prediction/generalization capacity of the model.
Hyperparameter settings
For NN models that employ only FC layers, i.e., nonlinear SDOF and MDOF dynamical systems, in § 4.5 and § 5.2, as well as the SC layers in the sparse NN models, the number of neurons in each hidden layer is chosen to be the same as (the size of input/output). For the neural networks that exhibit sparsity pattern, the basic neural network architecture, as shown in Figure 3, comprises four computational layers: one CONV layer with filters, one CONV layer with 1 filter, two BN layers and transferred SC layers. More (BN-CONV) layers are added to this architecture to construct deeper networks, as needed. For all CONV layers, the filter size is fixed to be 2 and the stride is 1. The number of filters and the number of BN-CONV layers are two hyperparameters that are determined for each specific problem. In each BN layer, the hyperparameter is fixed to be for numerical stability.
For all the NN models training, the penalty factor in the loss function is chosen for regularization using grid search from the set . For numerical stability purposes, the hyperparameter is fixed to be in the training data normalization. For Adam optimizer, an optimal hyperparameter is chosen based on a grid search from the set . All other hyperparameters in Adam optimizer are using recommended values listed in § 3.3.3. When training the NN architecture, the batch size is fixed to 1024 and the number of epochs is fixed to 300 for all cases.
Linear SDOF system with a short loading duration
In this subsection, a short loading duration of sec is considered, and this constitutes two cases – Case-1 and Case-2 – in Table 2. The two cases are used for investigating the influence of the number of harmonic frequencies in the loading on the model performance.
Case-1: 5 sec loading with 5 harmonic terms
The number of harmonic terms in the loading is . The training dataset contains (= 262144) samples, while the validation dataset composes of 5000 samples. The testing dataset includes 5000 samples. In this case, a single dense layer is first trained to predict displacement or acceleration. The total number of trainable parameters is 10302. The histories of training and validation loss are shown in Figure 5. The corresponding training, validation, and testing mean squared errors are given in Table 3. Second, a sparsity analysis is carried out on the trained FC models, where a threshold value chosen as the 5% of the maximum absolute value of trained weight matrix is used to filter out weights. The sparsity patterns of the FC layers after threshold cutoff are shown in Figure 6. Based on these sparsity patterns, the SC layer is constructed such that all the upper-triangular entries of the weight matrix in the SC layer are frozen to zeros and are not trained.
| Displacement | Acceleration | |
| Training error | ||
| Validation error | ||
| Testing error |
Table 3. Training, validation and testing mean squared errors of the trained dense networks for linear SDOF system (Case-1)

(a) (a) Displacement

(b) (b) Acceleration
Figure 5. Training and validation loss histories during training process of the dense FC model in Case-1: (a) Displacement; (b) Acceleration.

(a) (a) Displacement (#non-zero weights = 3979)

(b) (b) Acceleration (#non-zero weights = 3985)
Figure 6. Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-1: (a) Displacement; (b) Acceleration.
Next, the optimal number () of CONV layers and the optimal number of filters () in each CONV layer are investigated. In particular, the optimal number of filters () is selected from the set {1, 4, 8, 16, 32} by training two NN architectures with and for the displacement and acceleration responses, and the training and testing errors are given in Figure 7 and Figure 8. It can be seen that the errors are reducing as increases and the improvement slows down when . Therefore, is chosen in all NN architectures in this study. To investigate the optimal , more CONV layers are continuously added to the model. However, if random parameter initialization is used for deeper architectures, the performance of the trained model can be unstable, see Figure 9(a) where error increases when the number of CONV layers increases. To address this unstable training performance, transfer learning described in § 3.3.5 is used, and accordingly, the training and testing errors for the same architectures are shown in Figure 9(b). With transfer learning, the overall error decreases when more CONV layers are added, and stable training performance is achieved. Thus, transfer learning will be considered in training NN architectures in all the following numerical cases. With transfer learning, the performance of displacement and acceleration models with different numbers of CONV layers is examined in more detail, see Figure 10 where leads to an acceptable performance. Detailed information of the deep sparse NN architectures with is summarized in Table 4. Compared to the baseline single dense layer network with 10302 trainable parameters, this sparse network contains only 9169 trainable parameters (11.0% reduction). However, the displacement and acceleration testing errors of the sparse architectures () are and , respectively, which are smaller than that of the FC layers ( and ). Some predictions of sparse and baseline network models are compared against the analytical solutions in Figure 11, which again shows the competitive prediction accuracy of the sparse model as compared to the baseline dense network model.
| Layer type | # of trainable variables | Activation | Layer type | # of trainable variables | Activation | ||
| Layer 1 | Convolution | 48 | ReLU | Layer 8 | BN | 502 | None |
| Layer 2 | BN | 404 | None | Layer 9 | Convolution | 528 | ReLU |
| Layer 3 | Convolution | 528 | ReLU | Layer 10 | BN | 502 | None |
| Layer 4 | BN | 404 | None | Layer 11 | Convolution | 528 | ReLU |
| Layer 5 | Convolution | 528 | ReLU | Layer 12 | BN | 502 | None |
| Layer 6 | BN | 404 | None | Layer 13 | Convolution | 17 | None |
| Layer 7 | Convolution | 528 | ReLU | Layer 14 | SC | 5252 | None |
| Total # of trainable parameters: 9169 | |||||||
| Reduced percentage of trainable parameters compared to dense model: 11.00% |
Table 4. Summary of parameters in the sparse network architecture () for Case-1 and Case-2

Figure 7. Parametric studies of the number of filters in CONV layers in Case-1 for displacement.

Figure 8. Parametric studies of the number of filters in CONV layers in Case-1 for acceleration.

Figure 9. Comparison of different training strategies on multiple networks of displacement model (Case-1).

Figure 10. Parametric studies on the optimal number () of CONV layers in the sparse networks in Case-1: (a) Displacement; (b) Accelerations.

Figure 11. Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-1: Left – displacement; Right – acceleration.
Case-2: 5 sec loading with 25 harmonic terms
With other settings fixed to be the same as in Case-1, the number of harmonic terms is increased to for Case-2 (Table 2). Following the same procedures described in § 4.3.1, the histories of training and validation errors for the single FC layer model are shown in Figure 12. The sparsity pattern is obtained using the same threshold criterion as in Case-1 and a similar pattern is achieved, see Figure 13. Hence, the SC layer is again constructed with all the upper-triangular entries of the weight matrix in the SC layer frozen to zeros (non-trainable). The transferred SC layer is then combined with CONV layers () and the training and testing errors for different values are shown in Figure 14a and (b) for displacement and acceleration, respectively. Again, is chosen and results in the same NNs as in Table 4. The displacement and acceleration testing errors of sparse NNs () are and , respectively, and are smaller than that of the FC model, i.e., and , respectively. Some predictions of sparse and baseline dense network models on testing samples are plotted in Figure 15, which shows similar prediction accuracy of the two models.

(a) (a) Displacement

(b) (b) Acceleration
Figure 12. Training and validation loss histories during training process of the FC dense model in Case-2: (a) Displacement; (b) Acceleration.

(a) (a) Displacement (#non-zero weights = 3976)

(b) (b) Acceleration (#non-zero weights = 3985)
Figure 13. Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-2: (a) Displacement; (b) Acceleration.

Figure 15. Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-2: Left – displacement; Right – acceleration.
Linear SDOF system with a long loading duration
In this section, the performance of the proposed sparse NNs is investigated for a longer loading time duration, i.e., sec. With the time step size sec, this results in input and output vectors, see Case-3 and Case-4 in Table 2.
Case-3: 20 sec loading with 5 harmonic terms
In Case-3 the applied load contains 5 harmonic terms. Following similar procedures in § 4.3.2, a one-FC layer model is first trained with the results – training and validation loss – shown in Figure 16. Next, the sparsity patterns of the trained dense networks are obtained using the same criterion as Case-1 and are shown in Figure 17, which is different from Case-1 and Case-2 due to the changed time step size and loading duration. Based on that, an SC layer with a bandwidth of in the lower-triangular part is constructed. With the transferred non-zero weights, the SC layer is then combined with CONV layers as before, and the results for different numbers () of added CONV layers are shown in Figure 18a and (b), respectively, for displacement and acceleration predictions. According to the results, is chosen as they lead to desirable performance. The details of the final sparse network model are given in Table 5. Compared to the dense models, there is a 42.47% decrease in the number of trainable parameters in the sparse model.
| Layer type | # of trainable variables | Activation | Layer type | # of trainable variables | Activation |
| Layer 1 | Convolution | 48 | ReLU | Layer 11 | Convolution |
| Layer 2 | BN | 804 | None | Layer 12 | BN |
| Layer 3 | Convolution | 528 | ReLU | Layer 13 | Convolution |
| Layer 4 | BN | 804 | None | Layer 14 | BN |
| Layer 5 | Convolution | 528 | ReLU | Layer 15 | Convolution |
| Layer 6 | BN | 804 | None | Layer 16 | BN |
| Layer 7 | Convolution | 528 | ReLU | Layer 17 | Convolution |
| Layer 8 | BN | 804 | None | Layer 18 | BN |
| Layer 9 | Convolution | 528 | ReLU | Layer 19 | Convolution |
| Layer 10 | BN | 804 | None | Layer 20 | SC |
| Total # of trainable parameters: 23359 | |||||
| Reduced percentage of trainable parameters compared to dense model: 42.47% |
Table 5. Summary of parameters in the sparse network architecture () for Case-3, Case-4, and Case-6

Figure 16. Training and validation loss histories during training process of the FC dense model in Case-3: (a) Displacement; (b) Acceleration.

Figure 17. Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-3: (a) Displacement; (b). Acceleration.
In the meantime, the testing errors of the sparse models are 0.0023 (for displacement) and 0.0025 (for acceleration), which are lower than that of the dense model, i.e., 0.0035 for displacement and 0.0047 for acceleration. Finally, some predictions on the testing samples of sparse and dense models are shown in Figure 19.

Figure 19. Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-3: Left – displacement; Right – acceleration.
Case-4: 20 sec loading with 25 harmonic terms
Compared to Case-3, the number of harmonic terms, in this case, is increased to in Case-4. With one FC layer, the dense models for the displacement and acceleration are first trained, see the results in Figure 20, and the trained layers are then used to obtain their sparsity patterns, see Figure 21. Due to the same sparsity pattern as compared to Case-3, the same structure of the SC layer is constructed but with different transferred learned weights from the FC layer. The training results of the transferred SC layer integrated with a different number () of CONV layers are given in Figure 22, where it can be seen that leads to good accuracy. With , the resulted sparse NNs is the same as Case-3 of which the details are given in Table 5. This sparse NN leads to an over 42.47% decrease in the number of trainable parameters compared to the dense model. Meanwhile, the testing errors of the sparse model are 0.0024 (displacement) and 0.0025 (acceleration), which are smaller than that of the dense model, i.e. 0.0063 (displacement) and 0.0072 (acceleration). Some prediction samples from the testing dataset by the two models are presented in Figure 23.

(a) (a) Displacement

(b) (b) Acceleration
Figure 20. Training and validation loss histories during training process of the FC dense model in Case-4: (a) Displacement; (b) Acceleration.

(a) (a) Displacement (#non-zero weights = 8042)

(b) (b) Acceleration (#non-zero weights = 8097)
Figure 21. Sparsity patterns of the weight matrix of (single) FC layer in the dense models in Case-4: (a) Displacement; (b) Acceleration.

Figure 22. Parametric studies on the optimal number () of CONV layers in the sparse networks in Case-4: (a) Displacement; (b) Acceleration.

Figure 23. Performance comparison of the dense and sparse network models on three random samples from testing dataset in Case-4: Left – displacement; Right – acceleration.
Nonlinear SDOF system – Case-5
Considering long loading duration (20 sec) and 25 harmonic terms in the load, Case-5 is used to study the appropriate NN model for a nonlinear dynamical system. For the nonlinear case considering different values of cubic term coefficient , a single FC layer is not enough for accurate prediction. Therefore, a hyperparameter study on the number of FC hidden layers (chosen from the set ) is carried out and the results are shown in Table 6, which shows the number of FC layers needed to reach the accuracy requirement that the mean relative errors of testing samples for predicting displacement and acceleration are smaller than 10%. Moreover, it is found that the number of FC layers needed to reach a good prediction accuracy increases as the system nonlinearity increases. Several prediction samples of displacement and acceleration by the dense models with an appropriate number of FC layers are shown in Figure 24, Figure 25, Figure 26, and Figure 27, which again confirm the good accuracy of the proposed dense models.
| Cubic term coefficient | Number of needed FC layers | Displacement | Acceleration |
| 2 | 7.56% | 7.97% | |
| 2 | 7.74% | 8.88% | |
| 4 | 7.44% | 7.92% | |
| 5 | 8.63% | 8.76% |
Table 6. Number of needed FC layers in the dense networks for nonlinear SDOF system (Case-5) to reach acceptable accuracy (mean relative error of testing samples smaller than 10%) and the corresponding mean relative errors

Figure 24. Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient : Left – displacement; Right – acceleration.

Figure 25. Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient : Left – displacement; Right – acceleration.

Figure 26. Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient : Left – displacement; Right – acceleration.

Figure 27. Prediction by the dense network of three random samples from testing dataset in Case-5 with cubic coefficient : Left – displacement; Right – acceleration.
The sparsity patterns of the trained dense models are examined with the same criterion in § 4.3.1 and no sparsity patterns are observed for both displacement and acceleration. As a result, no SC layers can be constructed. To leverage the lightweight property of CONV layers to reduce the size of the dense model, the first hidden FC layer is replaced with CONV layers. In the meantime, other layers in the trained dense model are transferred. It should be noted that the strategy of replacing multiple FC layers is not adopted here because it would result in much deeper CONV layers, and thus make the training more expensive. For illustration purposes, the replacement strategy is applied to the most challenging case where the cubic term coefficient . The number () of added CONV layers is investigated by a parametric study with transfer learning. With , the mean relative errors of testing samples are 8.38% for displacement, and 8.65% for acceleration, which are close to that of the dense NN models, i.e., 8.63% for displacement and 8.76% for acceleration. However, the number of trainable parameters in the CONV enriched NNs models is 17.94% less than the original dense NNs models (see Table 7). The comparison of the predictions by the two types of models for several random testing samples is shown in Figure 28, where good prediction accuracy can be observed for both models.
| Layer type | # of trainable variables | Activation | Layer type | # of trainable variables | Activation | ||
| Layer 1 | Convolution | 48 | ReLU | Layer 9 | Convolution | 528 | ReLU |
| Layer 2 | BN | 804 | None | Layer 10 | BN | 804 | None |
| Layer 3 | Convolution | 528 | ReLU | Layer 11 | Convolution | 17 | None |
| Layer 4 | BN | 804 | None | Layer 12 | FC | 40602 | ReLU |
| Layer 5 | Convolution | 528 | ReLU | Layer 13 | FC | 40602 | ReLU |
| Layer 6 | BN | 804 | None | Layer 14 | FC | 40602 | ReLU |
| Layer 7 | Convolution | 528 | ReLU | Layer 15 | FC | 40602 | None |
| Layer 8 | BN | 804 | None | ||||
| Total # of trainable parameters: 166595 | |||||||
| Percentage reduction in trainable parameters compared to dense model: 17.94% |
Table 7. Summary of parameters in the CONV layers enriched network architecture () for Case-5 and Case-7
(a) Sample 1
(b) Sample 2
(c) Sample 3
Figure 28. Performance comparison of the CONV enriched NN model and the dense NN model on three random samples from testing dataset in Case-5 with cubic coefficient : Left – displacement; Right – acceleration.
Numerical examples for MDOF systems
A six-story multi-degree-of-freedom frame with Rayleigh damping is considered and the model details are given in § 2.2. The two cases, which correspond to linear and cubic nonlinear springs, are considered in this study. In this problem, the input is the ground motion acceleration, and the output is the displacement or acceleration of the sixth degree-of-freedom with observation time step size being 0.1 sec. Detailed input dataset space is given in Table 2. The same hyperparameter settings given in § 4.2 are used.
Case-6: Linear MDOF
The procedures for the CONV enriched sparse NNs training in § 4.3 are also employed in this section. First, a series of FC layer-based dense models with the number of hidden layers from the set are trained. With the harmonic loading frequencies drawn from the range , however, increasing the hidden layers is not to be able to improve the accuracy, and all the trained NNs result in over 10% mean relative errors for both training and testing datasets. It is conjectured that due to the large dataspace, both the NN model complexity and the training datasets size need to be increased substantially to achieve successful training, and this puts a much higher demand on the computational hardware and requires much more computational effort to train.
To solve the MDOF problem with the available computational resources in a reasonable time window, the frequency range is divided into three parts, i.e., , and . For each of the three dataspaces, a one-layer dense network is trained for the displacement or acceleration and its sparsity pattern is obtained using the same criterion in § 4. The resulted sparsity patterns are shown in Figure 29, Figure 30, and Figure 31, respectively, for the three frequency ranges. The mean squared testing errors given in Table 8 and the mean relative errors given in Table 9 for the dense model on displacement and acceleration predictions show that with the data range division strategy the model accuracy becomes acceptable (less than 10% mean relative errors). Next, SC layers are constructed based on the sparsity patterns, which are similar to Case-2 and Case-4 in § 4.3.2 and § 4.4.2. After that, the optimal number () of CONV layers to be added to SC layers is investigated and the training results with different ’s are plotted in Figure 32, Figure 33, and Figure 34. From the results, is chosen for the sparse NNs model, which gives adequate accuracy. The mean squared errors and mean relative errors of sparse NN approximation for displacement and acceleration are given in Table 10 and Table 11. By comparing Table 8 with Table 10 and Table 9 with Table 11, it can be concluded that the proposed sparse NN architecture can be a good candidate for a parsimonious model with competitive accuracy. For different loading frequency ranges, predictions of displacement and acceleration on random loading samples from the testing dataset are compared against the analytical solutions in Figure 35, Figure 36, and Figure 37, where fairly good accuracy can be observed. It is noted that the combined three NNs trained for the three subranges , and cannot fully fulfill the original task where loading frequencies are drawn randomly from the single range . This is because the samples that contain frequencies across multiple subranges cannot be predicted using the three NNs.
| Loading frequency range | Displacement | Acceleration |
| 0.0049 | 0.0051 | |
| 0.0020 | 0.0094 | |
| 0.0084 | 0.0183 |
Table 8. Mean squared testing errors of the trained dense networks for linear MDOF system (Case-6) with different loading frequency ranges
| Loading frequency range | Displacement | Acceleration |
| 1.27% | 1.30% | |
| 0.98% | 6.68% | |
| 6.21% | 8.96% |
Table 9. Mean relative errors over testing dataset of the trained dense networks for linear MDOF system (Case-6) with different loading frequency ranges
| Loading frequency range | Displacement | Acceleration |
| 0.0051 | 0.0064 | |
| 0.0022 | 0.0098 | |
| 0.0069 | 0.0159 |
Table 10. Mean squared testing errors of the trained sparse network () for MDOF system (Case-6) with different loading frequency ranges
| Loading frequency range | Displacement | Acceleration |
| 3.45% | 3.53% | |
| 1.33% | 6.84% | |
| 5.49% | 7.68% |
Table 11. Mean relative errors over testing dataset of the trained sparse network () for MDOF system (Case-6) with different loading frequency ranges
(a) (a) Displacement (#non-zero weights = 5523)
(b) (b) Acceleration (#non-zero weights = 5661)
Figure 29. Sparsity patterns of the weight matrix of (single) FC layer in the dense models for linear MDOF (Case-6) in the loading frequency range : (a) Displacement; (b) Acceleration.
Figure 30. (a) Displacement (#non-zero weights = 3227) (b) Acceleration (#non-zero weights = 3367). Sparsity patterns of the weight matrix of (single) FC layer in the dense models for linear MDOF (Case-6) in the loading frequency range : (a) Displacement; (b) Acceleration.
Figure 31. (a) Displacement (#non-zero weights = 2030) (b) Acceleration (#non-zero weights = 2247). Sparsity patterns of the weight matrix of (single) FC layer in the dense models for linear MDOF (Case-6) in the loading frequency range : (a) Displacement; (b) Acceleration.
Figure 32. Parametric studies on the optimal number () of CONV layers in the sparse networks for linear MDOF (Case-6) in the loading frequency range : (a) Displacement; (b) Acceleration.
Figure 33. Parametric studies on the optimal number () of CONV layers in the sparse networks for linear MDOF (Case-6) in the loading frequency range : (a) Displacement; (b) Acceleration.
(a) Displacement
(b) Acceleration
Figure 34. Parametric studies on the optimal number () of CONV layers in the sparse networks for linear MDOF (Case-6) in the loading frequency range : (a) Displacement; (b) Acceleration.
Figure 35. Performance comparison of the dense and sparse network models on three random samples from testing dataset for linear MDOF (Case-6) in the loading frequency range : Left – displacement; Right – acceleration.
Figure 36. Performance comparison of the dense and sparse network models on three random samples from testing dataset for linear MDOF (Case-6) in the loading frequency range : Left – displacement; Right – acceleration.
Figure 37. Performance comparison of the dense and sparse network models on three random samples from testing dataset for linear MDOF (Case-6) in the loading frequency range : Left – displacement; Right – acceleration.
Case-7: Nonlinear MDOF
Since nonlinear spring is considered in this case, a dense network with a single FC layer is unable to predict the responses accurately. Therefore, the hyperparameter study of dense networks in § 4.5 is conducted in this section as well. The number of FC layers needed to reach an acceptable accuracy, i.e., mean relative testing error smaller than 10%, is shown in Table 12 and can be seen to increase with the nonlinearity in the system. Figure 38, Figure 39, Figure 40, and Figure 41 shows several prediction samples of displacement and acceleration for the dynamical system with different nonlinearities and these results confirm the good accuracy of the presented network models.
| Cubic term coefficient | Number of needed FC layers | Displacement | Acceleration |
| 2 | 6.11% | 6.31% | |
| 2 | 7.04% | 7.72% | |
| 4 | 6.41% | 6.88% | |
| 5 | 6.86% | 7.00% |
Table 12. Number of needed FC layers of the trained dense networks for nonlinear MDOF system (Case-7) to reach the acceptable accuracy (mean relative error of testing samples smaller than 10%) and the corresponding mean relative errors
Figure 38. Prediction by the dense network of three random samples from testing dataset in Case-7 with cubic coefficient : Left – displacement; Right – acceleration.
Figure 39. (a) Sample 1; (b) Sample 2; (c) Sample 3. Prediction by the dense network of three random samples from testing dataset in Case-7 with cubic coefficient : Left – displacement; Right – acceleration.
Figure 40. Prediction by the dense network of three random samples from testing dataset in Case-7 with cubic coefficient : Left – displacement; Right – acceleration.
Figure 41. Prediction by the dense network of three random samples from testing dataset in Case-7 with cubic coefficient : Left – displacement; Right – acceleration.
By examining the sparsity with the criterion in § 4.3.1, it is found that there is no sparsity in the weight matrices in the dense network models, and therefore no sparse models can be constructed. Again, the same replacement strategy with lightweight CONV layers in § 4.5 is adopted for the MDOF case with the largest nonlinearity, i.e., . Transfer learning is used to investigate the number () of CONV layers needed to reach a similar accuracy level as with the original dense layer. With CONV layers, the mean relative errors of testing samples are 6.34% for displacement and 6.92% for acceleration, which are close to those from the original dense models, i.e., 6.86% for displacement and 7.00% for acceleration. Moreover, there is again a 17.94% decrease in the number of trainable parameters in the CONV enriched NN model as compared to the original dense model. Several predicted samples from the two NN models are shown in Figure 42, which confirms the competitive performance of the two NN architectures.
Figure 42. Performance comparison of the CONV enriched NN model and the dense NN model on three random samples from testing dataset in Case-7 with cubic coefficient : Left – displacement; Right – acceleration.
Conclusions
In this study, the use of deep neural network models as surrogates for linear and nonlinear structural dynamical systems is systematically examined. It is shown that for the considered linear SDOF and MDOF systems the fully connected neural network models exhibit sparsity patterns, i.e., some of the network weight parameters are much smaller than others. Moreover, the sparsity patterns can be different for different problem settings and may depend on the loading duration and the number of harmonic terms considered. This sparsity pattern can be utilized to prune a network architecture, which leads to lightweight and more memory-efficient network models. The performance of the sparse networks is further enhanced by adding parameter-efficient convolutional layers. The addition of the convolutional layers does not significantly increase the total number of trainable parameters due to their parameter sharing properties. As a result, the convolutional layer enhanced sparse network models are proposed for linear structural dynamics. The proposed sparse network models have competitive performance and are computationally efficient as compared to the fully connected NNs. Moreover, for training deep NN models, the transfer learning strategy proposed in this study is shown to have a stable training performance, i.e., loss function value continues to decrease as more convolutional layers are added.
For nonlinear dynamical response with cubic nonlinearities, it is shown that more fully connected network layers are needed as the magnitude of the cubic coefficient increases, i.e., a deeper network model is needed to learn a more complex system, as expected. Moreover, in this case, the sparsity patterns in the fully connected networks are not present. To take advantage of the lightweight property of CONV layers, several CONV layers are used to replace the first fully connected layer in the dense model and trained via transfer learning while other layers in the trained dense model are transferred. Compared to the fully connected NN model, the CONV enriched NN models have less trainable parameters while having similar testing performances. Thus, these models can be used as an alternative to fully connected NNs. Finally, in this study, only linear and nonlinear elastic systems are considered under harmonic excitations. Future work will explore the use of similar machine learning techniques for inelastic structural dynamical response prediction under more complex loading scenarios, e.g., seismic excitations and impact/blast loadings.
ACKNOWLEDGMENTS
The presented work is supported in part by the USNational Science Foundation through grant CMMI-1762277. Any opinions, findings, conclusions, and recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the sponsors.
Analytical solutions of linear SDOF and Mdof sYstems
Linear SDOF system
For linear SDOF systems in Eq. (1), the analytical solution of displacement can be expressed as
where the general solution and particular solution are calculated as
and
The acceleration can be computed by taking the second order derivatives of in Eq. (A 1), i.e.
where
and
Linear MDOF system
For linear MDOF system in Eq. (4), the analytical solution of displacement given by modal decomposition reads
where are eigenvalue-eigenvector pairs of the matrix and
with the general solution
and particular solution
Taking the second order derivative of in Eq. (A 7), the analytical solution of acceleration reads
where

(a) Displacement

(b) Acceleration
Figure 14. Parametric studies on the optimal number () of CONV layers in the sparse networks in Case-2: (a) Displacement; (b) Acceleration.

(a) (a) Displacement

(b) (b) Acceleration
Figure 18. Parametric studies on the optimal number () of CONV layers in the sparse networks in Case-3: (a). Displacement; (b) Acceleration.
References
- [1]C. Aggarwal, Neural Networks and Deep Learning. A Textbook, in, Cham: Springer International Publishing, 2018.
- [2]I. Goodfellow, Y. Bengio, A. Courville, Deep learning, MIT press, 2016.
- [3]M. Längkvist, L. Karlsson, A. Loutfi, A review of unsupervised feature learning and deep learning for time-series modeling, Pattern Recognition Letters, 42 (2014) 11-24.DOI
- [4]J. Schmidhuber, Deep learning in neural networks: An overview, Neural networks, 61 (2015) 85-117.
- [5]A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, 2012, pp. 1097-1105.
- [6]M.D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, in: European conference on computer vision, Springer, 2014, pp. 818-833.arxiv.org/abs/1311.2901
- [7]O. Abdel-Hamid, A. Mohamed, H. Jiang, L. Deng, G. Penn, D. Yu, Convolutional Neural Networks for Speech Recognition, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22 (2014) 1533-1545.
- [8]W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang, A. Stolcke, The Microsoft 2017 Conversational Speech Recognition System, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5934-5938.DOI
- [9]Y. Wu, M. Schuster, Z. Chen, Q.V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, Google's neural machine translation system: Bridging the gap between human and machine translation, arXiv preprint arXiv:1609.08144, (2016).arxiv.org/abs/1609.08144
- [10]J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, (2018).arxiv.org/abs/1810.04805
- [11]Y. Kim, Convolutional neural networks for sentence classification, arXiv preprint arXiv:1408.5882, (2014).arxiv.org/abs/1408.5882
- [12]A. Conneau, H. Schwenk, L. Barrault, Y. Lecun, Very deep convolutional networks for natural language processing, arXiv preprint arXiv:1606.01781, 2 (2016).
- [13]H. Sun, H.V. Burton, H. Huang, Machine learning applications for building structural design and performance assessment: State-of-the-art review, Journal of Building Engineering, 33 (2021) 101816.
- [14]J. Xue, Z. Xiang, G. Ou, Predicting single freestanding transmission tower time history response during complex wind input through a convolutional neural network based surrogate model, Engineering Structures, 233 (2021) 111859.
- [15]S. Mukherjee, D. Lu, B. Raghavan, P. Breitkopf, S. Dutta, M. Xiao, W. Zhang, Accelerating Large-scale Topology Optimization: State-of-the-Art and Challenges, Archives of Computational Methods in Engineering, (2021).DOI
- [16]X. Liu, F. Tao, W. Yu, A neural network enhanced system for learning nonlinear constitutive law and failure initiation criterion of composites using indirectly measurable data, Composite Structures, 252 (2020) 112658.DOI
- [17]A. Koeppe, F. Bamer, B. Markert, An intelligent nonlinear meta element for elastoplastic continua: deep learning using a new Time-distributed Residual U-Net architecture, Computer Methods in Applied Mechanics and Engineering, 366 (2020) 113088.DOI
- [18]Z. Xiang, J. Chen, Y. Bao, H. Li, An active learning method combining deep neural network and weighted sampling for structural reliability analysis, Mechanical Systems and Signal Processing, 140 (2020) 106684.
- [19]M. Li, Z. Wang, Deep learning for high-dimensional reliability analysis, Mechanical Systems and Signal Processing, 139 (2020) 106399.DOI
- [20]K. Morfidis, K. Kostinakis, Approaches to the rapid seismic damage prediction of r/c buildings using artificial neural networks, Engineering Structures, 165 (2018) 120-141.DOI
- [21]S. Naito, H. Tomozawa, Y. Mori, T. Nagata, N. Momma, H. Nakamura, H. Fujiwara, G. Shoji, Building-damage detection method based on machine learning utilizing aerial photographs of the Kumamoto earthquake, Earthquake Spectra, 36 (2020) 1166-1187.DOI
- [22]G. Fan, J. Li, H. Hao, Y. Xin, Data driven structural dynamic response reconstruction using segment based generative adversarial networks, Engineering Structures, 234 (2021) 111970.
- [23]T.A. Horton, I. Hajirasouliha, B. Davison, Z. Ozdemir, Accurate prediction of cyclic hysteresis behaviour of RBS connections using Deep Learning Neural Networks, Engineering Structures, 247 (2021) 113156.
- [24]N.A. Kallioras, G. Kazakis, N.D. Lagaros, Accelerated topology optimization by means of deep learning, Structural and Multidisciplinary Optimization, 62 (2020) 1185-1212.DOI
- [25]D.W. Abueidda, S. Koric, N.A. Sobh, Topology optimization of 2D structures with nonlinearities using deep learning, Computers & Structures, 237 (2020) 106283.DOI
- [26]L.C. Nguyen, H. Nguyen-Xuan, Deep learning for computational structural optimization, ISA Transactions, 103 (2020) 177-191.DOI
- [27]S. Masri, M. Nakamura, A. Chassiakos, T. Caughey, Neural network approach to detection of changes in structural parameters, Journal of engineering mechanics, 122 (1996) 350-360.DOI
- [28]D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning representations by back-propagating errors, nature, 323 (1986) 533-536.
- [29]R.-T. Wu, M.R. Jahanshahi, Deep convolutional neural network for structural dynamic response estimation and system identification, Journal of Engineering Mechanics, 145 (2019) 04018125.
- [30]R. Zhang, Z. Chen, S. Chen, J. Zheng, O. Büyüköztürk, H. Sun, Deep long short-term memory networks for nonlinear structural seismic response prediction, Computers & Structures, 220 (2019) 55-68.
- [31]M. Stoffel, R. Gulakala, F. Bamer, B. Markert, Artificial neural networks in structural dynamics: A new modular radial basis function approach vs. convolutional and feedforward topologies, Computer Methods in Applied Mechanics and Engineering, 364 (2020) 112989.DOI
- [32]M. Stoffel, F. Bamer, B. Markert, Deep convolutional neural networks in structural dynamics under consideration of viscoplastic material behaviour, Mechanics Research Communications, 108 (2020) 103565.DOI
- [33]Y. Yu, H. Yao, Y. Liu, Structural dynamics simulation using a novel physics-guided machine learning method, Engineering Applications of Artificial Intelligence, 96 (2020) 103947.
- [34]R. Zhang, Y. Liu, H. Sun, Physics-guided convolutional neural network (PhyCNN) for data-driven seismic response modeling, Engineering Structures, 215 (2020) 110704.arxiv.org/abs/1909.08118
- [35]R. Zhang, Y. Liu, H. Sun, Physics-Informed Multi-LSTM Networks for Metamodeling of Nonlinear Structures, arXiv preprint arXiv:2002.10253, (2020).DOI
- [36]S. Sadeghi Eshkevari, M. Takáč, S.N. Pakzad, M. Jahani, DynNet: Physics-based neural architecture design for nonlinear structural response modeling and prediction, Engineering Structures, 229 (2021) 111582.
- [37]C.M. Bishop, Pattern recognition, Machine learning, 128 (2006).
- [38]T.J. Hughes, The finite element method: linear static and dynamic finite element analysis, Courier Corporation, 2012.
- [39]K.P. Murphy, Machine learning: a probabilistic perspective, MIT press, 2012.
- [40]S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:1502.03167, (2015).arxiv.org/abs/1502.03167
- [41]J. Grus, Data science from scratch: first principles with python, O'Reilly Media, 2019.
- [42]D. Tax, R. Duin, Feature scaling in support vector data descriptions, Learning from Imbalanced Datasets, (2000) 25-30.
- [43]D.P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, (2014).arxiv.org/abs/1412.6980
- [44]K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026-1034.arxiv.org/abs/1502.01852
- [45]D.P. Kroese, T. Taimre, Z.I. Botev, Handbook of monte carlo methods, John Wiley & Sons, 2013.