betaFeedback

WorksComputer Science

Using Machine Learning and Neural Networks to Analyze and Predict Chaos in Multi-Pendulum and Chaotic Systems

Abstract

A chaotic system is a highly volatile system characterized by its sensitive dependence on initial conditions and outside factors. Chaotic systems are prevalent throughout the world today: in weather patterns, disease outbreaks, and even financial markets. Chaotic systems are seen in every field of science and humanities, so being able to predict these systems is greatly beneficial to society. In this study, we evaluate 10 different machine learning models and neural networks [1] based on Root Mean Squared Error (RMSE) and R^2 values for their ability to predict one of these systems, the multi-pendulum. We begin by generating synthetic data representing the angles of the pendulum over time using the Runge Kutta Method for solving 4th Order Differential Equations (ODE-RK4) [2]. At first, we used the single-step sliding window approach, predicting the 50st step after training for steps 0-49 and so forth. However, to more accurately cover chaotic motion and behavior in these systems, we transitioned to a time-step based approach. Here, we trained the model/network on many initial angles and tested it on a completely new set of initial angles, or 'in-between' to capture chaotic motion to its fullest extent. We also evaluated the stability of the system using Lyapunov exponents. We concluded that for a double pendulum, the best model was the Long Short Term Memory Network (LSTM)[3] for the sliding window and time step approaches in both friction and frictionless scenarios. For triple pendulum, the Vanilla Recurrent Neural Network (VRNN)[4] was the best for the sliding window and Gated Recurrent Network (GRU) [5] was the best for the time step approach, but for friction, LSTM was the best.

ARTICLE INFO

Keywords:

Figure 2.

ABSTRACT

Differential Equations (ODE-RK4) [2]. At first, we used the single-step sliding window approach, predicting the 50st step after training for steps 0-49 and so forth. However, to more accurately cover chaotic motion and behavior in these systems, we transitioned to a time-step based approach. Here, we trained the model/network on many initial angles and tested it on a completely new set of initial angles, or “in-between” to capture chaotic motion to its fullest extent. We also evaluated the stability of the system using Lyapunov exponents. We concluded that for a double pendulum, the best model was the Long Short Term Memory Network (LSTM)[3] for the sliding window and time step approaches in both friction and frictionless scenarios. For triple pendulum, the Vanilla Recurrent Neural Network (VRNN)[4] was the best for the sliding window and Gated Recurrent Network (GRU) [5] was the best for the time step approach, but for friction, LSTM was the best.

INTRODUCTION

Chaotic systems [6] are prevalent in various natural and engineered environments, from weather systems[7] and ecosystems to biomedicine [8] and financial markets[9]. These systems, characterized by their sensitivity to initial conditions[10], often exhibit unpredictable and complex behavior. Understanding and predicting the dynamics of chaotic systems[11] is crucial for better decision-making and control in engineering, economics, and meteorology, minimizing risks and enhancing performance. For example, the accurate prediction of the chaotic system of winds and weather could allow us to plan more efficient flight routes at more efficient times to minimize fuel usage and minimize travel time[12]. In addition, it could aid our ability to predict potential catastrophic weather events. Our research focuses on the multi-pendulum system, the simplest chaotic system. By being able to predict the multi-pendulum system, we can use insights from the machine learning models and neural networks [13] detailed in this paper to create machine learning models and neural networks that can predict various chaotic systems well using minimal resources. Furthermore, this research would also indicate models and/or neural networks that predict chaotic systems such as the multi-pendulum [14] the best, reducing time and resources needed to evaluate various machine learning techniques and neural networks.

The multi-pendulum system, specifically the double pendulum [15] consists of one pendulum attached to an immovable point, and a second pendulum attached to the end of the first pendulum Figure 1. For the triple pendulum [16], a third pendulum is attached to the second pendulum Figure 1. The multi-pendulum system is a simple yet powerful model that illustrates the complex dynamics arising from non-linear interactions. Despite its simplicity, predicting the future states of the multi-pendulum accurately is challenging due to its highly sensitive dependence on initial conditions and the intricate interplay between its components.

Illustration of the double and triple pendulum with angles, masses, and lengths labeled

Figure 1 (PDF p. 3). An illustration of the double and triple pendulum with angles, masses, and lengths of the pendulums labeled.

In relatively simple classical mechanics problems such as the multi-pendulum system, the position of the system is given by differential equations [17] Figure 3. To get the trajectory of a multi-pendulum system, we first need to solve the differential equations of motion. The double pendulum consists of two second-order differential equations.

Data Preprocessing Workflow

Figure 3. Data Preprocessing Workflow

Configuration vector V=[θ1,u1,θ2,u2]\mathbf{V} = [\theta_1, u_1, \theta_2, u_2] has two angles θ1\theta_1 and θ2\theta_2 and the angular velocity u1u_1 and u2u_2 defined as:

u1=dθ1/dtu_1 = \mathrm{d}\theta_1/\mathrm{d}t and u2=dθ2/dtu_2 = \mathrm{d}\theta_2/\mathrm{d}t

We let c=cos⁡(θ1−θ2)c = \cos(\theta_1 - \theta_2) and s=sin⁡(θ1−θ2)s = \sin(\theta_1 - \theta_2) so the equations are:

du1dt=m2gsin⁡(θ2)c−m2s(L1u12+L2u22c)−(m1+m2)gsin⁡(θ1)L(m1+m2s2)(1)\frac{\mathrm{d}u_1}{\mathrm{d}t} = \frac{m_2 g \sin(\theta_2)c - m_2s\left(L_1u_1^2 + L_2u_2^2c\right) - \left(m_1 + m_2\right)g\sin(\theta_1)} {L\left(m_1 + m_2s^2\right)} \tag*{(1)}
du2dt=(m1+m2)(L1u12s−gsin⁡(θ1)+gsin⁡(θ2)c)+m2L2u22scL2(m1+m2s2)(2)\frac{\mathrm{d}u_2}{\mathrm{d}t} = \frac{\left(m_1 + m_2\right)\left(L_1u_1^2s - g\sin(\theta_1) + g\sin(\theta_2)c\right) + m_2L_2u_2^2sc} {L_2\left(m_1 + m_2s^2\right)} \tag*{(2)}

(Proof of equations / editable equations in Electronic Supplementary Information)

For triple pendulum, the equations are:

du1dt=mf(od1_1⁡⋅od1_2⁡⋅od1_3⁡⋅r2−4⋅(−m3⋅od1_2⁡⋅r1+(m3⋅r12+m12⋅r3⋅r2)⋅cos⁡(θ1−θ2))⋅od1_4⁡−(od1_5⁡+4m3r12+4m12r3r2)⋅od1_6⁡)l1⋅od1_7⁡⋅m012⋅r2\frac{\mathrm{d}u_1}{\mathrm{d}t} = \frac{m_f\left(\operatorname{od1\_1}\cdot\operatorname{od1\_2}\cdot\operatorname{od1\_3}\cdot r_2 - 4\cdot\left(-m_3\cdot\operatorname{od1\_2}\cdot r_1+\left(m_3\cdot r_1^2+m_{12}\cdot r_3\cdot r_2\right)\cdot\cos(\theta_1-\theta_2)\right)\cdot\operatorname{od1\_4}-\left(\operatorname{od1\_5}+4m_3r_1^2+4m_{12}r_3r_2\right)\cdot\operatorname{od1\_6}\right)} {l_1\cdot\operatorname{od1\_7}\cdot m_{012}\cdot r_2}
du2dt=−m3⋅r1⋅m012⋅od2_1⁡⋅r2−(m3⋅(r1cos⁡(θ1−θ2)+r2cos⁡(θ1−θ3))⋅r1−(m3⋅r12+m12⋅r3⋅r2)⋅cos⁡(θ1−θ2))⋅od2_2⁡+m012⋅r3⋅r2⋅od2_3⁡l2⋅od1_7⁡⋅r2\frac{\mathrm{d}u_2}{\mathrm{d}t} = \frac{-m_3\cdot r_1\cdot m_{012}\cdot\operatorname{od2\_1}\cdot r_2-\left(m_3\cdot\left(r_1\cos(\theta_1-\theta_2)+r_2\cos(\theta_1-\theta_3)\right)\cdot r_1-\left(m_3\cdot r_1^2+m_{12}\cdot r_3\cdot r_2\right)\cdot\cos(\theta_1-\theta_2)\right)\cdot\operatorname{od2\_2}+m_{012}\cdot r_3\cdot r_2\cdot\operatorname{od2\_3}} {l_2\cdot\operatorname{od1\_7}\cdot r_2}
du3dt=−m12⋅(od1_2⁡)⋅(od3_1⁡)+m12⋅m012⋅(od3_2⁡)⋅r2−r1⋅m012⋅od3_3⁡l3⋅(m3⋅r12+m12⋅r3⋅r2)\frac{\mathrm{d}u_3}{\mathrm{d}t} = -\frac{m_{12}\cdot\left(\operatorname{od1\_2}\right)\cdot\left(\operatorname{od3\_1}\right)+m_{12}\cdot m_{012}\cdot\left(\operatorname{od3\_2}\right)\cdot r_2-r_1\cdot m_{012}\cdot\operatorname{od3\_3}} {l_3\cdot\left(m_3\cdot r_1^2+m_{12}\cdot r_3\cdot r_2\right)}
Animation of the double and triple pendulum

Figure up. Animation of the double and triple pendulum. 10s of animation. This was created by us using the differential equations from cloud4science et al. [7] for the double pendulum, and Yesilyurt [9] for the triple pendulum. Immovable point at (0,0), each pendulum is of length one. Chaotic motion is clearly presented.

We begin by solving the differential equations for the double pendulum for angles [18] from timestep 0 to 1000, with a step size of 0.001. We take the synthetic data and save it in a 2D NumPy array [19]: one array for θ1\theta_1 and one array for θ2\theta_2. We then modeled the pendulum through MatPlotLib [20]. We then analyzed the different pendulum data trajectories to determine the balance between predictability and chaos. Further details can be found in the Electronic Supplementary Information.

Workflow for model testing

Figure 4. Workflow for model testing.

Using the 10,000,000 synthetic data points for the initial angles [90∘90^\circ, 90∘90^\circ] where the angles are in the format [θ1\theta_1, θ2\theta_2] created for training and testing data, we initially formed one comprehensive dataset to use for training and testing. Our features initially included the two angle arrays, θ1\theta_1 and θ2\theta_2, which contain angle measurements at each of the 10,000,000 timesteps.

In our study, we employ both machine learning and neural network techniques to analyze and predict the behavior of the double and triple pendulum. Our method leverages the power of sequential and supervised neural networks and models to get more accurate predictions. Our work differs from previous research in several significant ways. One of the most similar projects, Evaluating Current Machine Learning Techniques On Predicting Chaotic Systems by S. Klinkachorn and J. Parmar from Stanford University [21], utilizes 4 of the same models, but uses a different approach in predictions. It involves minimal analysis of the double pendulum itself and no analysis of the triple pendulum. As such, our approach, while inspired by such methodologies, is generalized for multi-pendulum systems as a whole, showcasing its broader applicability in chaotic systems analysis. Among the models tested in this study, Long-Short Term Memory (LSTM) network and Autoregressive (AR) Model presented the best results; LSTM achieving an RMSE of 1.4E−21.4\mathrm{E}^{-2} and an R2R^2 score of 0.998, and AR achieving an RMSE of 1.5E−21.5\mathrm{E}^{-2} and an R² score of 0.998. From those results, we can successfully showcase the power of machine learning and deep learning in the scope of predicting chaos, and its application in the real world. Along with neural networks to assess the multi-pendulum system, we employed both the Lyapunov exponent and eigenvalues for the double pendulum differential equations to quantify the stability of the system at various initial conditions and equilibrium points.

METHODS AND MATERIALS

First, using the equations for the double pendulum the Runge-Kutta 4th Order Method (ODE-RK4), and these initial conditions { Figure 5A } we calculated the time evolution of the pendulum angles from 0 to 1000 seconds with a step size of 0.001. We then took the angle at each step as calculated by the differential equations and pushed the data points into a NumPy array. These 10,000,000 synthetic data points were then scaled using MinMaxScaler and normalized to ensure effective model training.

Image not extracted

Figure 5. A: Initial Conditions and B: ODE-RK4 Algorithm

Initial conditions:

g=9.81 m/s2g = 9.81\ \mathrm{m/s^2}

l1=1 ml_1 = 1\ \mathrm{m}

l2=1 ml_2 = 1\ \mathrm{m}

m1=1 kgm_1 = 1\ \mathrm{kg}

m1=1 kgm_1 = 1\ \mathrm{kg}

Key - gg: Gravitational Constant on Earth (ms2\frac{m}{s^2}), L: Length of Pendulum 1/2 (meters), m: mass of Pendulum 1/2 (kg), y: position (given recursively), k: Approximations of the derivative at different points within our iteration, h: Size of each step

After solving the differential equations with ODE-RK4, we created an animation using Matplotlib to plot the double and triple pendulum data points at a steep angle ([120,120]) to visualize the trajectory graph over a 10-second interval. { Figure 2A, Figure 2B }.

Due to high sensitivity to initial conditions { Figure 6 }, we tested different initial conditions to see how they would affect both trajectories of the double pendulum.

Figure 6. RK4 Result Graph Solved differential equations, showing the different angles for θ1\theta_1 and θ2\theta_2 over the time interval (0, 100). Initial angles for this graph: [90∘,90∘][90^\circ, 90^\circ] vs. RK4 Result Graph/ solved differential equations, showing the different angles for θ1\theta_1 and θ2\theta_2 over the time interval (0, 1000). Initial conditions for 6A: gravity: 9.81 ms2\frac{\mathrm{m}}{\mathrm{s}^2} (standard gravity), l1l_1: 1m, l2l_2: 2m, m1m_1: 1kg, m2m_2: 1kg, θ1\theta_1: 60°, θ2\theta_2: 90°. Initial conditions for 6B: gravity: 9.81 ms2\frac{\mathrm{m}}{\mathrm{s}^2} (standard gravity), l1l_1: 1m, l2l_2: 2m, m1m_1: 2kg, m2m_2: 1kg, θ1\theta_1: 60°, θ2\theta_2: 90°.

Figure 6A.

Changed condition

The difference between the graphs of Figure 6 exemplifies the influence that initial conditions have on the double pendulum. For simplicity and to generate a good baseline, we decided to stick to modifying only initial angles without considering friction instead of other variables of interest such as gravity, mass, or length. We later moved on to include friction in our analysis as well. We then analyzed the pendulums’ graphs and trajectories to determine patterns of chaos to gain a deeper understanding of the system. One seemingly apparent pattern would be the consistency of θ1\theta_1, but this pattern stems from the fact that the first pendulum goes through roughly the same angles. (citation needed)

Initially, we considered 1000 seconds of data as sufficient enough to model the chaotic double pendulum as this would adequately cover the true chaos in the system past the initial period of stability seen. Our goal is to be able to predict the motion of the pendulum at its most chaotic stages and see how models perform with as minimum data [22] as possible. Chaos develops in the double pendulum gradually. Because this data is minimal, we used Adam Optimizer [23] – a learning rate regulator – to prevent overfitting [24], with a minimum learning rate of 10−410^{-4}.

The initial dataset ingested into each model tested in this study had a size of 10,000,000 points, each representing an angle measure of the pendulum at a specific time step in that 1000 second interval. The feature of the model was a stacked NumPy array of the angles [θ1,θ2][\theta_1, \theta_2] created by ODE-RK4’s solution to the differential equation (More information about the approach can be found in Electronic Supplementary Information).

The following machine learning models and neural networks were trained and evaluated for this study: Random Forest Regressor (RF) [25], Multi-Layer Perceptron Regressor (MLPRegressor) [26], Long-Short Term Memory Neural Network (LSTM), Vanilla Recurrent Neural Network(VRNN), BiDirectionalRNN (BIRNN) [27], StackedRNN(STRNN), SGDRegressor (SGD) [28], Gated Recurrent Network (GRU), Autoregressive Model (AR) [29], and Feed Forward Neural Network (FFNN). All models were implemented with Python’s SciKit-Learn module. PyTorch was used to convert data into tensors for all neural networks [30]. Our results were evaluated using two metrics: R2R^2 score and Root Mean Squared Error (RMSE). We did this as each metric has its own purpose: R2R^2 is a numerical representation of the data’s fit to the model, whereas RMSE is a numerical representation of how well the model is able to extrapolate the data given. We consider RMSE as the preferred analysis metric as it provides insight on a model’s ability to make predictions, rather than its applicability to the double pendulum model specifically.

Triple pendulum analysis consisted of the same steps, except using differential equations pulled from Yesilyurt and input sequences being a stacked NumPy array with angles [θ1,θ2,θ3][\theta_1, \theta_2, \theta_3]. ODE-RK4 was used in the same way as for the double pendulum. Initial conditions were the same for each of the three pendulums, specifically [90∘,90∘,90∘][90^\circ, 90^\circ, 90^\circ] with standard mass, gravity, and length. This is similar to the double pendulum where initial conditions [90∘,90∘][90^\circ, 90^\circ] for each of the two pendulums, with the same standard unchanged conditions. The exact same models were trained and analyzed with the same metrics and analysis practices as with the double pendulum.

Neural Network Sliding Window Approach Model Architecture

Figure 7. Neural Network Sliding Window Approach Model Architecture Angle measures of the double and triple pendulums at the first 50 time steps were taken. The model trains on these data points and predicts the 51st data point.

Sliding Window

To evaluate the neural network, we used a single-step sliding window approach for predictions [31]. In this approach, sequences of 49 inputs (one input is a set of angles); and predicts one step (one set of angles at the consequent timestep). In the first iteration, angles from timestep 0 to timestep 49 are part of the sequence of inputs, and the model predicts the 50th. In the second iteration, angles from timestep 1 to 50 from the actual data are included in the input sequence. [32] From this, the model predicts the 51st step. This process continues over the 10,000,000 data points in our training set. The sliding window approach is widely used in time series problems such as the double pendulum, as it makes it easier for the model to capture temporal dependencies in the data. [33]

This sliding window approach is good for short term predictions confined to a specific set of initial angles, but not necessarily for multiple different initial angles. [34] In fact, chaotic systems, by definition, are characterized by their sensitive dependence on initial conditions [35], and at this point, initial conditions weren’t being changed in our project and true total chaos wasn’t being predicted; rather, we were just fitting a very erratic curve. In addition, as we were using the first 49 data points for training and never testing on the same data points, we were not able to construct an accurately visualized trajectory. As a result, we started to work backwards to come up with a new approach that would address these three issues.

Time-step Based Approach

After brainstorming, we came up with the idea of training our best models on many different certain initial angles with a timestep based approach, and then testing it on a completely new untrained set of initial angles within the range of initial angles tested. For example, if our model was trained on initial angles [120∘120^\circ, 0], [120∘120^\circ, 0.1∘0.1^\circ]...[120∘120^\circ, 2.9∘2.9^\circ]...[120∘120^\circ, 3.0∘3.0^\circ], we would pick an initial angle such as [120∘120^\circ, 2.05∘2.05^\circ] and ask our model to predict the trajectory. In addition, this approach doesn’t use a sliding window, but an input feature, the timestep: the time at which each angle is observed in the double pendulum system, which allows the model to predict for a specific point in time, allowing us to easily visualize and compare it with [36]. Lastly, the approach takes in the initial angle in every timestep input, allowing the model to differentiate the initial models and come up with its own way to predict previously untrained “inbetweens”, or untrained initial angles, whose trajectories chaotically differ from the trained trajectories, allowing us to truly–to a certain extent–predict the trajectory of chaotic systems [37]. This approach also allows us to fully predict the pendulum from 0s to 10s, allowing us to make a visualized trajectory graph comparison,and fully showing the capability of the model to predict the entire interval of chaos. This approach fully captures the chaos of the double pendulum system and allows for patterns to be recognized by these models to predict unknown systems as we train multiple different types of models using the same approach and preprocessing [38]. This allows us to see which type of machine learning model or neural network proves most promising at generalizing and predicting chaotic systems as a whole. By creating the novel time-step approach, we were able to successfully predict true chaotic motion, and showcase the power of machine learning and artificial intelligence in solving those problems.

Again, by using ODE-RK4, we generated synthetic data for the double pendulum. However this time only over 10 seconds for each initial angle. We had to figure out the most optimal angle for chaos, as we reduced the training window to 10s for faster computing time of multiple different initial angles instead of only one. We trained on 25 initial angles including [120∘120^\circ, 0], [120∘120^\circ, 0.1∘0.1^\circ] … [120∘120^\circ, 1.0∘1.0^\circ], [120∘120^\circ, 1.1∘1.1^\circ] … [120∘120^\circ, 2.0∘2.0^\circ] ...[120∘120^\circ, 3.0∘3.0^\circ]. These were all combined into a stacked Numpy array, but with further subdivisions for each initial angle to ensure training on each initial angle separately. The final dataset contained 2,000 data points for each initial angle, of which there were thirty trained on. In total, our dataset consisted of 60,000 unique data points to train and test our models on, and an extra 2,000 data points for testing on the “inbetween” of [120°, 2.05°].

The same models were trained for the time-step based approach as the sliding window approach, with the exception of SGD Regressor and Random Forest. We then compared the visualized trajectories for each model prediction, the graph between predicted, the actual angles, and the RMSE to determine the best model for predicting the chaotic motion of the multi-pendulum system.

We then created a program to test our model on completely new conditions (eg. [120°, 2.05°]), and evaluate how accurate the prediction compares to the actual values. Training on these completely new conditions means that the model has to adapt to a pattern it has never recognized before. A slight tweak in initial angles drastically changes the time-angle relation of the pendulum, making it almost “unpredictable.” Based on our results, we were able to verify which models performed the best in our previous approach, and also examined which models would be able to predict based on completely unseen data the best. Our results were again evaluated using two metrics: R2R^2 score and Root Mean Squared Error (RMSE). We also compared the visualized trajectories for each model prediction again to see how our models performed in relation to each other, and in relation to the sliding window approach.

In order to understand the dynamics of the double pendulum further, we conducted stability analysis to find the non-chaotic points of the system, using the Lyapunov exponent [18] with the differential equations by finding the slope of the regression line of the data: (t,ln⁡(error(t)))(t, \ln(error(t))) for 10000 timesteps at every starting condition θ1\theta_1, θ2∈[0,180]\theta_2 \in[0, 180] with increments of 15° and θ1\theta_1 perturbation of 0.1°. The error at tt was calculated with error(t)=∣θperturbed−θactual∣error(t) = \lvert\theta_{perturbed} - \theta_{actual}\rvert. Using this, we were able to find relatively non-chaotic initial conditions of the double pendulum system.

RESULTS AND DISCUSSION

Angle and Trajectory Analysis

Many groups, such as Chattopadhyay et. al [39] and Goodfellow et. al. (2016), focus on using deep neural networks for predicting the dynamics of chaotic systems [39][40]. These studies typically take advantage of single-layer or shallow neural networks for modeling the dynamics of the multi-pendulum system [41]. Our first result was our analysis of the double pendulum solutions [42]. It revealed that when disregarding outside factors such as friction or air resistance, there were certain patterns in terms of chaotic motion based on the initial angles of the pendulum. Specifically for the double pendulum, when [θ1,θ2]≥[89.1089∘,59.6026∘][\theta_1,\theta_2] \ge[89.1089^\circ,59.6026^\circ], the pendulum demonstrated extremely chaotic motion, fully rotating around the origin in multiple instances over time Figure 8A. However, if [θ1,θ2]≥[59.6026∘,89.1089∘][\theta_1,\theta_2] \ge[59.6026^\circ,89.1089^\circ], chaos is a lot less prevalent; the trajectory almost resembles what a single pendulum might look like Figure 8B. From this data, we hypothesized that an increase in θ1\theta_{1} has a greater correlation with chaos in the pendulum than θ2\theta_{2}, verifying this through initial angles that is by testing various different initial angles.

Figure 8. Fig 8A, 8B, 8C, 8D. Relative Chaotic vs. Non-Chaotic Motion: Frictionless; Figure 8A depicts the ODE-RK4 solution graph with initial angles [θ1,θ2]=[90∘,60∘]\left[\theta_1,\theta_2\right]=\left[90^\circ,60^\circ\right]; Figure 8B depicts the ODE-RK4 solution graph with initial angles [θ1,θ2]=[60∘,90∘]\left[\theta_1,\theta_2\right]=\left[60^\circ,90^\circ\right]. These graphs highlight the border point between chaos and stability. Figure 8C depicts the ODE-RK4 solution graph with initial angles [θ1,θ2]=[120∘,0∘]\left[\theta_1,\theta_2\right]=\left[120^\circ,0^\circ\right]. This demonstrates almost unpredictable chaos. By changing the initial angles to [θ1,θ2]=[0∘,120∘]\left[\theta_1,\theta_2\right]=\left[0^\circ,120^\circ\right], we see a significant decrease in chaotic behavior {Figure 8D}. A, B, C, D all suggest that θ1\theta_1 affects chaotic motion in a frictionless system to a greater magnitude than θ2\theta_2.

Double Pendulum plot showing Theta1 and Theta2 over time

Figure 8A.

Double Pendulum plot showing Theta1 and Theta2 over time

Figure 8B.

Double pendulum simulation with friction for Theta1 and Theta2

(a)

Double pendulum simulation with friction for Theta1 and Theta2

(b)

Double pendulum simulation with friction for Theta1 and Theta2

(c)

Figure 9.

9D:

Figure 9 (PDF p. 14). Fig 9A, 9B, 9C, 9D. Relative Chaotic vs. Non-Chaotic Motion In Friction; Figure 9A depicts the ODE-RK4 solution graph with initial angles [θ1,θ2]=[90∘,60∘][\theta_1,\theta_2] = [90^\circ,60^\circ]; Figure 9B depicts the ODE-RK4 solution graph with initial angles [θ1,θ2]≥[60∘,90∘][\theta_1,\theta_2] \ge[60^\circ, 90^\circ]. Figure 9C depicts the ODE-RK4 solution graph with initial angles [θ1,θ2]=[120∘,0∘][\theta_1,\theta_2] = [120^\circ, 0^\circ]. Figure 9D depicts the ODE-RK4 solution graph with initial angles [θ1,θ2]≥[0∘,120∘][\theta_1,\theta_2] \ge[0^\circ, 120^\circ]. All solutions were derived with the application of friction into the differential equations.

The application of friction into the differential equations was done through the use of a damping constant, damping1damping1 for the first pendulum, and damping2damping2 for the second pendulum. By analyzing the initial angles and their solutions, we were able to demonstrate that the opposite trend occurred for friction: instead of θ1\theta_1 having greater impact on the chaos of the system, θ2\theta_2 had a greater impact, shown by the stable pendulum motion in Figure 9A and Figure 9C and the slightly jagged turns at the highest potential energy points of the pendulum in Figure 9B and Figure 9D. In addition, larger angles, as expected, demonstrated the greatest chaos for friction, with figures 9A and 9B demonstrating limited chaotic motion compared to Figures 9C and 9D.

From these graphs, we initially concluded that training our first models with the sliding window approach at initial angle [90∘,90∘][90^\circ, 90^\circ] without friction would be suitable for the double pendulum, as it demonstrated the best balance between chaos and predictability for a large time array (0s to 1000s). Therefore, we based our initial sliding window approach models entirely on the basis of this analysis. However, after realizing that the sliding window approach did not model chaotic motion and after creating the time-step approach, we had to change our time interval and the magnitude of the initial angle. In the sliding window approach, training and testing the model on a 10,000,000 point dataset was extremely time-consuming on our machines with limited computing capability. Training on multiple angles, like in the time-step based approach, with 1000s seconds of motion for each angle, would cost too much computational power and time, so we had to decrease the interval. However, decreasing the time frame allows for less chaos to develop in the system, so we had to increase our initial angle magnitude as well. Our analysis of angle [120∘,0∘][120^\circ, 0^\circ] led us to believe that this angle would develop the greatest chaos in the smaller time frame, and would still be predictable by the models. For the multiple different initial angles, we found that incrementing θ2\theta_2 by 0.1∘0.1^\circ gave us the most predictable deviation from the previous angle. Our final choice of angles for the time-step approach was all angles from [120∘,0∘][120^\circ, 0^\circ] to [120∘,3∘][120^\circ, 3^\circ], with increments of 0.1∘0.1^\circ in θ2\theta_2, giving us 30 initial angles to train on, or 60,000 unique data points for training. We could then test on an “in-between” (e.g. [120∘,2.05∘][120^\circ, 2.05^\circ]) and see how the models would perform on new unseen time-series data. Here, we applied both friction-based and frictionless scenarios with the same initial angles to train on, comparing model performance on relatively non-chaotic motion with friction opposed to highly chaotic motion without friction.

Double Pendulum

RMSE and R² comparison of different models for the time-step based approach on trained initial conditions with friction

Figure 12. Fig. 12: Double Pendulum Friction Model Results Time Step Performances on various ML and NN algorithms on the dataset for timestep based approach. Evaluation metrics used were R2R^2 and RMSE for initial angle [120^\circ, 0^\circ]

RMSE and $R^2$ comparison of different models using the time-step based approach on unknown initial conditions with friction; bar chart comparing LSTM, FFNN, AR, VRNN, GRU, BiRNN, SRNN, and MLP

Figure 13. Double Pendulum Friction Model Results Time Step Based Approach - Testing on unknown initial angle Performances on various ML and NN algorithms on the dataset. Evaluation metrics used were R2R^2 and RMSE for testing on unknown initial angle [120°, 2.05°]

14A:

Figure 14A.

14B:

Trajectory plot with axes ranging from approximately −2 to 2 horizontally and −2 to 1 vertically, showing blue and red double-pendulum trajectories with colored markers

Figure 14B.

VRNN trajectory with friction for time step based approach, panel A

(a)

VRNN trajectory with friction for time step based approach, panel B

(b)

Figure 16. Figure 16A, 16B; VRNN Trajectory WITH FRICTION for Time step Based Approach: 10s of trajectory. Figure A is the trajectory for the model that was trained on many different initial angles and tested on an initial angle that it was not trained on [120°, 2.05°]. Figure B is the trajectory for the model that was trained on many different initial angles and tested on one of its trained angles [120°, 0°]. In both figures, the orange is the actual trajectory and blue is the predicted trajectory.

When tested with the sliding window approach at an initial angle [90°, 90°], the best-performing model overall was LSTM with the lowest RMSE value of 5.75E−55.75\mathrm{E}^{-5} Figure 10A. This was followed by GRU with an RMSE of 6.2E−46.2\mathrm{E}^{-4} Figure 10A. The worst models were RF and AR with RMSE values of 1.5E−21.5\mathrm{E}^{-2} and 2.9E−22.9\mathrm{E}^{-2}, respectively Figure 10A. Typically, all the neural networks performed better than all of the simple machine learning models, which is not a surprise as Recurrent Neural Networks are known to be best suited for time-series problems such as chaotic mechanical systems.

Figure 10. Fig. 10: Double Pendulum Sliding Window Model Results Performances on various ML and NN algorithms on the dataset. Evaluation metrics used were R2R^2 and RMSE for initial angle [90∘,90∘][90^\circ, 90^\circ]. All results did not take the aspect of friction into consideration.

However, once we realized the limited capabilities of the sliding window approach in modeling chaos, we switched to the new time-step based approach, where we actually got similar results.

The multiplicative increased data wrought from each set of initial angles caused decreased accuracy for specific pre-trained initial angles than for the sliding window approach. However, the second implementation allowed us to predict in-between unknown angles.

First, testing on an angle that was trained on ([120∘,0∘][120^\circ, 0^\circ]) without friction, the best model was LSTM again with an RMSE of 2.7E−22.7\mathrm{E}^{-2} and an R2R^2 value of 0.991 Figure 11, beating all the models with superior performance and best trajectory match between predicted and actual Figure 14A. GRU and VRNN also came close to LSTM in terms of results, with an RMSE of 3.84E−23.84\mathrm{E}^{-2} and 4.02E−24.02\mathrm{E}^{-2}, respectively.

RMSE and R² comparison of different models for the time-step based approach on trained initial conditions without friction

Figure 11. Fig. 11: Double Pendulum Time-Step No Friction Model Results Performances on various ML and NN algorithms on the dataset. Evaluation metrics used were R2R^2 and RMSE for initial angle [120^\circ, 0^\circ]. All results did not take the aspect of friction into consideration.

Image not extracted

Figure 14. Figure 14A, 14B; LSTM Trajectory WITHOUT FRICTION for Time Step Based Approach: 10s of trajectory. Figure A is the trajectory for the model that was trained on many different initial angles and tested on an initial angle that it was already trained on [120∘120^\circ, 0∘0^\circ]. Figure B is the trajectory for the model that was trained on many different initial angles and tested on an “in-between” of [120∘120^\circ, 2.05∘2.05^\circ]. In both trajectory visualizations, red is the actual trajectory and blue is the predicted trajectory.

These results, although promising, are not necessarily modeling real world chaotic systems. This prompted a change, shifting to friction based systems. This was mainly driven by the need to stabilize chaotic trajectories and improve model performance on unseen conditions. Friction inherently dampens chaotic motion, reducing sensitivity to small perturbations in initial conditions. This allowed models to learn patterns that it might not have caught in a frictionless system, due to its extreme variability. In addition, most real world chaotic systems have damping or dilation of chaos over time: weather catastrophes, medicinal effects, and even the stock market with bigger companies. So including friction in our paper makes the results amenable to the physics oriented scope and other real world chaotic systems.

With friction applied, LSTM was the best model at an initial angle [120∘,0∘][120^\circ, 0^\circ] with an RMSE of 9.5E−39.5\mathrm{E}^{-3} and an R2R^2 value of 0.9981, beating the GRU by a small margin (1.496E−21.496\mathrm{E}^{-2} RMSE; 0.9968 R2R^2 score). The trajectories between the predicted pendulum and actual pendulum are almost indistinguishable for LSTM on a friction-based system Figure 15A. This tells us that LSTM is the best model at predicting chaotic systems with two variables, followed by GRU and VRNN. The worst model was AR for a friction-based system, with an RMSE of 1.14E−11.14\mathrm{E}^{-1}, which suggests that it is not effective in predicting chaotic systems with two variables. A greater number of variables would increase chaos in the system, so AR is not a suitable model to predict chaotic motion.

LSTM trajectory with friction, panel A

(a)

(b)

Figure 15. Figure 15A, 15B; LSTM Trajectory WITH FRICTION for Time step Based Approach: 10s of trajectory. Figure A is the trajectory for the model that was trained on many different initial angles and tested on an initial angle that it was already trained on [120∘120^\circ, 0∘0^\circ]. Figure B is the trajectory for the model that was trained on many different initial angles and tested on an “in-between” of [120∘120^\circ, 2.05∘2.05^\circ]. In both trajectory visualizations, red is the actual trajectory and blue is the predicted trajectory.

The key aspect of the second approach, and the basis of our results, was training on multiple different initial angles to see the capabilities of the model. The aim was to predict on datasets with completely different initial angles, changing patterns and other temporal dependencies that were captured by the model for the angle it was trained on. Since these systems are so sensitive to initial conditions that they are considered “unpredictable,” evaluating the performance of our models in this manner thoroughly models chaotic systems as a whole. We first tried trying to do this without friction. When tested on [120∘,2.05∘][120^\circ, 2.05^\circ], the results were not desirable for frictionless models Figure 14B as shown by the big difference in the trajectories of the pendulum for the first 10 seconds of motion for the LSTM, our best model. Without friction, pendulums are extremely volatile in changing conditions. The results were dramatically worse; the LSTM returned an RMSE of 0.26 and an R2R^2 of 0.23 on the unknown initial angle. Because the frictionless pendulum was so chaotic, we decided that it was not worth continuing to test new models after our best model without friction had done terribly without friction. To accommodate for this, we had to shift to modeling pendulums with friction to understand the capability of the models better in prediction on unknown angles, following the justification for damping in real-world scenarios. For friction, the models performed very differently to how the models performed on a frictionless system, with the LSTM returning an RMSE of 1.5E−21.5\mathrm{E}^{-2} and an R2R^2 of 0.996 on unknown angle [120°, 2.05°]. VRNN was one of the best model on a friction-based system with an already trained angle, so it also performed similar to the LSTM in this friction system on the new initial angle, with an RMSE of 1.6E−21.6_{\mathrm{E}}^{-2} and an R2R^2 of 0.9959 on unknown angle [120°, 2.05°]. These results were highly desirable, and we can see that the trajectories of the actual vs. predicted closely line up in Figure 15B as it did in Figure 15A. This reveals that LSTM and VRNN are useful even for predicting chaotic situations that they have not always been presented before, but only in reasonably chaotic scenarios (friction-based systems) with two variables as opposed to highly chaotic simulations.

Triple Pendulum

For the triple pendulum, we similarly found that different initial angles exhibited different levels of chaos when disregarding outside factors such as friction or air resistance. When [θ1,θ2,θ3]≥[94.737∘,39.131,33.333∘][\theta_1,\theta_2,\theta_3] \ge[94.737^\circ, 39.131, 33.333^\circ], the pendulum demonstrated extremely chaotic behavior. When [θ1,θ2,θ3]≤[64.285∘,72∘,91.818∘][\theta_1,\theta_2,\theta_3] \le[64.285^\circ, 72^\circ, 91.818^\circ], the pendulum demonstrates stability, almost resembling the trajectory of a single pendulum. Similar to the double pendulum, we hypothesized that an increase in θ1\theta_1 has a greater correlation with chaos in the pendulum than θ2\theta_2, which has a greater correlation with chaos than θ3\theta_3.

Figure 17A.

Triple pendulum trajectories for non-chaotic motion

Figure 17B. Figure 17A, 17B; Chaotic vs. Non-Chaotic Motion; Figure 17A depicts the graph with initial angles [θ1,θ2,θ3]=[90∘,60∘,45∘][\theta_1,\theta_2,\theta_3] = [90^\circ, 60^\circ, 45^\circ]; Figure 17B depicts the graph with initial angles [θ1,θ2,θ3]=[45∘,60∘,90∘][\theta_1,\theta_2,\theta_3] = [45^\circ, 60^\circ, 90^\circ]. These graphs highlight the border point between chaos and stability. Figure A depicts a chaotic trajectory, while Figure B depicts a stable trajectory.

From these graphs, we concluded that training our first models at an initial angle [90∘,60∘,45∘][90^\circ, 60^\circ, 45^\circ] would be suitable for the triple pendulum, as it demonstrated the best balance between chaos and predictability.

RMSE and $R^2$ comparison of different models

Figure 18A.

RMSE and $R^{2}$ comparison of different models for the triple pendulum no friction model

Figure 18B. Triple Pendulum No Friction Model Results (Time-Step Approach) Performances on various ML and NN algorithms on the dataset. Evaluation metrics used were R2R^{2} and RMSE for initial angles [90∘90^\circ, 90∘90^\circ, 90∘90^\circ], [80∘80^\circ, 80∘80^\circ, 80∘80^\circ] for the old approach (Figure 18A). Following the methodology of the double pendulum, the initial angles for the new approach started at [120∘120^\circ, 0∘0^\circ, 0.1∘0.1^\circ], and we incremented the third angle by 0.1∘0.1^\circ until we reached 3.0∘3.0^\circ (Figure 18B). All results did not take the aspect of friction into consideration.

RMSE and $R^{2}$ comparison of different models

Figure 18C.

RMSE and $R^2$ Comparison of Different Models

Figure 18D. Fig. 18C, 18D: Triple Pendulum Friction Model Results (Time-Step Approach); Performances on various ML and NN algorithms on the dataset. Evaluation metrics used were R2R^2 and RMSE on the trained angles (18C) and on the unknown angle [120°, 0°, 2.05°] (18D). Following the methodology of the double pendulum, the initial angles for the new approach started at [120°, 0°, 0.1°], and we incremented the third angle by 0.1° until we reached 3.0°. All of the results considered the aspect of friction.

Figure 19A.

19B:

LSTM and GRU trajectory with friction for time step based approach

Figure 19B. Figure 19A, 19B; LSTM and GRU Trajectories WITH FRICTION for Time Step Based Approach: 10s of trajectory. Figure A is the trajectory for the LSTM model that was trained on many different initial angles and tested on an “in-between” angle [120°, 0°, 2.05°]. Figure B is the trajectory for the GRU model that was trained on many different initial angles and tested on an “in-between” of [120°, 0°, 2.05°]. In both trajectory visualizations, red is the actual trajectory and blue is the predicted trajectory.

With the sliding window approach, the Vanilla RNN performed the best out of all the other models, with an R2R^2 score of 0.999999 (perfect data fit), and an RMSE of 1.825E−31.825E^{-3}. We concluded that the Vanilla RNN is the best model to predict the trajectory of the triple pendulum. Contrary to the double pendulum, where the LSTM produced the best results, we hypothesized that the Vanilla RNN captured the motion of the triple pendulum better because the triple pendulum is more sensitive to changes in initial conditions that lead to rapid divergences in trajectories. As the complexity of a chaotic system increases (e.g., moving from a double to a triple pendulum), the ability to predict the system's state over time becomes more difficult. Our data shows that double-pendulum models produced more effective results than triple pendulum results. This increased unpredictability in the triple pendulum’s dynamics aligns well with the Vanilla RNN, which can adjust rapidly to local fluctuations.

For our time-step based approach, we trained the models on the angles: [θ1,θ2,θ3]=[120∘,0∘,0.1∘],[120∘,0∘,0.2∘],…[120∘,0∘,3.0∘][\theta_1, \theta_2, \theta_3] = [120^\circ, 0^\circ, 0.1^\circ], [120^\circ, 0^\circ, 0.2^\circ], \ldots[120^\circ, 0^\circ, 3.0^\circ], varying the third angle throughout the range [0.1,3.0][0.1, 3.0] with a step of 0.1. With the time-step-based approach, the Gradient Recurrent Network (GRU) performed the best, although not by a lot. With an R2R^2 score of 0.995333 and an RMSE of 1.688E−11.688E^{-1}, the GRU slightly outperformed the LSTM (R2R^2 score of 0.9926 and an RMSE of 2.175E−12.175E^{-1}), using RMSE as our main metric comparison. Based on the GRU's desirable performance, we can conclude that the GRU effectively captures motion given variations in the sets of initial angles in the triple pendulum. These results show that the GRU can generalize well to new, unseen configurations, allowing it to predict the chaotic motion of the triple pendulum.

Now, we moved to test this new approach with friction. To ensure all the models were given the same testing conditions, we decided to test all the models on the angles [θ1,θ2,θ3]=[\theta_1, \theta_2, \theta_3] = [120∘,0∘,2.05∘][120^\circ, 0^\circ, 2.05^\circ]. This ensured that the models were not tested on the trained angle, while still given the opportunity to be trained on the relative “neighborhood” around the angle.

When we tested this new approach with friction, the GRU also performed the best, with an R2\mathrm{R}^{2} score of 0.98823 and an RMSE of 9.112E−39.112\mathrm{E}^{-3}. The LSTM (R2\mathrm{R}^{2} score of 0.99746 and an RMSE of 1.081E−11.081\mathrm{E}^{-1}) and the Bidirectional RNN (R2\mathrm{R}^{2} score of 0.99372 and an RMSE of 1.73E−11.73\mathrm{E}^{-1}) performed well too. We thus conclude that the GRU will best predict the triple pendulum motion when a coefficient like friction is added, diminishing the chaotic motion.

Overall, our results for the triple pendulum – both the old and the new approach – validates our conclusion from the double pendulum. typically, neural networks performed better than simple machine learning models. For the triple pendulum specifically, GRU is known to be best suited in predicting chaotic mechanical systems, both with the time-step approach and the sliding window approach.

Stability Analysis

After coding all our models, we moved on to stability analysis of the pendulum to get a final grasp of its dynamics. At conditions where the Lyapunov exponent was >0> 0, the system displays chaotic motion as small perturbations in initial conditions are exponentially deviating from the non-perturbed trajectory. Figure 22

Heatmaps of all Root Mean Square Error values for the models and networks coded with the new method

Figure 22. Heatmaps of all Root Mean Square Error (RMSE) values for the models and networks coded with the new method. The lighter the box, the greater the RMSE and the darker the box, the lower the RMSE.

Heatmap of Lyapunov Exponents with Initial Conditions

Figure 20. Chaoticness at Initial Conditions: Figure 20 depicts the heatmap which shows the Lyapunov exponent evaluated at various initial conditions for 10 seconds, or 5000 timesteps. The exponent qualitatively assesses the sensitivity to initial conditions: high positive values represent conditions where small deviations grow rapidly, suggesting chaotic behavior, while values close to 0 or negative suggest stability or return to the setpoint. For example, the error of a double pendulum with initial conditions [150.1,75,0][150.1,75, 0] deviates from one with [150,75,0][150, 75, 0] at a rate of e8.241000=1.008274e^{\frac{8.24}{1000}} = 1.008274, or 0.8274% increase after each timestep. This behavior indicates that the system will exponentially move away from the trajectory. On the

other hand, the least chaotic point [165.1,15,0][165.1, 15, 0] deviates at a rate of 0.998, or a -0.1998% increase per timestep, showing that the point tends towards the setpoint’s trajectory over time.

In order to assess the stability of the double pendulum system, we found the eigenvalues[40] for the Jacobian matrix of the differential equations at the equilibrium points, namely [0,0][0, 0] and [180∘,180∘][180^\circ, 180^\circ]. At [0,0][0, 0], the eigenvalues were [0+5.7873513j,0−5.7873513j,0+2.3971994j,0−2.3971994j][0+5.7873513j, 0-5.7873513j, 0+2.3971994j, 0-2.3971994j], indicating that this point is an undamped oscillator based on Daniel Katzman et al. [10]. Similar analysis at [180,180][180, 180] gives [−5.7873513,−2.3971994,5.7873513,2.3971994][-5.7873513, -2.3971994, 5.7873513, 2.3971994], indicative of an unstable saddle point [10]. This behavior can be further seen with its phase portrait and response vs. time. Figure 23

Figure 21. Fig 21A, 21B. Stability of Equilibrium Points: Figure 21A depicts the phase portrait and angular displacement over time for [0,0][0, 0] after an initial perturbation of u1u_1 and u2u_2 of ±1\pm1 degree/s; Figure 21B depicts the phase portrait and angular displacement over time for [180∘,180∘][180^\circ, 180^\circ] with a similar perturbation. These graphs highlight how variations in the initial conditions from stable and unstable equilibrium points can lead to divergent results. That is either returning to the setpoint or not reaching the setpoint again, shows the chaotic nature of the system.

Understanding the stability of the multi-pendulum system allows us to assess the reliability of our models as stable initial conditions are able to be predicted within a reasonable error for longer than unstable ones. Additionally, through the Jacobian matrix in the eigenvalue stability, we were able to highlight how parameters affect one another and the overall stability of the system using the partial derivatives of each variable [θ1,ω1,θ2,ω2][\theta_1, \omega_1, \theta_2, \omega_2].

Discussion

Chaotic systems shape the world around us, from climate change and earthquakes to disease spread, financial markets, and urban development. These networks exhibit extreme sensitivity to initial conditions, where a small change can lead to drastically different outcomes. Chaotic systems can be predicted through many computational methods: using differential equation modeling and solving using integration techniques, simulations and real-time data, etc. However, these methods are often extremely time-consuming and inefficient. By harnessing the power of machine learning and deep learning, we can expedite the process of being able to predict these scenarios, and apply it to many fields. In this project, we successfully prove the power of machine learning models in predicting these chaotic systems accurately and efficiently, revolutionizing the field. In addition, we created a novel method of solving the problem efficiently and thoroughly: the time-step based approach, one of the most significant contributions of this paper. Moreover, our works focus on friction models that model chaotic systems in a realistic pattern of damping over time.

Overall, our results from this project suggest that LSTM was the best model to successfully predict chaotic systems with up to 3 features, even ones it hasn’t seen before in moderately chaotic scenarios. This can be attributed to LSTM’s gated architecture, which allows it to retain critical temporal information while ignoring irrelevant data, making it well-suited for modeling the sensitive dependencies inherent in chaotic systems. In all tests, AR underperformed the most out of the models, which can be attributed to its reliance on linear dependencies, making it inadequate in predicting the non-linear dynamics of a chaotic system. In a broader perspective, most Recurrent Neural Networks like LSTM, VRNN and BIRNN and Convolutional Neural Networks like GRU performed exceptionally, because of their ability to retain time series information, making them useful in predicting other chaotic systems in addition to the double pendulum. Feed Forward Networks like the Autoregressive Model and FFNN underperformed due to their linear dependencies, showing their inability to accurately predict chaotic systems.

For future studies involving investigating machine learning in chaotic systems, we can claim that the majority of systems with two or three variable features using the LSTM model for predictions will achieve an RMSE anywhere between 0.009 and 0.1 for any tests with the right preprocessing. This will revolutionize previously time-consuming algorithms, allowing for the discovery of further discovery of machine learning applications.

While our study demonstrates the capability of machine learning models in predicting chaotic systems, the following limitations exist that put into perspective our result findings.

  1. Limitation of Scope: While our results are significant for proving the power of machine learning in predicting every-day chaotic systems, they are limited based on the number of variables at play. In this study, we can claim that AI and machine learning will accurately predict chaotic systems with up to 3 changing features, as we were able to predict both the double and triple pendulum accurately. However, adding more features could drastically decrease accuracy, but better data normalization for certain systems could improve the accuracy.

  1. Computational Constraints: Given the limited computational resources, we narrowed the time-step approach down to a 10-second prediction interval. While this gave ample data for modeling short-term chaos, longer intervals may have been useful in uncovering more dynamics and providing better generalization estimates of the model. Future research would harness more computational power to mitigate any bias due to this.

  1. Model Biased to Noise-Free Data: The current work has relied purely on simulated data using the solution of the equations of motion via ODE-RK4. This synthetic data is generated in a perfect, idealized environment with RK4 without any noise. Real chaotic systems are susceptible to noise, various kinds of perturbations, and measurement inaccuracies. Future research would include testing the performance of the models on noisy data to better relate model predictions to real chaotic conditions and scenarios. This research would also include getting the equipment to track data for a physical multi-pendulum system, increasing the robustness of our models and reducing possible artifactual results due to simulated data.

  1. Uniform Hyperparameters: To ensure fairness and keep the results of our models comparable, all of our models shared identical hyperparameters. However, doing this may result in the best models not achieving high performance. Most models, in particular the LSTM and GRU architectures, always require intensive tuning over learning rates, dropout rates, and sizes of hidden layers to achieve optimal performance. The subsequent studies shall model-specific hyperparameter optimizations in order to achieve a more generalized model with increased accuracy.

  1. Many Assumed Constants in The Datasets:

There are some assumptions made in the generation of our datasets, and the data may not represent the full spectrum of chaotic systems found in the real world. The variation in the system parameters such as mass or gravitational fields would be a goal for future research on this topic.

By overcoming the above limitations, future studies could extend our work toward more robust and general models applicable for chaos systems. Based off of our research, some specific models stick out

CONCLUSION

Time Step Approach for Frictionless System Trained Angle RMSE Heatmap

Figure 1 (PDF p. 30).

Our study aimed to showcase the power of machine learning techniques in predicting chaotic and almost “unpredictable” systems, specifically the multi-pendulum system. For double pendulum, with our sliding window approach, LSTM had the highest R2R^2 of 0.999998 and an RMSE of 5.75E−55.75E^{-5}. However, the sliding window approach had significant limitations when applied to chaotic systems. It relied on short-term data dependencies, which are insufficient and impractical for modeling the long-term evolution of chaotic trajectories. Consequently, the models, including the LSTM, were merely fitting an erratic curve rather than capturing the underlying chaotic dynamics. This realization prompted the transition to the time-step-based approach. With this new approach for double pendulum, we tested both frictionless and friction-based systems. In a frictionless system on the trained initial angle [120°, 0°], LSTM had the highest R2R^2 of 0.991 and the lowest RMSE of 2.7E−22.7E^{-2}. This was followed closely by GRU with an R2R^2 of 0.982 and an RMSE of 3.84E−23.84E^{-2}. In a friction-based system on the trained angle [120°, 0°], LSTM achieved the highest R2R^2 of 0.9987 and the lowest RMSE of 9.546E−39.546E^{-3}. Following close second was GRU with an R2R^2 of 0.9968 and an RMSE of 1.496E−21.496E^{-2}. When testing on the new chaotic initial angle [120°, 2.05°] without friction, LSTM performed the best with an RMSE of 3.1E−13.1E^{-1}, matching the general patterns of the actual angles of the double pendulum but unable to predict the trajectory accurately as intended due to the unpredictable chaos of the pendulum at this point. When tested on the unknown angle [120°, 2.05°] with friction, LSTM achieved the best result of 1.526E−21.526E^{-2} and an R2R^2 of 0.9966.

For the triple pendulum, with our sliding window approach, the Vanilla RNN produced the most effective results, with an R2R^2 score of 0.999999 and an RMSE of 1.825E−41.825E^{-4}. With the time-step based approach the GRU had the best results, with an R2R^2 score of 0.995333 and an RMSE of 1.688E−21.688E^{-2}. When we tested the time-step based approach with friction on the untrained angle, the GRU also produced the best results, an R2R^2 score of 0.99909 and an RMSE of 6.497E−36.497E^{-3}. Overall, we conclude that based on the results of the novel time-step based approach, recurrent neural networks like the GRU and the LSTM are successfully able to predict the chaotic motion of the triple pendulum systems.

Further future research directions include applying these models and evaluating their performance on other chaotic systems such as the Lorenz Attractor [43] (typical model of chaos), and more industry based systems such in stock market predictions [44], weather pattern predictions [45], etc. Our research is extremely relevant to the rising interest in Artificial Intelligence (AI) to allow for faster and less intensive computation of complex tasks. Chaotic systems are prevalent in everyday life; being able to accurately predict these systems could revolutionize the field of Artificial Intelligence.

CRediT authorship contribution statement

Vasista Ramachandruni: Conceptualization, Methodology, Validation, Investigation, Formal Analysis, Data Curation, Writing - Original Draft, Visualization, Project Administration. Sai Hruday Reddy Nara: Conceptualization, Methodology, Validation, Investigation, Formal Analysis, Data Curation, Writing - Original Draft, Visualization. Geo Lalu: Methodology, Data Curation, Validation, Investigation, Formal Analysis, Writing - Original Draft, Visualization. Sabrina Yang: Methodology, Validation, Investigation, Formal Analysis, Writing - Original Draft, Visualization. Mohit Ramesh Kumar: Methodology, Validation, Investigation, Formal Analysis, Writing - Original Draft, Visualization. Aarjav Jain: Methodology, Validation, Investigation, Formal Analysis, Writing - Original Draft, Visualization. Pratham Mehta: Validation, Investigation, Formal Analysis, Writing - Original Draft. Hankyu Koo: Validation, Investigation, Formal Analysis, Visualization. Jason Damonte: Investigation, Formal Analysis, Writing - Original Draft. Marx Akl: Supervision, Conceptualization, Resources, Writing - Review and Editing.

Declaration of Competing Interest

The authors declare that they have no conflicts of interest.

Glossary

Glossary located in Electronic Supplementary Information.

Data Availability

Data is available in the GitHub repositories linked at https://github.com/CTNN-ASDRP.

Acknowledgments

We thank the Aspiring Scholars Directed Research Program (ASDRP) for providing us with access to a research poster printer for symposiums and research conferences. We would like to thank Ved Vyas for the autoregressive model for double pendulum and Feed Forward NN with friction for double pendulum. We would also like to thank Joshveer Singh for training random forest and LSTM models for triple pendulum and writing summaries of potential sources. We thank Arnav Bansal for his contributions on the early work of the double pendulum friction models and for initial analysis of the double pendulum. We thank Huifang Qin, Professor of Computer Science and Artificial Intelligence at the ASDRP Department of Computer Science and Engineering and Professor Robert Downing, Chair Emeritus of the ASDRP Department of Computer Science and former Professor of Computer Science and Astrophysics and Engineering for reviewing the contents of this manuscript and offering valuable feedback.

References

  1. [2]Runge-Kutta Method - an overview | ScienceDirect Topics. (n.d.). Www.sciencedirect.com. https://www.sciencedirect.com/topics/mathematics/runge-kutta-method
  2. [3]Deep Learning | Introduction to Long Short Term Memory. (2019, January 16). GeeksforGeeks. https://www.geeksforgeeks.org/deep-learning-introduction-to-long-short-term-memory/
  3. [4]RNN — PyTorch 2.0 documentation. (n.d.). Pytorch.org. https://pytorch.org/docs/stable/generated/torch.nn.RNN.html
  4. [5]Shen, G., Tan, Q., Zhang, H., Zeng, P., & Xu, J. (2018). Deep Learning with Gated Recurrent Unit Networks for Financial Sequence Predictions. Procedia Computer Science, 131, 895–903. https://doi.org/10.1016/j.procs.2018.04.298
  5. [6]Bishop, R. (2015). Chaos (Stanford Encyclopedia of Philosophy). Stanford.edu. https://plato.stanford.edu/entries/chaos/
  6. [7]Climate of chaos: Why warming makes weather less predictable | Stanford Doerr School of Sustainability. (2021, December 14). Sustainability.stanford.edu. https://sustainability.stanford.edu/news/climate-chaos-why-warming-makes-weather-less-predictable
  7. [8]Halloran, J. P., & Ahmet Erdemir. (2011). Adaptive Surrogate Modeling for Expedited Estimation of Nonlinear Tissue Properties Through Inverse Finite Element Analysis. Annals of Biomedical Engineering, 39(9), 2388–2397. https://doi.org/10.1007/s10439-011-0317-2
  8. [9]Kliuchnikov, I., Sigova, M., & Beizerov, N. (2017). Chaos Theory in Finance. Procedia Computer Science, 119, 368–375. https://doi.org/10.1016/j.procs.2017.11.196
  9. [10]Sensitivity to Initial Conditions in Chaos - Wolfram Demonstrations Project. (n.d.). Demonstrations.wolfram.com. https://demonstrations.wolfram.com/SensitivityToInitialConditionsInChaos/
  10. [11]Arslan Basharat, and Mubarak Shah. Time Series Prediction by Chaotic Modeling of Nonlinear Dynamical Systems. 1 Sept. 2009, https://doi.org/10.1109/iccv.2009.5459429
  11. [12]Flight chaos made worse by engineer delay, report finds. (2024, March 14). Www.bbc.com. https://www.bbc.com/news/business-68563068
  12. [13]LSTM — PyTorch 2.5 documentation. (2023). Pytorch.org. https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html
  13. [14]Jain, H., Ranjan, A., & Gupta, K. (2013). Analysis of Chaos in Double Pendulum. 26, 171–176. https://doi.org/10.1109/icetet.2013.50
  14. [15]cloud4science. (2021, February 28). Deep Learning with Real Data and the Chaotic Double Pendulum. Cloud Computing for Science and Engineering. https://cloud4scieng.org/2021/02/28/deep-learning-with-real-data-and-the-chaotic-double-pendulum/DOI
  15. [16](2019). Motion of a Triple Rod Pendulum. Authorea. https://www.authorea.com/users/259349/articles/412491-motion-of-a-triple-rod-pendulum
  16. [18]D’Alessio, Serge. “An Analytical, Numerical and Experimental Study of the Double Pendulum.” European Journal of Physics, vol. 44, no. 1, 3 Nov. 2022, p. 015002, https://doi.org/10.1088/1361-6404/ac986b
  17. [19]numpy.array — NumPy v2.1 Manual. (2024). Numpy.org. https://numpy.org/doc/2.1/reference/generated/numpy.array.html
  18. [20]Deep Learning. (2024, June 18). MIT Press. https://mitpress.mit.edu/9780262035613/deep-learning/
  19. [21]Klinkachorn, S., & Parmar, J. (n.d.). Evaluating Current Machine Learning Techniques On Predicting Chaotic Systems. https://cs229.stanford.edu/proj2019spr/report/38.pdf
  20. [22]HÜBINGER, B., et al. “APPROACHING NONLINEAR DYNAMICS by STUDYING the MOTION of a PENDULUM III: PREDICTABILITY and CONTROL of CHAOTIC MOTION.” International Journal of Bifurcation and Chaos, vol. 04, no. 04, Aug. 1994, pp. 773–784, https://doi.org/10.1142/s0218127494000575
  21. [23]Binghamton University, State University of New York. (2024). Lyapunov Exponent. In Introduction to the Modeling and Analysis of Complex_Systems. https://math.libretexts.org/@go/page/7819
  22. [24]Bejani, Mohammad Mahdi, and Mehdi Ghatee. “A Systematic Review on Overfitting Control in Shallow and Deep Neural Networks.” Artificial Intelligence Review, vol. 54, no. 8, 3 Mar. 2021, pp. 6391–6438, https://doi.org/10.1007/s10462-021-09975-1
  23. [25]sklearn.ensemble.RandomForestClassifier — scikit-learn 0.20.3 documentation. (2018). Scikit-Learn.org. https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
  24. [26]sklearn.neural_network.MLPRegressor — scikit-learn 0.21.3 documentation. (2010). Scikit-Learn.org. https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html
  25. [27]Bidirectional Recurrent Neural Network. (2023, June 8). GeeksforGeeks. https://www.geeksforgeeks.org/bidirectional-recurrent-neural-network/
  26. [28]SGDRegressor. (2024). Scikit-Learn. https://scikit-learn.org/1.5/modules/generated/sklearn.linear_model.SGDRegressor.html
  27. [29]Autoregressive Model - an overview | ScienceDirect Topics. (n.d.). Www.sciencedirect.com. https://www.sciencedirect.com/topics/mathematics/autoregressive-model
  28. [30]Novac, Ovidiu-Constantin, et al. “Analysis of the Application Efficiency of TensorFlow and PyTorch in Convolutional Neural Network.” Sensors, vol. 22, no. 22, 16 Nov. 2022, p. 8872, https://doi.org/10.3390/s22228872
  29. [31]Chu, C.-S. J. (1995). Time series segmentation: A sliding window approach. Information Sciences, 85(1-3), 147–173. https://doi.org/10.1016/0020-0255(95)00021-g
  30. [32]Kaiser, E., Kutz, J. N., & Brunton, S. L. (2018). Sparse identification of nonlinear dynamics for model predictive control in the low-data limit. Proceedings of the Royal Society a Mathematical Physical and Engineering Sciences, 474(2219), 20180335–20180335. https://doi.org/10.1098/rspa.2018.0335
  31. [33]Cai, Y., Zhao, R., Wang, H., Chen, L., Lian, Y., & Zhong, Y. (2023). CNN-LSTM Driving Style Classification Model Based on Driver Operation Time Series Data. IEEE Access, 11, 16203–16212. https://doi.org/10.1109/access.2023.3245146
  32. [34]Laguna, J. O., Olaya, A. G., & Borrajo, D. (2011). A Dynamic Sliding Window Approach for Activity Recognition. Lecture Notes in Computer Science, 219–230. https://doi.org/10.1007/978-3-642-22362-4_19
  33. [35]Wang, Yangeng, et al. “Sensitive Dependence on Initial Conditions between Dynamical Systems and Their Induced Hyperspace Dynamical Systems.” Topology and Its Applications, vol. 156, no. 4, Feb. 2009, pp. 803–811, https://doi.org/10.1016/j.topol.2008.10.014
  34. [36]“Making Sense of Data II.” Google Books, 2024, books.google.com/books?hl=en&lr=&id=kbrIAgAAQBAJ&oi=fnd&pg=PA19&dq=which +allows+the+model+to+predict+for+a+specific+point+in+time
  35. [37]Sangiorgio, M., Dercole, F., & Giorgio Guariso. (2021). Deep Learning in Multi-step Prediction of Chaotic Dynamics. SpringerLink. https://doi.org/10.1007-978-3-030-94482-7
  36. [38]Jiménez-López, Javier, and Víctor J García-Garrido. “Chaos and Regularity in the Double Pendulum with Lagrangian Descriptors.” International Journal of Bifurcation and Chaos, 28 Nov. 2024, https://doi.org/10.1142/s0218127424502018
  37. [39]Chattopadhyay, A., Hassanzadeh, P., & Subramanian, D. (2020). Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network. Nonlinear Processes in Geophysics, 27(3), 373–389. https://doi.org/10.5194/npg-27-373-2020
  38. [40]Deep Learning. (2016). Deeplearningbook.org. https://www.deeplearningbook.org/
  39. [41]Bianchini, Monica, and Franco Scarselli. “On the Complexity of Neural Network Classifiers: A Comparison between Shallow and Deep Architectures.” IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 8, Aug. 2014, pp. 1553–1565, https://doi.org/10.1109/tnnls.2013.2293637
  40. [42]University of Michigan. (2024). Using eigenvalues and eigenvectors to find stability and solve ODEs. In Chemical Process Dynamics and Controls. https://eng.libretexts.org/@go/page/22502
  41. [43]Étienne Ghys. (2013). The Lorenz Attractor, a Paradigm for Chaos. Progress in Mathematical Physics, 1–54. https://doi.org/10.1007/978-3-0348-0697-8_1
  42. [44]Igor Klioutchnikov, Mariia Sigova, & Beizerov, N. (2017). Chaos Theory in Finance. Procedia Computer Science, 119, 368–375. https://doi.org/10.1016/j.procs.2017.11.196
  43. [45]S, P. J., & M, S. A. (2024). Nonlinear Dynamics and Chaos: Applications for Prediction of Weather and Climate. ArXiv.org. https://arxiv.org/abs/physics/0104056

Paper details

vlnc:W_nku3f6jq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents