WorksPhysics and Astronomy
Probabilistic Forecasting for Dynamical Systems with Missing or Imperfect Data
Abstract
The modeling of dynamical systems is essential in many fields, but applying machine learning techniques is often challenging due to incomplete or noisy data. This study introduces a variant of stochastic interpolation (SI) for probabilistic forecasting, estimating future states as distributions rather than single-point predictions. We explore its mathematical foundations and demonstrate its effectiveness on various dynamical systems, including the challenging WeatherBench dataset.
Introduction
The modeling of dynamical systems is of paramount importance across numerous scientific and engineering disciplines, including weather prediction, climate science, finance, and biology [7] (Tu 2012). Traditional methods for predicting the behavior of these systems often rely on explicit mathematical models derived from known physical laws. Such approaches are well summarized in Reich and Cotter [2015] and typically employ a combination of numerical methods, Bayesian inference, and sensitivity analysis. However, these methods can be limited by their underlying assumptions and the complexity of the systems they aim to model [19, 18]. In particular, they require a high-fidelity model with sufficiently high resolution and accurate input data. Furthermore, such approaches often demand significant computational resources, making them difficult to compute within a reasonable time frame [Moin and Mahesh, 1998, Lynch, 2008, Wedi et al., 2015b].
In recent years, machine learning (ML) has emerged as a powerful tool for predicting dynamical systems, leveraging its ability to learn patterns and relationships from data without requiring explicit knowledge of the underlying physical laws [17, 6] (Pathak et al. 2022b; Lam et al. 2022). ML approaches are well suited for this task, as they can automatically identify patterns in high-dimensional data without the need for a precise mathematical formulation of all physical processes. This is particularly advantageous when complete physical models are either unknown or computationally intractable. Furthermore, ML can efficiently capture multi-scale phenomena while maintaining computational efficiency during inference.
The application of machine learning models to dynamical systems spans a wide range of fields, including finance, epidemiology, rumor propagation, and predictive maintenance [10]. Our primary motivation is the recent success of machine learning techniques in weather and climate prediction, where vast amounts of meteorological data have been analyzed to forecast weather patterns and climate changes [6] (Pathak et al. 2022b; Lam et al. 2022).
Machine learning models, particularly those based on deep learning, excel at processing large and complex datasets. In the context of dynamical systems, these models can be trained on historical data to learn system behavior and predict future states—essentially integrating the system over a very large time step. Among the most effective models are Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRUs), and Transformers [8] (Vaswani et al. 2017). These models are specifically designed for sequential data and retain memory of previous inputs, making them well-suited for time series prediction in dynamical systems. Convolutional Neural Networks (CNNs) have also been adapted for time series prediction, particularly in spatiotemporal problems [Shi et al., 2015, Li et al., 2020, Liu et al., 2021, Zakariaei et al., 2024].
Most traditional models for dynamical systems aim for deterministic predictions, providing a single forecast of the system’s future state. However, with incomplete or noisy data, such predictions can be misleading or unrealistic due to large errors. In contrast, probabilistic approaches account for this uncertainty by forecasting a distribution of possible future states rather than a single outcome. This shift from point predictions to probability distributions is crucial for many complex systems, where small uncertainties in initial conditions or missing data can lead to vastly different outcomes, and where quantifying uncertainty is essential for decision-making.
Motivation: Machine learning techniques have made significant strides in predicting dynamical systems. However, despite their success, these models face several challenges. One of the primary difficulties lies in the quality, quantity, and completeness of the data [15, 16, 11]. High-quality, large-scale datasets are essential for training accurate machine learning models, yet such data are often low resolution, incomplete, or noisy. As we show here, this violates standard machine learning assumptions. A model may receive incorrect or insufficient input (e.g. low-resolution data), preventing it from making accurate predictions. In such cases, deterministic predictions fail, necessitating the use of probabilistic forecasting, an approach that is intuitive for practitioners in the field. Instead of predicting a single outcome, one also quantifies uncertainty by leveraging historical data [11] (Smith 2024).
One way to address this issue is through ensemble forecasting [Kalnay, 2003]. This technique is widely used in numerical weather prediction (NWP) as it accounts for uncertainties in initial conditions and model physics, producing a range of possible outcomes to enhance forecast reliability and accuracy. However, ensemble forecasting relies on extensive physical simulations and requires significant computational resources.
This study explores a framework for addressing this challenge using recent advances in machine learning. Specifically, we employ a variant of flow matching to map the current state distribution to the future state distribution. Flow matching is a family of techniques, including stochastic interpolation (SI) [2] (Lipman et al. 2022) and score matching [Song et al., 2020], that enables learning a transformation between two distributions. While these techniques are commonly used for image generation by mapping Gaussian distributions to images, they can be readily adapted to match the distribution of the current state to the distribution of future states.
Recent work by Chen et al. [2024] introduced the use of SI and the Föllmer process for probabilistic forecasting via stochastic differential equations. Here, we propose a similar approach to handle missing and noisy data but adopt a deterministic framework instead.
Stochastic interpolation (SI) is a homotopy-based technique used in machine learning and statistics to generate new data points by probabilistically interpolating between known distributions [2] (Lipman et al. 2022). Unlike deterministic methods, which produce a single interpolated value, flow matching introduces randomness into the prediction, enabling the generation of a diverse range of possible outputs.
By leveraging probability distributions and stochastic processes, flow matching based generative models can capture the inherent variability and complexity of real-world data, resulting in more robust and versatile models.
In this work, we apply Stochastic Interpolation (SI) to probabilistic forecasting in dynamical systems and demonstrate that it is a natural choice, particularly when the dynamics are unresolved due to noise or when some states are unavailable. Specifically, we build on previous approaches that transition from Stochastic Differential Equations (SDEs) to Ordinary Differential Equations (ODEs), facilitating easier and more accurate integration [Yang et al., 2024].
As recently shown, selecting an appropriate loss function during training significantly improves inference efficiency [Lipman et al., 2024]. To achieve this, we further employ SI to encode and decode transport maps, introducing physical perturbations to non-Gaussian data samples. This enables sampling from the base state and propagating these samples to future states.
Finally, we demonstrate that our approach is applicable to various problems, with a key application in weather prediction, where it enhances forecasts by quantifying uncertainty.
Mathematical Foundation
In this section, we lay the mathematical foundation of the ideas behind the methods proposed in this work. Assume that there is a dynamical system of the form
Here, is the state vector and is a function that depends on , the time and some parameters . We assume that is smooth and differentiable so that given an initial condition one could compute using some numerical integration method [Stuart and Humphries, 1998]. The learning problem arises in the case when the function or the initial conditions are unknown and we have observations about . In this case, our goal is to learn the function given the observations . There are a number of approaches by which this can be done [8, 17] (Reich and Cotter 2015). Assume first that the data on is measured in small intervals of time that for simplicity we assume to be constant. Then, we can solve the optimization problem
where is some functional architecture that is sufficiently expressive to approximate and are parameters in . If the original function is known, then one can choose and . However, in many cases is only a surrogate of and the parameters are very different than .
The second, less trivial case is when the time interval is very large (relative to the smoothness of the problem). In such cases, one cannot use simple finite difference to approximate the derivative. Instead, we note that since the differential equation has a unique solution and its trajectories do not intersect, we have that
Here is the function that integrates the ODE from time to . We can then approximate using an appropriate functional architecture thus predicting at later time given observation on it in earlier times.
The two scenarios above fall under the case where we can predict the future based on the current values of . Formally, we define the following:
Definition 1 Closed System. Let be the space of some observed data from a dynamical system . We say the system is closed if given we can uniquely recover for all and finite bounded , where is some constant. Practically, given the data, and a constant we can estimate a function such that
Definition 1 implies that we can learn the function in equation 3 assuming that we have sufficient amount of data in sufficient accuracy and an expressive enough architecture to approximates . In this case, the focus of any ML based method should be given to the appropriate architecture that can express perhaps using some of the known structure of the problem. This concept and its limitations are illustrated using the predator prey model. This example demonstrates how a seemingly simple dynamical system can exhibit complex behavior through the interaction of just two variables. While theoretically deterministic, the system can becomes practically unpredictable if measured in regions where the trajectories are very close.
Example 1 The Predator Prey Model [Murray, 2007, Chapter 3]: The predator prey model is
The trajectories for different starting points are shown in Figure 1. Assuming that we record data in sufficient accuracy, the trajectory at any time can be determined by the measurement point at earlier times, thus, justifying the model in
Figure 1. Trajectories for the Predator Pray Model. Note that the trajectories get very close but do not intersect.
equation 3. Furthermore, since the system is periodic, it is easy to set a learning problem that can integrate the system for a long time.
Note however, that the trajectories cluster at the lower left corner. This implies that even though the trajectories never meet in theory, they may be very close, so numerically, if the data is noisy, the corner can be thought of as a bifurcation point.
The system is therefore closed if the data is accurate, however, the system is open if there is noise on the data such that late times can be significantly influenced by earlier times.
The predator prey model draws our attention to a different scenario that is much more common in practice. Assume that some data is missing or, that the data is polluted by noise. For example, in weather prediction, where the primitive equations are integrated to approximate global atmospheric flow, it is common that we do not observe the pressure, temperature or wind speed in sufficient resolution. In this case, having information about the past is, in general, insufficient to predict the future. In this case we define the system as open.
While we could use classical machine learning techniques to solve closed systems it is unreasonable to attempt and solve open systems with similar techniques. This is because the system does not have a unique solution, given partial or noisy data. To demonstrate that we return to the predator prey model, but this time with partial or noisy data.
Example 2 The predator Pray Model with Partial or Noisy Data: *Consider again the predator prey model, but this time assume that we record only. Now assume that we know that at , and our goal is to predict the solution at time , which is very far into the future. Clearly, this is impossible to do. Nonetheless, we can run many simulations where and where is some density. For example, we choose . In this case we run simulations to obtain the results pre-
sented in Figure 2 (left). Clearly, the results are far from unique. Nonetheless, the points at can be thought of as samples from a probability density function. This probability density function resides on a curve. Our goal is now transformed from learning the function in equation (3) to learn a probability density function .

Figure 2. Left: The solution for and at . Right:The solution for where at
A second case that is of interest is when the data is noisy. Consider the case that . In this case we consider and we contaminate it with Gaussian noise with 0 mean and 0.05 standard deviation. Again, we attempt to predict the data at . We plot the results from 1000 simulations in Figure 2 (right). Again, we see that the incomplete information transforms into a density estimation problem.
Example 2 represents a much more realistic scenario, since in reality data is almost always sampled only partially or on low resolution and includes noise.
Comment 1 *It is important to note that in many cases one can transform an open system into a closed one by considering higher order dynamics. For example, in the preditor prey model, one could (formaly) eliminate by setting
and then subsituting into the second system, obtaining a second order system. This however, requires data that is sufficiently accurate to be differentiated more that once.*
Motivated by the two examples above, we now form the learning problem of probabilistic forecasting (see, e.g. Reich and Cotter [2015]).
Definition 2 Probabilistic forecasting. Probabilistic forecasting predicts future outcomes by providing a range of possible scenarios along with their associated probabilities, rather than a single point estimate. To be specific, let the initial vector and assume that is obtained from integrating the dynamical system . Probabilistic forecasting refers to estimating and sampling from the distribution .
This approach acknowledges that a unique prediction may not be attainable but allows for generating samples of future outcomes. In contrast, deterministic prediction assumes a closed system and highly accurate data, making it often inapplicable.
Probabilistic Forecasting and Stochastic Interpolation
In this section, we discuss a version of Stochastic Interpolation (SI). SI is the technique that we use to propagate the uncertainty and also sample from the initial distribution.
To this end, let us consider the case that we observe a noisy or a part of the system. In particular, let be a vector that is obtained from an unknown dynamical system and let
Here is some sampling operator that typically reduces the dimension of and . The distinction between and is important. The vector represents the full state of the system, while represents only partial knowledge of the system. For many systems, it is impossible to observe the complete state space. For example, for local weather prediction we may have temperature stations but not wind stations, where wind speed is essential to model the advection of heat. The inability to obtain information about the full state implies that most likely it is impossible to deliver a deterministic prediction, and thus stochastic prediction is the natural choice.
Given , our goal is to sample from the distribution of observed partial state , given samples from the distribution on the data . Clearly, we do not have a functional expression for neither nor , however, if we are able to obtain many samples from both distributions, we can use them to build such a distribution. This assumption is unrealistic without further assumptions in the general case since we typically have only a single time series. A few assumptions are common and we use them in our experiments. The most common assumptions are, i) The time dependent process is autoregressive. This implies that we can treat every instance (or instances) of the time series as and use the future time series as . Such an assumption is very common although it may be unrealistic, and ii) Periodicity and clustering. For many problems, especially problems that relate to earth science, there is a natural periodicity and data can be clustered using the natural cycle of the year.
Given our ability to obtain data that represents samples from the probability at time 0 and at time , we are able to use the data and provide a stochastic prediction. To this end, we note that stochastic prediction is nothing but a problem of probability transformation which is in the base of mass transport [4]. While efficient solutions for low dimensions problems have been addressed [14] (Villani et al. 2009), solutions for problems in high dimensions are only recently being developed.
One recent and highly effective technique to solve such a problem is stochastic interpolation (SI) [1], which is a technique that belongs to a family of flow matching methods [2] (Lipman et al. 2022; Song et al. 2020). The basic idea of stochastic interpolation is to generate a simple interpolant for all possible points in and . A simple interpolant of this kind is linear and reads
where , , and is a parameter. The points are associated from a distribution that converges to at and to at . In SI one learns (estimates) the velocity
by solving the stochastic optimization problem
Here is an interpolant for the velocity that is given at points . A common model that is used for is a deep neural network. While for simple problems such models can be composed of a simple network with a small number of hidden layers, for complex problems, especially problems that relate to space-time predictions, complex models such as U-Nets are commonly used [Ronneberger et al., 2015].
Assume that the a velocity model is trained and uses it to integrate from time to , that is
Note that we use a deterministic framework rather than a stochastic framework. This allows to incorporate high accuracy integrators that can take larger step sizes. However, this implies that the solution of the ODE given a single initial condition gives a single prediction. In order to sample from the target distribution, we sample from and then use the ODE (equation 10) to push many samples forward. We thus obtain different samples for (see next section) and use them in order to sample from .
Comment 2 Note that the ODE obtained for is not physical. It merely used to interpolate the density from time to . To demonstrate we continue with the predator prey model.
Sampling the Perturbed States
Sampling from the distribution in a meaningful way is not trivial. There are a number of approaches to achieve this goal. A brute force approach can search through the data for so-called similar states, for example, given a particular state we can search other states in the data set such that . While this approach is possible in low dimension it is difficult if not impossible in very high dimensions. For example, for global predictions, it is difficult to find two states that are very close everywhere. To this end, we turn to a machine learning approach that is designed to generate realistic perturbations.
We turn our attention to Variational Auto Encoders (VAE’s). In particular, we use a flow based VAE [12] (Vahdat et al. 2021). Variational Autoencoders are particularly useful. In the encoding stage, they push the data from the original distribution to a vector that is sampled from a Gaussian, . Next, in the decoding stage, the model pushes the vector back to . Since the Gaussian space is convex, small perturbation can be applied to the latent vector which lead to samples from that are centered around .
The difference between standard VAEs and flow based VAEs is that give , VAEs attempt to learn a transformation to a Gaussian space directly. However, as have been demonstrated in Ruto and Haber [2021] VAEs are only able to create maps to a latent state that is similar to but not Gaussian, making it difficult to perturb and sample from them. VAEs that are flow based can generate a Gaussian map to a much higher accuracy. Furthermore, such flows can learn the decoding from a Gaussian back to the original distribution. Flows that can transform the points between the two distributions are sometimes referred to as symmetric flows [2] (Ho et al. 2020).
Such flows can be considered as a special case of SI, an encoder-decoder scheme for a physical state being encoded to a Gaussian state . To this end, let us define the linear interpolant as
The velocities associated with the interpolant as simply
Note that the flow starts at towards at and arrive to at . This is the encoding state. In the second part of the flow we learn a decoding map that pushes the points back to . Note also that is symmetric about which is used in training. Training these models is straight forward and is done in a similar way to the training of our stochastic interpolation model, namely we solve a stochastic optimization problem of the form
Note that we can use the same velocity for the reverse process. Given the velocity we now generate perturb samples in the following way. First, we integrate the ODE to , that is we solve the ODE
This yield the state . We then perturb the state
where and is a hyper-parameter. We then integrate the ODE equation 14 from to starting from obtaining a perturb state . The integration is done in batch mode, that is, we integrate vectors simultaneously to obtain samples from the initial state around . These states are then used to obtain samples from as explained in § 3.
Experiments
Our goal in this section is to apply the proposed framework to a set of various temporal datasets and show how it can be used to estimate the uncertainty in a forecast. We experiment with two synthetic datasets that can be fully analyzed and a realistic dataset for weather prediction. Further experiments on other data sets can be found in the appendix.
Datasets
We now describe the datasets considered in our experiments. Lotka–Volterra predator–prey model: This is a nonlinear dynamical system. It follows the equation 5, where , , and . The initial distribution of states that is Gaussian with a mean of and standard deviation of the final distribution of states is a distribution obtained after fine numerical integration over . Note that the length of the integration time is very long which implies that the output probability space is widely spread which makes this simple problem of predicting the final distribution from an initial distribution very difficult.
MovingMNIST: Moving MNIST is a variation of the well known MNIST dataset [Srivastava et al., 2015]. The dataset is a synthetic video dataset designed to test sequence prediction models. It features 20-frame sequences where two MNIST digits move with random trajectories. Following the setup described in Srivastava et al. [2015], we trained our model on 10,000 randomly generated trajectories and then used the standard publicly available dataset of 10,000 trajectories for testing.
WeatherBench: The dataset used for global weather prediction in this work is derived from WeatherBench Rasp et al. [2020], which provides three scaled-down subsets of the original ERA5 reanalysis data Hersbach et al. [ [20]. The specific variables and model configurations are detailed in appendix B.1. Our forecasting models use the 6-hour reanalysis window as the model time step.
The statistics and configurations for each of the datasets during experiments are mentioned in the appendix B. We have two additional real datasets, Vancouver93 and CloudCast, in the appendix for additional analysis. In the next section we give details for experiments and results on the predator-prey model, the MovingMNIST and the WeatherBench datasets.
Results
Our goal is to learn a homotopy based deterministic functional map for complex timeseries to predict the future state from the current state but with uncertainty. To do so, we evaluate an ensemble of predictions from an ensemble of initial states. The initial states are obtained by using the auto-encoder described in § 4. We then push those states forward obtaining an ensemble of final states. We then report on the statistics of the final states.
The Predator Pray Model: We apply SI to learn a deterministic functional map on predator-prey model 5.1 for a long time integration. The initial state of the data is sampled as a group of noisy initial states, that is, we consider the case that . In this case we consider and we contaminate it with the noise vector, . Our goal is to predict the state, at time . The distribution of 250 noisy points for and the final distributions for using numerical integration and SI can be seen in Figure 3. For this simple case, a multilayer perceptron (MLP) is used to learn the mapping. It can be noticed that SI is able to match with a sample from the final complex distribution, specifically the areas with high probabilities are captured very well. Some more statistical comparisons of the variables’ state are reported in appendix E.1, to support our results. Appendix D shows some results on Vancouver93 where the learned final distribution from simple distribution to a complex conditional distribution is clearly evident.
Figure 3. Comparison of actual final distribution and that obtained using SI on the predator-prey Model. Trajectories for transport learned by SI are in blue. Note that the trajectories are not physical.
The same concept can be extended to high-dimensional spatiotemporal image space instead of vector space without any loss of generality. Like in Lipman et al. [2022], Karras et al. [2024], Bieder et al. [2024], we use a U-Net architecture from Dhariwal and Nichol [2021] to learn the velocity.
Moving MNIST: We use the Moving MNIST to train a network (U-Net) to predict 10 time frames into the future, given 10 past frames. The hyperparameter settings of the U-Net are in appendix G.1. A set of initial states is obtained using the auto-encoder creating 50 initial and final states. Using SI and the algorithm proposed in 3 we sample predictions of final states. The predictions can be observed in Figure 4. The predictions are similar yet different from the true final state. Statistical comparisons of the ensemble predictions using SI with the ensemble of real targets can be seen in § 5.2 and in appendix E.3.
Figure 4. Six of 50 Moving MNIST trajectory predictions obtained using SI and their ensemble mean and standard deviation.
WeatherBench: We now use the WeatherBench dataset to build a forecasting model using SI. We use a similar U-Net as in Moving MNIST for learning the functional mapping. The hyperparameter settings are in appendix G.1. A particular time step is selected from the testset and 77 most similar samples are searched from the testset to collect the initial states, where similarity in measured by the norm. We hence have the ensemble of actual initial and final states of the true data. This ensample is used as ground truth and allows an unbiased testing of our method. Using our technique, we generate an ensemble of initial and final states. A few samples of generated states along with the mean and standard deviation of the ensemble of 78 forecasts can be seen in Figure 5. We use the standard variables U10 and T850 for visual observation here. Some statistical comparisons of our ensemble prediction, variables Z500, T2m, U10 and T850, can be seen in § 5.2 and in appendix E.5.
Figure 5. Two sample stochastic forecasts of U10 and T850 after 2 days obtained using SI and the ensemble mean and standard deviation for 78 forecasts.
Metrics: Below, we describe the metrics used to evaluate the performance of our model across different datasets. For each ensemble of predictions generated by our model, there is a corresponding ensemble of target states. The goal is to ensure that the statistical characteristics of the predicted ensemble closely match those of the target ensemble. The simplest metric for comparing these high-dimensional distributions is the mean score, which represents the average of the pixel values in a state. Similarly, we compute the standard deviation score. For two distributions to be considered similar, these scores should ideally align. More similar the scores are two distributions, more similar are the distributions. Table 1 presents the mean and standard deviation scores for both the target ensemble and predicted ensemble, demonstrating a strong alignment between the two. Inspired by methods used in turbulence flow field analysis, we further compare the predicted and target ensembles by computing their mean state and standard deviation state. To quantify the similarity between these states, we employ standard image comparison metrics: Mean Squared Error (MSE), Mean Absolute Error (MAE), and Structural Similarity Index Measure (SSIM). These metrics are applied to both the ensemble mean states and the ensemble standard deviation states, as shown in Tables 2 and 3. Lower the MSE and MAE are, better the uncertainty is captured. Higher the SSIM, the uncertainty is captured.
As can be readily observed in the tables, our approach provides an accurate representation of the mean and standard deviation, which are the first and second order statistics of each data set. The definition of the metrics are in appendix C.
| Data | True Mean Score | Our Mean Score | True Std Dev Score | Our Std Dev Score |
| Predator–Prey | 7.55e-1 | 7.43e-1 | 1.04 | 1.07 |
| MovingMNIST | 6.01e-2 | 5.58e-2 | 2.21e-1 | 1.87e-1 |
| WeatherBench | 1.36e4 | 1.36e4 | 9.15e1 | 9.66e1 |
Table 1. Comparison of mean score and standard deviation score of ensemble predictions.
| Data | MSE(↓) | MAE(↓) | SSIM(↑) |
| Predator–Prey | 8.0e-3 | 8.8e-2 | NA |
| MovingMNIST | 2.5e-3 | 1.8e-2 | 0.843 |
| WeatherBench | 1.9e4 | 5.2e1 | 0.868 |
Table 2. Accuarcy of our ensemble mean.
Perturbation of Physical States using SI: Perturbation of a physical state in a dynamical system refers to a deliberate natural deviation in a system’s parameters or state variables. Using the algorithm proposed in 4, we generate perturbed states for MovingMNIST, and WeatherBench. All the generated perturbed states are very much realistic. Some sample perturbed states from MovingMNIST can be seen in Figure 6. The physical disturbances like bending or deformation of digits, change in position and shape is clearly captured.
Figure 6. Three random perturbed states of a Moving MNIST sample state.
| Data | MSE() | MAE() | SSIM() |
| Predator–Prey | 2.7e-2 | 1.6e-1 | NA |
| MovingMNIST | 3.6e-3 | 2.5e-2 | 0.744 |
| WeatherBench | 1.1e4 | 3.6e1 | 0.608 |
Table 3. Accuracy of our ensemble standard deviation.
Similarly, we generated perturbed states for WeatherBench as well. Some samples of perturbed states from WeatherBench’s U10 and T850 variables (as these are easier to visualise for humans) can be seen in Figure 7 and Figure 8 respectively. As it can be seen the perturbations are very much physical. Additional studies can be seen in appendix § 13.
Figure 7. Three random perturbed states of a WeatherBench U10 state.
Figure 8. Perturbed states of a WeatherBench T850 state.
Conclusion
Probabilist forecasting is an important topic for many scientific problems. It expands traditional machine learning techniques beyond paired input and outputs, to mapping between distributions. While generative AI has been focused on mapping Gaussians to data, similar methodologies can be applied for the generation of predictions given current states. In this work we have coupled a stochastic interpolation to propagate the current state distribution with an auto-encoder
that allows us to sample from the current distribution. We have shown that this approach can lead to a computationally efficient way to sample future states even for long integration times and for highly non-Gaussian distributions in high dimensions.
Appendix
Siddharth Rout
Eldad Haber
Stéphane Gaudreault
Institute of Applied Mathematics, University of British Columbia, Vancouver, BC, Canada
Department of Earth, Ocean and Atmospheric Sciences, University of British Columbia, Vancouver, BC, Canada
Recherche en prévision numérique atmosphérique, Environnement et Changement climatique Canada, Dorval, QC, Canada
Additional Datasets
Datasets
Vancouver 93 Temperature Trend (Vancouver93): This is a real nonlinear chaotic high dimensional dynamical system, in which only a single state (temperature) is recorded. The daily average temperatures at 93 different weather stations in and around Vancouver, BC, Canada are captured for the past 34 years from sources such as National Oceanic and Atmospheric Administration (NOAA), the Government of Canada, and Germany’s national meteorological service (DWD) through Meteostat’s python library Lamprecht. Essentially, it is a time series of 12,419 records at 93 stations. The complex urban coastal area of Vancouver with coasts, mountains, valleys, and islands makes it an interesting location where forecasting is very difficult than in general Vannini et al. [2012], Leroyer et al. [2014]. The historical temperature alone insufficient to predict the temperature in the future, as it requires additional variables like precipitation, pressure, and many more variables at higher resolution. This makes the dataset fit the proposed framework.
CloudCast Nielsen et al. [2021]: a real physical nonlinear chaotic spatiotemporal dynamical system. The dataset comprises 70,080 satellite images capturing 11 different cloud types for multiple layers of the atmosphere annotated on a pixel level every 15 minutes from January 2017 to December 2018 and has a resolution of pixels ( km).
Datasets
Table 4 describes the statistics, train-test splits, image sequences used for modeling, and frame resolutions.
| Dataset | History | Prediction | |||
| Predator-Prey | 10,000 | 500 | (1, 1, 2) | 1 | 1 |
| Vancouver93 | 9,935 | 2,484 | (93, 1, 1) | 5 (5 days) | 5 (5 days) |
| Moving MNIST | 10,000 | 10,000 | (1, 64, 64) | 10 | 10 |
| CloudCast | 52,560 | 17,520 | (1, 128, 128) | 4 (1 Hr) | 4 (1 Hr) |
| WeatherBench (5.625°) | 324,336 | 17,520 | (48, 32, 64) | 8 (48 Hrs) | 8 (48 Hrs) |
Table 4. Datasets statistics, training and testing splits, image sequences, and resolutions
WeatherBench
The original ERA5 dataset Hersbach et al. [ [20] has a resolution of recorded every hour over almost 40 years from 1979 to 2018 across 37 vertical levels for latitude-longitude grid. The raw data is extremely bulky for running experiments, even with powerful computing resources. We hence the typically used reduced resolutions (: latitude-longitude grid, : latitude-longitude grid) as per the transformations made by Rasp et al. [2020]. We, however, stick to using the configuration set by Nguyen et al. [2023] for standard comparison. The prediction model considers 6 atmospheric variables at 7 vertical levels, 3 surface variables, and 3 constant fields, resulting in 48 input channels in total for predicting four target variables that are considered for most medium-range NWP models like the state of the art IFS\Wedi et al. [2015a] and are often used for benchmarking in previous deep learning work as well like Lam et al. [2023], Pathak et al. [2022a], they are geopotential at 500 hPa (Z500), the temperature at 850 hPa (T850), the temperature at 2 meters from the ground (T2m), and zonal wind speed at 10 meters from the ground (U10). We use a leap period of 6 hours as a single timestep and hence our model takes in 8 timesteps (48 hours or 2 days) to predict for the next 8 timesteps (48 hours or 2 days). According to the same setting used the data from 1979 to 2015 as training, for the year 2016 as validation set and for the years 2017 and 2018 as testing set. The details of the variables considered are in Table 5.
| Type | Variable Name | Abbrev. | ECMWF ID | Levels |
| Static | Land-sea mask | LSM | 172 | |
| Static | Orography | OROG | 228002 | |
| Static | Soil Type | SLT | 43 | |
| Single | 2 metre temperature | T2m | 167 | |
| Single | 10 metre U wind component | U10 | 165 | |
| Single | 10 metre V wind component | V10 | 166 | |
| Atmospheric | Geopotential | Z | 129 | 50, 250, 500, 600, 700, 850, 925 |
| Atmospheric | U wind component | U | 131 | 50, 250, 500, 600, 700, 850, 925 |
| Atmospheric | V wind component | V | 132 | 50, 250, 500, 600, 700, 850, 925 |
| Atmospheric | Temperature | T | 130 | 50, 250, 500, 600, 700, 850, 925 |
| Atmospheric | Specific humidity | Q | 133 | 50, 250, 500, 600, 700, 850, 925 |
| Atmospheric | Relative humidity | R | 157 | 50, 250, 500, 600, 700, 850, 925 |
Table 5. Variables considered for global weather prediction model.
Comparison Metrics
Ensemble Mean and Ensemble Standard Deviation
Let be a set of images, such that , , , , and . An image is defined as
where is called a pixel in an image .
Ensemble mean state (image), , is defined as
Ensemble mean score is defined as
where .
Ensemble standard deviation state (image), , is defined as
Ensemble variance score is defined as
where .
MSE, MAE, SSIM
where:
is the number of images in the dataset,
is the height of the images,
is the width of the images,
is the number of channels (e.g., 3 for RGB images),
is the true pixel value at position , and
is the predicted pixel value at position .
MAX is the maximum possible pixel value of the image (e.g., 255 for an 8-bit image),
MSE is the Mean Squared Error between the original and compressed image.
is the average of ,
is the average of ,
is the variance of ,
is the variance of ,
is the covariance of and ,
and are two variables to stabilize the division with weak denominator,
is the dynamic range of the pixel values (typically, this is 255 for 8-bit images),
and are small constants (typically, and ).
Additional Results
Probabilistic Forecasting For Vancouver Temperature: We now use the Vancouver93 dataset to build a forecasting model using SI. The approximating function we use is a residual-network based TCN inspired by Bai et al. [ [3] along with time embedding on each layer. In this case we propose a model that uses a sequence of the last 5 days’ temperatures to predict the next 5 days’ temperatures. For this problem, it is easily to visualize how SI learns the transportation from one distribution to another.
In the first experiment with this dataset, we take all the cases from the test-set where the temperature at Station 1 is and we compare the actual distribution of temperatures at the same stations after 10 days. Essentially, if we look at the phase diagrams between any two stations for distributions of states after 10 days apart, we can visualize how the final distributions obtained using SI are very similar to the distributions we obtain from test data. Such distributions Often they look like skewed distributions, like we can see in Figure 9 which shows the an extremely skewed distribution matching very well with the state after 5 days and similarly we can see in Figure 10 which shows the distribution matching very well with the state after 10 days. Appendix § 11.§ 2 shows the matching histograms of the distributions for the later case. Some metrics for comparison of results can be seen in Table 6 to Table 8.

Figure 9. Phase diagram showing initial state and final states from data and model using SI after 5 days in between station 2 and station 10.

Figure 10. Phase diagram showing initial state and final states from data and model using SI after 10 days in between station 20 and station 1.
| Data | True Mean | Our Mean | True Std. Dev. | Our Std. Dev. |
| Vancouver93 | 1.85e1 | 1.89e1 | 4.05 | 3.97 |
| CloudCast | 1.53e-2 | 1.51e-2 | 9.84e-2 | 9.84e-2 |
Table 6. Comparison of mean and standard deviation of ensemble predictions.
| Data | MSE | MAE | SSIM |
| Vancouver93 | 1.8e-1 | 3.4e-1 | NA |
| CloudCast | 4.9e-7 | 6.0e-4 | 0.85 |
Table 8. Similarity metrics for ensemble standard deviation.
| Data | MSE | MAE | SSIM |
| Vancouver93 | 5.9e-1 | 6.1e-1 | NA |
| CloudCast | 2.0e-7 | 6.0e-4 | 0.99 |
Table 7. Similarity metrics for ensemble average.
Statistical Comparison of Our Ensemble Predictions
Predator-Prey Model
Figures in 11 shows the histograms of and respectively. It can be noticed that the histogram of an actual state variable is matching very well with the histogram obtained from our results using SI.
Figure 11. Histograms of actual final distribution of (left) and (right) compared with that obtained using SI on the predator-prey model
Vacouver93
Figure 12 shows the histograms of the two distributions which are very similar to suggest that SI efficiently learns the transport map to the distribution of temperatures at a station for this problem whose deterministic solution is very tough.
Figure 12. Histograms of distributions observed after 10 days and distribution obtained using SI excluding outliers.
Moving MNIST
Let us take as input, initial 10 frames from a case in Moving MNIST test set as history and actual 10 subsequent frames in the sequence as future. For such a case Figure 13 and Figure 14 show statistical images for small perturbation and large perturbation respectively. 100 perturbed samples are taken and their final states are predicted using SI. The figures showcase how similar the ensemble mean and the ensemble standard deviations are. This is a fair justification that the uncertainty designed on Moving MNIST is captured very well.
Figure 13. Statistical comparison of Moving MNIST sequences predicted using SI with random perturbed initial states.

Figure 14. Statistical comparison of Moving MNIST sequence with random highly perturbed initial states.
Cloudcast
Predictions
As per the default configuration of 4 timeframe sequences from the test set of Cloudcast dataset is used to predict 4 subsequent timeframes in the future. A point to notice is each time step equals to 15 minutes in real time and hence the differential changes is what we observe. For one such a case Figure 15 shows twelve different predictions. They are very similar but not the same once we zoom into high resolution. Figure 16 shows the prediction after 3 hours by autoregression to showcase how the clouds look noticeably different. The small cloud patches are visibly different in shape and sizes. Cloud being a complex and nonlinear dynamical system, the slight non-noticeable difference in 1 hour can lead to very noticeably different predictions after 3 hours.

Figure 15. Random predictions of differential change from a single sequence from cloudcast testset.

Figure 16. Random predictions from a single sequence from CloudCast testset for prediction after 3 hours.
Statistical comparison
Figure 17 shows statistical images for generated outputs on CloudCast testset using SI. Similar samples are taken as the set of initial states to predict their final states using SI. The figures showcase how similar the ensemble mean and the ensemble standard deviations are.
Figure 17. Statistical comparison of CloudCast sequences predicted using SI for a single initial state.
WeatherBench
Predictions
Figure 18 shows some of predictions using SI with mild perturbations.
Statistical comparison
Figure 19 shows the statistical comparison of ensemble predictions with 20 mildly perturbed states. Figure 20 shows the statistical comparison of ensemble predictions with 78 strongly perturbed states. The mean and standard deviation of the states for 2 day (48 hours predictions) can be easily compared. Table 9 and table 10 shows the metrics to compare the accuracy of our ensemble mean and ensemble standard deviation for 6 hour and 2 day predictions respectively.
| Variable | Ensemble Mean | Ensemble Std. Dev. | ||||||||
| True Score | Our Score | MSE() | MAE() | SSIM() | True Score | Our Score | MSE() | MAE() | SSIM() | |
| Z500 | 5.40e4 | 5.40e4 | 2.00e4 | 1.04e2 | 0.992 | 3.96e2 | 3.74e2 | 1.83e4 | 9.16e1 | 0.850 |
| T2m | 2.78e2 | 2.78e2 | 2.62 | 9.33e-1 | 0.986 | 1.78 | 1.84 | 8.90e-1 | 5.60e-1 | 0.745 |
| U10 | -1.85e-1 | -2.70e-1 | 1.27 | 8.40e-1 | 0.886 | 2.51 | 2.34 | 1.12 | 7.35e-1 | 0.708 |
| T850 | -2.43e-2 | -3.74e-2 | 1.62 | 9.79e-1 | 0.812 | 3.75 | 3.58 | 1.29 | 8.26e-1 | 0.774 |
Table 9. Similarity metrics for weatherBench ensemble prediction after 6 hours.
Perturbations using SI: Additional Study
| Variable | Ensemble Mean | Ensemble Std. Dev. | ||||||||
| True Score | Our Score | MSE() | MAE() | SSIM() | True Score | Our Score | MSE() | MAE() | SSIM() | |
| Z500 | 5.40e4 | 5.40e4 | 7.86e4 | 2.04e2 | 0.978 | 3.58e2 | 3.78e2 | 4.52e4 | 1.41e2 | 0.630 |
| T2m | 2.78e2 | 2.78e2 | 3.05 | 1.06 | 0.986 | 1.83 | 1.95 | 1.54 | 7.3e-1 | 0.681 |
| U10 | -1.16e-1 | -1.10e-1 | 2.42 | 1.17 | 0.820 | 2.42 | 2.58 | 2.19 | 1.04 | 0.561 |
| T850 | -8.09e-2 | -8.12e-2 | 4.30 | 1.53 | 0.688 | 3.64 | 3.72 | 3.12 | 1.28 | 0.561 |
Table 10. Similarity metrics for weatherBench ensemble prediction after 2 days.
Perturbations using SI: Additional Study
MovingMNIST
Figure in 21 shows how the perturbed state generated using SI is better than Gaussian perturbations. Also, a crucial factor for the sensitivity of perturbation is showcased, where a generated sequence from MovingMNIST has a different digit is shown in the third row of the figure. With mild perturbaton, however, the mean of the 100 samples matches very well with the original state. The standard deviation of those states show the scope of perturbation, which is good. Figure 22 shows how a perturbed state varies with different noise levels. With larger noise, the digits seems to transform, like ’one’ turns to ’four’, ’eight’ turns to ’three’, and so on. The model understands than on transitioning, the digits should turn into another digit.
WeatherBench
Figure in 23 shows the four curated important variables for weather prediction, Z500, T2m, U10 and T850, perturbed with different values of noise.
Hyperparameter Settings and Computational Resources
UNet Training
Table 11 shows the hyperparameter settings for training on MovingMNIST and CloudCast datasets. While table 12 shows the settings for training on WeatherBench.
| Hyperparameter | Symbol | Value |
| Learning Rate | ||
| Batch Size | 64 | |
| Number of Epochs | 200 | |
| Optimizer | - | Adam |
| Dropout | - | 0.1 |
| Number of Attention Heads | - | 4 |
| Number of Residual Blocks | - | 2 |
Table 11. Neural Network Hyperparameters for training Moving MNIST and CloudCast
| Hyperparameter | Symbol | Value |
| Learning Rate | ||
| Batch Size | 8 | |
| Number of Epochs | 50 | |
| Optimizer | - | Adam |
| Dropout | - | 0.1 |
| Number of Attention Heads | - | 4 |
| Number of Residual Blocks | - | 2 |
Table 12. Neural Network Hyperparameters for training WeatherBench
Computational Resources
All our experiments are conducted using an NVIDIA RTX-A6000 GPU with 48GB of memory.
Figure 18. Four sample stochastic forecasts of Z500, T2m, U10 and T850 after 2 days obtained using SI.
Figure 19. Statistical comparisons of Z500, T2m, U10 and T850 for 2 day ensemble forecasting using SI using 20 mildly perturbed samples.
Figure 20. Statistical comparisons of Z500, T2m, U10 and T850 for 2 day ensemble forecasting using SI using 78 strongly perturbed samples.

Figure 21. Statistical comparison of 100 perturbed states from a single MovingMNIST sequence using SI.
Figure 22. Perturbed states for a single MovingMNIST sequence using SI for different levels of noise.

Figure 23. Perturbed states for a WeatherBench state using SI for different levels of noise.
References
- [1]Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209.15571, 2022.arxiv.org/abs/2209.15571
- [2]Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.arxiv.org/abs/2303.08797
- [3]Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.arxiv.org/abs/1803.01271
- [4]J. D. Benamou and Y. Brenier. A computational fluid mechanics solution to the monge kantorovich mass transfer problem. SIAM J. Math. Analysis, 35:61–97, 2003.
- [6]Cristian Bodnar, Wessel P Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, et al. Aurora: A foundation model of the atmosphere. arXiv preprint arXiv:2405.13063, 2024.
- [7]Michael Brin and Garrett Stuck. Introduction to dynamical systems. Cambridge university press, 2002.DOI
- [8]Ashesh Chattopadhyay, Pedram Hassanzadeh, and Devika Subramanian. Data-driven predictions of a multiscale lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network. Nonlinear Processes in Geophysics, 27(3):373–389, 2020.
- [10]Sibo Cheng, César Quilodrán-Casas, Said Ouala, Alban Farchi, Che Liu, Pierre Tandeo, Ronan Fablet, Didier Lucor, Bertrand Iooss, Julien Brajard, et al. Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review. IEEE/CAA Journal of Automatica Sinica, 10(6):1361–1387, 2023a.arxiv.org/abs/2303.10462
- [11]Sibo Cheng, César Quilodrán-Casas, Said Ouala, Alban Farchi, Che Liu, Pierre Tandeo, Ronan Fablet, Didier Lucor, Bertrand Iooss, Julien Brajard, Dunhui Xiao, Tijana Janjic, Weiping Ding, Yike Guo, Alberto Carrassi, Marc Bocquet, and Rossella Arcucci. Machine learning with data assimilation and uncertainty quantification for dynamical systems: A review. IEEE/CAA Journal of Automatica Sinica, 10(6):1361–1387, 2023b.arxiv.org/abs/2303.10462
- [12]Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.05789, 2019.arxiv.org/abs/1903.05789
- [14]Ronald Aylmer Fisher. Statistical methods for research workers. In Breakthroughs in statistics: Methodology and distribution, pages 66–70. Springer, 1970.
- [15]Deepak Ganesan, Sylvia Ratnasamy, Hanbiao Wang, and Deborah Estrin. Coping with irregular spatio-temporal sampling in sensor networks. ACM SIGCOMM Computer Communication Review, 34(1):125–130, 2004.DOI
- [16]Alan J. Geer and Peter Bauer. Observation errors in all-sky data assimilation. Quarterly Journal of the Royal Meteorological Society, 137(661):2024–2037, 2011.DOI
- [17]Amin Ghadami and Bogdan I. Epureanu. Data-driven prediction in dynamical systems: recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 380(2229):20210213, 2022.DOI
- [18]John Guckenheimer and Philip Holmes. Nonlinear oscillations, dynamical systems, and bifurcations of vector fields, volume 42. Springer Science & Business Media, 2013.
- [19]John Guckenheimer and Patrick Worfolk. Dynamical systems: Some computational problems. In Bifurcations and Periodic Orbits of Vector Fields, pages 241–277. Springer, 1993.arxiv.org/abs/chao-dyn/9304010
- [20]Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bidlot, Massimo Bonavita, Giovanna De Chiara, Per Dahlgren, Dick Dee, Michail Diamantakis, Rossana Dragani, Johannes Flemming, Richard Forbes, Manuel Fuentes, Alan Geer, Leo Haimberger, Sean Healy, Robin J. Hogan, Eliás Hólm, Marta Janisková, Sarah Keeley, Patrick Laloyaux, Philippe Lopez, Cristina Lupu, Gabor Radnoti, Patricia de Rosnay, Iryna Rozum, Freja Vamborg, Sebastien Villaume, and Jean-Noël Thépaut. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730):1999–2049, 2020.