WorksComputer Science
Dynamical State-Space Diffusion for Spatiotemporal Prediction
Abstract
Diffusion models for temporal prediction typically corrupt each future frame with independent noise, leaving temporal dependence entirely to the denoiser. We propose Dynamical State-Space Diffusion (DSSD), which learns the temporal structure of the corruption process itself. DSSD builds a temporal kernel from the damped oscillatory modes of a stable diagonal state-space model, whose learned decay rates and frequencies let the corruption capture multiple timescales and oscillatory correlations, while the denoiser architecture remains unchanged. Running this model on a finite clip, however, requires the state before the first frame, which is unobserved; starting from a zero state leaves the early frames with less noise than the diffusion schedule prescribes. We avoid this by using the Cholesky factor of the model's stationary covariance as the temporal operator. We prove that this operator is causal, exactly invertible, keeps each frame's noise variance equal to that of standard diffusion, and reduces to the identity at the clean end of the process, and that it is exactly the Kalman innovations form of the state-space model. The denoiser is trained to predict the temporally correlated noise, and sampling removes and reapplies the temporal mixing at each step. On scientific-field and robot-video datasets, DSSD outperforms diffusion baselines that share its denoiser, and the evolution of its learned modes during training shows that the temporal operator adapts to each dataset.
Introduction
Diffusion models generate data by learning to reverse a forward process that gradually corrupts clean samples with Gaussian noise. This denoising view has led to strong image and video generators [18, 36, 31, 19, 4], and it has also become attractive for temporal prediction [40, 10, 29, 37]. In forecasting, a model must represent not one future but a distribution over plausible futures conditioned on an observed history, making diffusion a natural fit for uncertain spatiotemporal signals such as video, weather, and multivariate time series.
Most diffusion frameworks for temporal prediction use a forward process that adds temporally independent noise to each future frame, relying on the denoising network to learn temporal dependencies [40, 10]. However, neighboring states are often temporally coupled through motion, transport, or oscillation. This motivates incorporating temporal structure into the forward process itself, exposing the denoiser to temporally structured signal and noise.
Dynamical Diffusion (DyDiff) [16] incorporates temporal dependence into diffusion by applying an exponential moving average (EMA) operator to both the clean sequence and the diffusion noise. This introduces temporal correlations while preserving a closed-form Gaussian forward process and an invertible temporal operator for sampling. However, at each diffusion step, the EMA operator imposes a single exponential decay across frames. Its decay rate follows a predefined schedule rather than being learned from data, limiting the operator to one memory scale and preventing it from expressing oscillatory correlations.
We propose Dynamical State-Space Diffusion (DSSD), a novel diffusion algorithm that incorporates a learnable state-space operator into forward corruption and reverse sampling. Inspired by S4D [15], the model combines multiple damped oscillatory modes whose decay rates, frequencies, and mixture weights are learned jointly with the denoiser. As illustrated in Figure 1, DSSD uses these learned dynamics to causally mix observed and future latents and to introduce temporal correlations into the diffusion noise. The state-space parameters are shared across sequences and spatial locations, while the denoiser architecture remains unchanged.

Figure 1. DSSD forward process. Learned state-space dynamics causally mix observed and future latents and separately correlate noise within the future window. The mixed future latents and correlated noise are combined at diffusion step to form .
Constructing a finite-window temporal operator from this learned kernel requires care at the start of the prediction window. Each output depends on both current and past noise inputs, including those preceding the window. Initializing the state-space model at zero omits the contribution of these earlier inputs, leaving the early frames with less noise variance than prescribed. This violates the requirement that every future frame receive the same noise variance at a given diffusion step. DSSD avoids this problem by using the Cholesky factor of the kernel’s stationary finite-window covariance, yielding a causal and exactly invertible operator with unit marginal noise variance. We use corresponding factors to transform the signal and noise, train the denoiser to predict the resulting correlated noise, and adapt each sampling step to undo and reapply the temporal transform.
Our contributions are summarized as follows:
We introduce DSSD, a dynamics-aware diffusion framework that uses a learnable diagonal state-space operator to define the temporal coordinates of the forward and reverse processes, rather than placing all temporal modeling burden on the denoiser.
We construct a stationary-covariance Cholesky operator that avoids finite-window boundary effects, is causal and exactly invertible, and preserves unit marginal noise variance. This is equivalent to a Kalman innovations realization, with DyDiff’s EMA operator recovered as a single-mode case.
Experiments on scientific-field (SEVIR, Turbulence) and robot-video (BAIR, RoboNet) datasets show that DSSD outperforms diffusion baselines sharing the same denoiser. Analysis of mode evolution during training shows that DSSD’s temporal operator adapts to each dataset.
Related Work
Diffusion models for temporal prediction. Conditional diffusion models have been applied to video prediction [40, 19], precipitation nowcasting [10], and probabilistic time-series forecasting [29, 37, 25]. These methods usually model temporal dependence through the conditioning scheme and the denoiser, for example by masking past frames, generating autoregressively, or adding temporal attention and convolution layers. The forward process remains standard, where each future time step receives independent noise. DSSD instead changes the temporal geometry of the forward process and leaves the denoiser unchanged, so it applies to existing diffusion algorithms.
Structured and learned forward processes. For video generation, temporally correlated noise priors share noise components across frames [11] or warp noise along optical flow [6]; these methods structure the noise but do not apply a matched temporal transform to the clean signal. Blurring and inverse heat dissipation apply a fixed linear operator to the signal, but along the spatial frequency axis rather than the prediction horizon [20, 30]. Other works learn part of the forward process, such as the noise schedule [24] or a data-dependent transformation of the corruption [3]. DYffusion [33] replaces diffusion steps with a learned temporal interpolator, so its reverse process advances physical time, whereas DSSD keeps Gaussian diffusion and changes only its covariance. DyDiff [16] applies a fixed EMA operator to both signal and noise, limiting its ability to represent multi-scale or oscillatory temporal correlations. DSSD learns this operator as a state-space model jointly with the denoiser, keeps the forward process causal, invertible and variance preserving, and recovers DyDiff as an exact special case.
State-space models. Structured state-space models parameterize linear recurrences with stable, structured transition matrices and scale to long sequences [14 b], and selective variants make these dynamics input dependent [13]. In diffusion models, SSMs have mainly served as network layers, either as attention-free backbones (Yan et al., 2024) or as temporal layers of time-series denoisers [25]. Linear-Gaussian state-space models also underlie probabilistic forecasting, where they describe how the data are generated [28]. In contrast, DSSD uses an SSM to define the temporal covariance of the forward process, whose stationary innovations factor gives the diffusion forward operator.
Preliminaries
Conditional diffusion for temporal prediction
Given observed frames of a spatiotemporal signal, we predict the next . Frames are encoded by a pretrained autoencoder [31] into observed latents and future latents , where is the dimension of a flattened latent frame. We model by diffusing only the future while conditioning the denoiser on the clean observations.
Diffusion models [18] corrupt with Gaussian noise. For a variance-preserving schedule , , whose product decreases from nearly 1 to 0,
A denoiser is trained to predict the added noise by minimizing , with sampled uniformly. The added noise is independent across future frames, while the denoising network learns temporal dependencies from the data.
Futures are generated from Gaussian noise by repeated denoising, for example with the deterministic DDIM step [36] for , where and . Throughout, temporal matrices act along the frame axis and apply the same transformation independently to each latent coordinate.
State-space models
Structured state-space models (SSMs) [14 b] describe a sequence through a hidden state that summarizes its past. For the continuous-time linear system , , zero-order-hold discretization with step size gives the recurrence
with the discretized input vector. From a zero initial state, unrolling gives a causal convolution whose kernel weights the input steps in the past.
S4D [15] makes diagonal. Taking eigenvalues together with their complex conjugates, with , and conjugate coefficients for conjugate states, the kernel becomes a real-valued sum of damped oscillatory modes,
where the amplitude and phase are determined by and . Each mode is a damped oscillation with decay rate and frequency : one zero-frequency mode gives a geometric decay, and several modes combine decay rates and oscillations in one kernel.
When driven by i.i.d. zero-mean, unit-variance inputs from the infinite past, a stable SSM has a stationary output whose covariance depends only on the lag ,
Dynamical State-Space Diffusion
We first define how a temporal operator enters the forward process, then build it from a learned SSM kernel through its stationary covariance and Cholesky factor, and finally describe training, sampling and the scope of the guarantees.
Dynamics-aware forward process
Following Guo et al. [16], we couple the strength of temporal mixing to the noise level through a dynamics schedule with , so that (no mixing) as the data become clean and mixing is strongest at high noise. Let denote a temporal operator on frames at dynamics level , and partition the operator on the full window as
The blocks act on the observed and future frames of § 3.1: carries the observed history into the future window, mixes future frames with one another, and the zero block keeps the observations free of any influence from the future. DSSD replaces equation (1) by
The clean future is mixed together with the observed history, so the first future frame inherits the temporal state of the observations, whereas the noise is mixed within the future window alone, without an observed prefix. For fixed dynamics parameters, is Gaussian with mean and covariance , so training inputs are still sampled in closed form, and recovers equation (1). As in DDIM [36], the process is specified by these marginals and a coupling in which all diffusion steps share the same ; a Markov forward chain is not required.
We require the operator to be (i) causal, mixing a frame only with itself and preceding frames, i.e., lower triangular; (ii) invertible, so that clean frames and noise can be recovered during sampling; (iii) of unit per-frame variance, , so that every frame receives noise of variance exactly as in equation (1) and the actual noise level given to the denoiser describes every frame; and (iv) the identity at , so that the clean end of the process is unchanged. These are design requirements for DSSD, rather than necessary conditions for diffusion models in general.
State-space temporal kernel
DSSD describes the temporal correlation of the mixed frames with a diagonal SSM (§ 3.2) whose discretization step is set by the dynamics level. For modes with learnable decay rates and frequencies , we set
For , each discrete pole has modulus . Figure 2(c) illustrates these stable poles and their conjugate pairs. As the step grows without bound and every mode forgets immediately, whereas at high noise the step is smallest and memory is longest. The step is determined by the dynamics schedule, while the learned rates and frequencies control individual modes; reproduces the EMA decay of DyDiff. We use amplitudes and set every phase in equation 3 to zero, giving positive real modal coefficients, and normalize the resulting kernel to unit norm,
Figure 2. Noise correlations and state-space modes in DSSD. (a) Illustrative noise correlation matrices for DPM, DyDiff, and DSSD. (b) Corresponding noise correlations by lag: DyDiff’s EMA decays exponentially, whereas DSSD can express oscillatory correlations. (c) An illustrative six-mode DSSD configuration shows discrete poles and their conjugates inside the unit circle. Filled and hollow markers denote conjugate partners.
This zero-phase choice fixes , which keeps the normalization bounded away from zero. We use this kernel for memory of past inputs and add a separate current-input term. The resulting impulse response is
Thus the SSM kernel is delayed by one frame: weights the current input and weights an input frames in the past. For independent unit-variance inputs, is the variance contributed by the current input, while the kernel distributes the remaining variance over the past. At , we define by its limit. The dynamics parameters comprise only scalars, shared across frames, pixels, samples and diffusion steps. Learning these parameters jointly with the denoiser allows the noise correlations to adapt to the data’s temporal structure beyond a fixed exponential decay.
Causal innovations operator
Applying the impulse response equation 9 as a convolution over a finite window requires inputs preceding that window, or their accumulated state. Starting from a zero state discards the memory of earlier inputs, so the early frames have variance below one and (iii) fails: the first frame has variance . For example, at it receives only half of the prescribed noise variance. DSSD instead assumes that the SSM has reached its stationary regime. By equation 4, driving the impulse response equation 9 with i.i.d. unit-variance inputs yields a stationary unit-variance process, and DSSD uses the causal factor of the covariance of consecutive outputs,
where chol returns the unique lower-triangular factor with positive diagonal satisfying , and all infinite sums admit closed forms (Appendix A.2). For Gaussian inputs, row of expresses output as the part predictable from preceding outputs plus an independent innovation, whose scale is . Thus defines the memory kernel, includes the current-input term, records stationary correlations, and realizes them on a finite window. Figure 2(a–Figure 2b) compares these noise correlations with those of standard diffusion and DyDiff.
Proposition 1 (Admissibility). For every , window length and parameters : (a) is positive definite, so exists and is causal and invertible; (b) ; (c) for all , hence at every diffusion step; (d) the leading block of equals , and ; and (e) as .
Properties (a), (b) and (e) establish requirements (i)–(iv), and (c) bounds from below every divisor in the triangular solves used during sampling. The schedule-wide bound is strictly positive when ; for , invertibility requires . These bounds do not guarantee a well-conditioned inverse, which we check numerically (Appendix § 10.10). The first part of (d) makes the operator consistent with the observed history, so the signal branch transforms and continues into the future; the second part is used in § 4.5. All proofs are given in Appendix § 10.
The Cholesky factor also has a state-space realization. It is an innovations representation of the stationary process defined by , rather than the original S4D convolution matrix.
Proposition 2 (State-space recurrence). For , is the finite-window Kalman innovations realization ([21]) of the SSM of § 4.2, started at its stationary state covariance and zero prior state mean: is computed by a linear recurrence with a -dimensional state (Appendix § 10.§ 4) whose innovation scales satisfy , and .
The state is the best estimate of the SSM state given the frames read so far, and the innovation scale shrinks as the state becomes better determined. In this view the two branches of equation (6) use different histories: the signal branch runs the recurrence with the observed latents as inputs and continues with the future latents, whereas the noise branch restarts from the stationary initialization. For long windows, this recurrence reduces the cost of applying the operator to linear in (Appendix § 10.§ 4).
Proposition 3 (EMA and richer correlations). Let . With a single mode, and , is the EMA operator of DyDiff ([16]), and , whose autocorrelation is positive at every lag . In contrast, the DSSD family admits a single oscillating mode with .
At each fixed dynamics level, the unrestricted DSSD covariance family therefore includes the EMA covariance (Corollary 1) and correlations beyond it: an oscillating mode can make neighboring frames anticorrelated, a pattern that a monotonically decaying memory cannot produce (Appendix § 10.§ 5).
Training and sampling
Training. The denoiser parameters and the dynamics parameters are trained jointly by regressing the mixed noise,
with given by equation (6) and sampled uniformly. For a fixed operator, the minimizer is the posterior mean of the mixed noise. Because the noise enters only through , the score of is a fixed linear transform of this posterior mean (Appendix § 10.8), so the correlated target requires no whitening. Since every coordinate of the target has unit variance for every , learning cannot lower the loss by shrinking the noise. Because enters both the corrupted input and the target, learning selects the temporal coordinates in which the denoiser operates rather than identifying the physical dynamics of the data. Initialization and numerical checks are given in Appendices A.9 and A.10. Algorithm 1 summarizes joint training of the denoiser and dynamics parameters.
Sampling. We use deterministic DDIM steps that undo the mixing of the current step and reapply that of the next one. Given the prediction at step and the next step ,
Sampling starts from with , and the last step uses , where the operator is the identity and is returned. Both inverses are triangular solves whose positive divisors satisfy Proposition 1(c), and with the update reduces to the standard DDIM step. Algorithm 2 gives the complete sampling procedure, including initialization and the clean endpoint. The following result justifies this update.
Proposition 4 (Sampler consistency). Fix the dynamics parameters , assume finite second moments of the data, and let be the unrestricted minimizer of equation 11. For , , , and equation 12 returns under the shared coupling. On an interior time interval where is twice continuously differentiable and , for a velocity field whose continuity equation is solved by the marginals .
Under the flow regularity conditions in Appendix A.6, equation 12 is a first-order discretization of a flow that transports the marginals of the forward process [1], as DDIM is for standard diffusion. With the ideal denoiser, its last step returns the posterior mean of the future.
What the temporal operator changes
Since every operator is invertible, one may ask whether DSSD merely reparameterizes standard diffusion. It does not: the mixing varies with the diffusion step and differs between the signal and noise branches of equation 6. Let , and let be the standard corruption equation 1 at the same noise level.
Proposition 5 (Information ordering). For every , . Consequently, given , a variable with the same joint distribution with as can be generated from and independent noise without access to , and, for data with finite second moments,
Temporal mixing therefore creates no information about the future at any single noise level; for the EMA operator, attenuates only the direction of the first future frame (Appendix A.7). DSSD instead changes the geometry of the regression problem that a finite denoiser solves across noise levels, and the path along which samples are generated. These changes may help a finite denoiser and sampler, but the proposition does not establish such a benefit; whether they help is tested empirically in Section 5.
Experiments
Proposition 5 rules out an information advantage from temporal mixing at a fixed noise level. We therefore test whether learning the temporal operator improves finite-step forecasting over standard diffusion DPM and DyDiff. We evaluate scientific fields from SEVIR [39] and Turbulence [41], and robot videos from BAIR [9] and RoboNet [8].
Experimental setup
Within each dataset, all methods use the same frozen framewise VAE and Stable Video Diffusion-inspired video U-Net [4]. We train for 50,000 optimizer updates and use a shared 50-step DDIM sampling grid [36]. Task windows and training settings are detailed in Appendix C, and example forecasts for all four datasets in Appendix D.1. Metrics include the continuous ranked probability score (CRPS), critical success index (CSI) and Fréchet video distance (FVD); all are defined in Appendix § 12.§ 3. In Tables 1 and 2, bold and underline mark the best and second-best scores for each dataset and metric.
| Dataset | Method | CRPS ↓ | Raw | w8avg | w8max | CSI w5 ↑ | Temporal variogram ↓ | Energy score ↓ | RMSE ↓ |
| SEVIR | DPM | <u>4.174</u> | <u>2.737</u> | <u>6.599</u> | <u>0.5518</u> | <u>2.339</u> | <u>10.862</u> | <u>15.420</u> | |
| SEVIR | DyDiff | 4.642 | 3.169 | 7.955 | 0.3911 | 4.492 | 13.024 | 25.540 | |
| SEVIR | DSSD (Ours) | 3.975 | 2.421 | 5.995 | 0.5707 | 2.129 | 10.263 | 14.585 | |
| Turbulence | DPM | 0.01148 | <u>0.00751</u> | <u>0.01065</u> | 0.9641 | <u>0.00262</u> | <u>0.01920</u> | <u>0.02600</u> | |
| Turbulence | DyDiff | <u>0.01100</u> | 0.00752 | 0.01123 | <u>0.9674</u> | 0.00260 | 0.01791 | 0.02427 | |
| Turbulence | DSSD (Ours) | 0.00756 | 0.00440 | 0.00757 | 0.9779 | 0.00147 | 0.01178 | 0.01582 |
Table 1. Scientific-field prediction results.
| Dataset | Method | CRPS ↓ | RMSE ↓ | Energy ↓ | Vario. ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |
| BAIR | DPM | 17.788 | 42.907 | 30.176 | 11.078 | 15.036 | 0.5938 | 0.2246 | <u>1329.2</u> |
| BAIR | DyDiff | <u>17.691</u> | <u>42.815</u> | <u>30.092</u> | 11.412 | <u>15.149</u> | <u>0.5999</u> | <u>0.2220</u> | 1417.4 |
| BAIR | DSSD (Ours) | 16.825 | 42.367 | 29.661 | 8.548 | 15.717 | 0.6205 | 0.2007 | 766.0 |
| RoboNet | DPM | 8.782 | <u>23.523</u> | <u>15.130</u> | <u>5.939</u> | <u>22.680</u> | 0.7889 | 0.1370 | <u>659.2</u> |
| RoboNet | DyDiff | <u>8.383</u> | 23.735 | 14.705 | 6.504 | 22.630 | 0.7865 | 0.1377 | 770.6 |
| RoboNet | DSSD (Ours) | 8.108 | 23.485 | 14.548 | 4.409 | 22.693 | 0.7858 | 0.1379 | 430.2 |
Table 2. Robot-video prediction results.
Scientific-field prediction
DSSD has the lowest scores on all three CRPS variants, temporal variogram, energy score, and RMSE, and the highest CSI on both datasets. Its raw CRPS is 3.975 versus 4.174 for the strongest SEVIR baseline (DPM), and 0.00756 versus 0.01100 for the strongest Turbulence baseline (DyDiff).
Robot-video prediction
On BAIR, DSSD outperforms both DPM and DyDiff across all eight reported metrics. On RoboNet, DSSD leads six of eight metrics, while DPM achieves the best SSIM and LPIPS. On both datasets, DSSD shows substantial improvements in FVD and temporal variogram scores, whereas RMSE gains are modest.
Prediction quality across horizons
Figure 3 compares prediction quality across horizons on BAIR. Panel (a) shows an example predicted sequence and absolute-error maps for each method. In this example, DSSD produces more accurate predictions throughout the sequence, with particularly smaller absolute errors than DyDiff at intermediate time steps. Panel (b) reports frame-wise CRPS() and temporal variogram scores, averaged over all BAIR test windows at each prediction horizon. DSSD maintains lower scores than both baselines throughout the evaluated horizons, with its advantage persisting into the later frames. These results indicate sustained improvements in probabilistic prediction and closer agreement with ground-truth changes between consecutive frames.
Figure 3. BAIR forecast quality across future frames. (a) Example predictions and absolute-error maps. (b) Test-set frame-wise pooled CRPS and temporal variogram scores; lower is better.
Analysis of learned temporal modes
The learned operator is compact, with only parameters (§ 4.2). Figure 4 tracks these modes from 10,000 to 50,000 updates through the centered residue logits , their norm (the residue-logit norm), and the normalized entropy of (Appendix § 12.§ 4). The residue-logit norm measures logit spread around the mean, whereas lower entropy indicates more concentrated residue weights across modes. On SEVIR, the residue-logit norm increases from 1.53 to 4.11 while entropy decreases. Entropy also decreases on BAIR and RoboNet: SEVIR and RoboNet concentrate residue weight on the first mode, and BAIR on the second (Appendix § 13.§ 3). On Turbulence, the residue shares remain nearly uniform, yet the modes continue to evolve: at , the first mode’s pole radius decreases from 0.937 to 0.889, indicating faster decay and shorter temporal memory. These observations indicate dataset-dependent adaptation of both mode weights and timescales.
Figure 4. DSSD mode-parameter diagnostics. (a) SEVIR centered residue logits and their norm. (b) Normalized residue entropy on four datasets.
Conclusion
We introduce DSSD, the novel diffusion framework for spatiotemporal prediction to make the coupling between spatial denoising and temporal dynamics jointly learnable. Its forward process adds noise while mixing frames through a learnable state-space operator, and its reverse process combines learned denoising with inversion of the temporal mixing. Joint training of the state-space dynamics and denoiser allows this coupled process to adapt to the data without changing the denoiser architecture. We establish a theoretical foundation through a stationary-covariance construction that ensures causal, invertible temporal mixing with the prescribed noise variance, together with a consistency analysis of the sampling update. Experiments on scientific fields and robot videos show improved probabilistic forecasts and temporal consistency, while analysis of the learned modes demonstrates dataset-dependent adaptation. We expect this framework to help accelerate the development of dynamics-aware video foundation models at higher resolutions and larger scales. Its learnable temporal dynamics may also support online adaptation to changing temporal patterns in streaming data, with potential applications in robotics.
AI Use Statement
We used OpenAI Codex and Anthropic Claude to assist with implementing training, sampling, evaluation, and experiment-management code based on our specifications. These tools also assisted with typesetting and editing proofs, writing numerical verification scripts, providing feedback on experimental design, drafting and editing the manuscript, searching and summarizing literature, and formatting references.
The theoretical model, operator construction, mathematical claims, and proofs were developed by the authors. AI assistance with proofs was limited to presentation and numerical checks; it did not supply proof arguments. AI tools were not used to develop the conceptual framework, propose or refine hypotheses, generate synthetic datasets, or conduct qualitative and thematic data analysis. Translation was not applicable.
We reviewed all AI-assisted work, checked the proofs against our derivations, and verified the algebra through independent numerical implementations. We tested the code and checked citations against primary sources. The authors take responsibility for the final text, claims, code, and other artifacts.
Ethics Statement
This work involves no human subjects, crowdsourced annotation, or personally identifiable information; no institutional review was required.
Reproducibility Statement
Section 4 specifies the method and theoretical assumptions; Appendix A provides complete proofs, and Appendix A.10 details initialization and numerical implementation. Appendix C specifies the data processing, model and optimization settings, evaluation protocols, and compute resources used in our experiments.
We use publicly accessible datasets—SEVIR, BAIR, and RoboNet—and a simulation-based benchmark, Turbulence. Third-party datasets are used in accordance with their terms and are not redistributed.
We will release model weights, training and sampling code, and configurations for the models and datasets used in this paper. The release will include environment specifications and instructions for data preparation and reproducing the experiments. Run records and numerical tests support verification of the experimental settings and theoretical identities.
Mathematical Details and Proofs
We first derive the stationary covariance, then prove the operator properties and its state-space realization, and finally analyze regression and sampling. Throughout, the dynamics parameters are fixed unless joint optimization is explicitly discussed; , and with . For , we omit and when no confusion arises and write , , and , so that in equation (8). As in the main text, matrices act on the time axis and are shared by all coordinates of each frame. The endpoint is defined by and ; expressions involving are evaluated only for . We distinguish the SSM memory kernel , the full impulse response , its stationary covariance , and the finite-window innovations factor .
Notation
Table 3 collects the symbols used in § 3 and § 4 and in this appendix, grouped in the order in which they appear. Bold lowercase letters denote vectors or stacked latent frames, bold uppercase letters denote matrices, and all temporal matrices act along the frame axis, identically for every latent coordinate. A few auxiliary symbols local to a single proof step, such as in Appendix § 10.§ 3, are defined where they appear.
| Symbol | Meaning | Defined in |
| Data and diffusion | ||
| Numbers of observed and future frames | § 3.1 | |
| Length of the window on which a temporal operator acts; | § 4.1 | |
| Dimension of a flattened latent frame | § 3.1 | |
| Observed latents (clean conditioning history) | § 3.1 | |
| Future latents to be predicted | § 3.1 | |
| Number of diffusion steps; current step | Eq. (1) | |
| Next, less noisy step of a sampling update, | § 3.1 | |
| Noise schedule and signal level | Eq. (1) | |
| White Gaussian noise, , before temporal mixing | Eq. (1) | |
| Noisy future latents at step | Eq. (1), Eq. (6) | |
| Denoiser and its parameters | § 3.1 | |
| Conditional distribution of the future given the history | § 3.1 | |
| Symbol | Meaning | Defined in |
| Marginal of the forward process at step | §4.4 | |
| State-space models (generic) | ||
| Continuous time | §3.2 | |
| State, scalar input and scalar output of an SSM | (2) | |
| State, input and output parameters | §3.2 | |
| Step size and zero-order-hold discretization, | (2) | |
| Convolution kernel of the discretized SSM; weight of the input steps in the past | (3) | |
| Number of complex-conjugate mode pairs | §3.2 | |
| Continuous-time eigenvalue of mode , ; in DSSD, | §3.2, (7) | |
| Amplitude and phase of mode | (3) | |
| Dynamics schedule and learned modes | ||
| Mixing strength, | §4.1 | |
| Dynamics schedule, ; 1 means no mixing | §4.1 | |
| Generic dynamics level, | §4.1 | |
| Learned decay rate () and frequency () of mode | (7) | |
| Learned residue logit of mode ; | §4.2 | |
| Dynamics parameters , scalars | §4.2 | |
| Discretization step tied to the dynamics level, | (7) | |
| Discrete pole of mode , , with modulus | §4.2 | |
| Kernel, covariance and operator | ||
| Memory kernel before and after normalization to unit norm | (8) | |
| Impulse response: weights the current input, the input frames earlier | (9) | |
| Stationary autocorrelation at lag , ; | (10) | |
| Stationary covariance of consecutive outputs, | (10) | |
| Lower-triangular Cholesky factor with positive diagonal | (10) | |
| Temporal operator, | (10) | |
| Blocks of : history on history, history into future, future on future | (5) | |
| Operator applied to the noise, on the future window alone | (6) | |
| Innovations recurrence (Proposition 2) | ||
| Input sequence of the recurrence, | Prop. 2 | |
| Output sequence of the operator | Prop. 2 | |
| Innovation scale, | Prop. 2 | |
| Pre-input SSM state, containing only inputs before frame | (20) | |
| State estimate given | (22) | |
| Stationary state covariance | App. A.4 | |
| Conditional state covariance given | (21) | |
| Kalman gain | (21) | |
| Direct current-input term of the realization, | (20) | |
| Training and sampling | ||
| Mixed-noise regression loss | Eq. (11) | |
| Denoiser prediction of the mixed noise, | Eq. (12) | |
| Estimates of the clean future and of the white noise at step | Eq. (12) | |
| White Gaussian noise that initializes sampling, | §4.4 | |
| Velocity field transported by the sampler | Prop. 4 | |
| Analysis (Proposition 5) | ||
| Standard corruption Eq. (1) at the same noise level | §4.5 | |
| , future signal in whitened noise coordinates | §4.5 |
Table 3. Notation.
| Symbol | Meaning | Defined in |
| Estimator of | Eq. (13) | |
| Appendix-only notation | ||
| App. A | ||
| , | Pole modulus and angle of mode | App. A |
| Normalizer of the kernel, | App. A.2 | |
| Unnormalized kernel autocorrelation, | App. A.2 | |
| Closed form of | Eq. (14) | |
| Scalar i.i.d. inputs of the stationary process, | App. A.3 | |
| Matrix of the EMA recurrence | Cor. 1 | |
| Lower-triangular matrix of the unrolled recurrence | App. A.4 | |
| Noise variance | App. A.6 | |
| Noise operator | App. A.6 | |
| , , | Coefficients of | Eq. (23) |
| Unrestricted minimizer of Eq. (11) | App. A.6 | |
| Smooth, compactly supported test function | App. A.6 | |
| App. A.7 | ||
| Auxiliary white noise independent of all other variables | App. A.7 | |
| Mean of | App. A.8 | |
| Shorthand in Algorithms 1 and 2 | ||
| , , | , and returned by BuildOperators | App. B |
| , | Frozen encoder and decoder of the autoencoder | App. B |
| Minibatch size | Alg. 1 | |
| , | Mixed signal and mixed noise of training example | Alg. 1 |
| Sampling grid of denoising steps, | Alg. 2 | |
| Indices | ||
| Frame (row) index within a window | ||
| Delay of an input in the impulse response | ||
| Delay of an input in the memory kernel | ||
| Lag between two frames | ||
| , | Mode indices |
Table 1 (PDF p. 15).
Closed-form sums
For and real , the real part of the geometric series gives
where the denominator equals and is written in a form that avoids cancellation. Using with and ,
For , the definition of in equation 10 splits into the term and the terms with , and . Since , this gives
Because the amplitudes are positive and sum to one, and . As , every , so for while , giving the endpoint .
Admissibility (Proposition 1)
Proof of Proposition 1. Let be i.i.d. and . Since , the series converges in mean square, is a stationary Gaussian process, and ; hence is the covariance of .
(a) For , split , where depends only on . The first term equals with lower triangular and diagonal entries , and it is independent of . Thus . A positive definite matrix has a unique Cholesky factor, which is lower triangular with a positive diagonal and therefore causal and invertible.
(b) .
(c) A conditional variance of a Gaussian vector depends only on its covariance, so we may compute it for with . Because is lower triangular with a positive diagonal, is an invertible linear function of , and is independent of it. Hence . For the process itself, are functions of , whereas with a function of and independent of the past. For jointly Gaussian variables, conditioning on more information cannot increase the conditional variance, so . Finally, . This gives a positive uniform lower bound if . For , each finite step with remains admissible, but the zero-signal endpoint would have and lies outside the construction.
(d) The leading block of the Toeplitz matrix is . In the block form equation 5, and is lower triangular with a positive diagonal, so by uniqueness. The trailing block of is likewise ; multiplying out the block form yields
Moreover, the first row of is because , so the noise branch of equation 6 leaves its first frame unchanged.
(e) At , by Appendix A.2, so and . As , both coefficients in equation 17 tend to zero, while and by the unit norm of and the Cauchy–Schwarz inequality. Hence for every , so , and continuity of the Cholesky factor on positive definite matrices gives .
Corollary 1 (EMA special case). If , and , then for every the operator is the matrix of the recurrence , , i.e., the exponential moving average operator of DyDiff [16].
Proof. At both matrices are the identity. For , , , , and , so . Let be the matrix of the recurrence, with and for . For ,
using . Since is lower triangular with a positive diagonal, .
State-space recurrence (Proposition 2)
State-space realization. Unlike the post-input state in equation 2, we use a pre-input state that contains only inputs before frame . This convention implements the one-frame delay in equation 9, with a separate direct term for the current input. For , define the -dimensional real system
Each block rotates by and scales it by , so . Hence, with , the process of Appendix A.3 satisfies
Because the spectral radius is below one, is stationary with covariance solving , and is independent of .
Innovations. Let and , so and . Conditional on , the pair is Gaussian with means , variance and cross-covariance , because is independent of and of . Gaussian conditioning gives the Kalman recursion
Let . By the defining property of conditional expectation, is uncorrelated with every function of , and each with is such a function. The are therefore uncorrelated, jointly Gaussian and of unit variance, hence i.i.d. . Substituting into equation 21 gives the recurrence of Proposition 2,
whose input sequence is . Both and its estimate precede the current output; the latter is initialized at zero mean with uncertainty , not at a known zero physical state.
Proof of Proposition 2. Unrolling equation 22 gives for a lower-triangular with diagonal entries . Then , and uniqueness of the Cholesky factor gives , in particular . Since the recurrence is linear and its coefficients do not depend on the input, it computes for any input sequence . Furthermore, , and by stationarity , which cannot increase as the conditioning set grows.
In the setting of Corollary 1, determines the state, so and the recurrence closes on the previous output. The gains depend only on and cost per dynamics level, after which applying the recurrence to -dimensional frames costs , compared with for equation (10).
Strictly richer correlations
Proof of Proposition 3. The EMA special case follows from Corollary 1, whose proof gives . To exhibit a correlation outside that family, fix , take and , so that , and write . Then , and , so
By equation (17) with and ,
which is negative if and only if , that is, . Since and , this holds exactly when , and such exists because the right-hand side is positive.
For example, and give . The required frequency lies within the clipping range of Appendix § 10.10 whenever , i.e., , which covers every when , including .
Sampler consistency (Proposition 4)
Write , , , and , so that equation 6 reads
with the same for every . For , and are invertible by Proposition 1(a).
Proof of Proposition 4. Posterior means. The minimizer of equation 11 at step is . Taking the conditional expectation of equation 23 given , and using , yields . Solving for the first expectation gives exactly in equation 12 with , and . By linearity of equation 23 at step , , which is the update equation 12.
Transport. The entries of are smooth in by Appendix § 10.§ 2, the Cholesky factor is smooth on positive definite matrices, and is affine in ; hence , and are twice continuously differentiable on interior time intervals for . For , the operator is identically and the same conclusion follows directly. Define . For a smooth, compactly supported test function and data with finite second moments,
by the tower property, which is the weak form of the continuity equation for .
Consistency. By the first part, , so by Taylor’s theorem, where the remainder is uniform wherever the posterior means are bounded.
Assuming the velocity admits a unique flow and the continuity equation has a unique solution in the considered class, integrating backward from the exact terminal marginal reproduces every marginal on the interior interval, and equation 12 is its first-order discretization. Extension to a degenerate endpoint requires separate control of the flow limit. At the last step, gives , and , so the update returns . Two approximations remain, and both are shared with DDIM. First, by the law of total covariance, exceeds the covariance of by , so a large step omits conditional variance. Second, the initialization matches for arbitrary data distributions at , provided , since then and . At it is generally approximate; special data distributions can still yield equality. These statements concern the ideal denoiser under the forward marginals, not errors along a learned sampler’s trajectories.
Information ordering (Proposition 5)
Proof of Proposition 5. Fix , write , and , and let . Multiplying equation 18 by on the left and by on the right gives , hence .
Given , the map is invertible, so estimators based on and on achieve the same errors. Let be independent of everything else and define . Conditional on , is Gaussian with mean and covariance , which is the conditional law of . Hence and have the same joint distribution, and is computed from and without . For any estimator with finite risk, is a function of , so , where the inequality holds because the conditional mean minimizes the squared error for data with finite second moments, and the last equality uses the independence of . Minimizing over yields equation (13).
For the EMA operator of Corollary 1 with , every future column of has global index at least one, so its entries are , whereas has first column and the same entries elsewhere. Hence and : only the first future frame is attenuated. For a general DSSD operator, has rank at most .
Colored-noise regression and the score
Fix and , and write , and . For each , the minimizer of equation (11) over unrestricted functions is . The marginal is the mixture . Differentiating under the integral, which is valid under standard integrability conditions,
since . For any fixed operator, the colored target therefore determines the score, and the correlation between target coordinates requires no whitening of equation (11). Whitening would regress on times the target, that is, on itself; since conditional expectation commutes with the fixed matrix , the unrestricted minimizers of the two losses are related by , so they carry the same information, and the unwhitened minimizer is the one that enters the sampling update equation (12) directly. The two losses do weight the coordinates of the residual differently, which can matter for a finite denoiser; this is part of the regression geometry discussed in § 4.5. This statement concerns a fixed ; the value of that minimizes equation (11) need not minimize a forecasting error.
Nonzero-frequency initialization
For , , which vanishes for every and when . The normalized kernel, the autocorrelation, the operator equation (10) and the loss equation (11) depend on only through , so at . With a zero gradient, first-order optimizers such as Adam leave both the parameter and its moment estimates unchanged, so a mode initialized at zero frequency remains non-oscillatory throughout training. This motivates the nonzero initial frequencies of Appendix § 10.10.
Stable parameterization and optimization
Bounded modes. Decay rates and frequencies are clipped to and , and amplitudes are kept positive by the softmax. The closed-form sums and the Cholesky factorization are computed in double precision.
Initialization. Decay rates are log-spaced in , frequencies are set to , and amplitudes are uniform. Nonzero initial frequencies are necessary by the argument of Appendix § 10.9.
Operator certificates. Proposition 1 holds exactly, but finite-precision inverses can still amplify errors when a learned operator is ill conditioned. Before training and at every evaluation, we therefore check the operator on a fixed grid of 33 dynamics levels taken from the schedule, at window lengths and : the reconstruction error of the full and history-conditioned inverses, unit variance, causality, the identity at , and a condition number below . The checks are run in single and double precision, and training stops if any of them fails. These checks certify only the tested dynamics levels and checkpoints, not every possible parameter value or every point between grid levels.
Training and Sampling Algorithms
Algorithms 1 and 2 summarize the mixed-noise training objective and deterministic sampler. The pretrained encoder and decoder are frozen. All temporal matrices act independently on each latent coordinate.
Shared operator construction. Both algorithms use BuildOperators(), with the current dynamics parameters and window lengths , understood. It returns three matrices: , which maps the observed history into the future; , which mixes future signal frames; and , which mixes future noise. The routine constructs the normalized SSM kernel and impulse response in equation 8 and equation 9, computes their stationary covariance matrices for lengths and using Appendix A.2, and takes their lower-triangular Cholesky factors as in equation 10. The blocks and come from the length- factor, while . In general, : signal mixing includes the observed history, whereas noise mixing uses a separate future window. At , the routine returns . Parameter bounds and double-precision covariance calculations follow Appendix A.10. The construction is differentiable in during training.
| Algorithm 1 | Training of DSSD |
| Require: | Training sequences; frozen encoder ; window lengths , ; batch size ; schedule ; mixing strength ; optimizer and update budget |
| Ensure: | Trained parameters |
| 1: | Initialize as in Appendix A.10 |
| 2: | while update budget remains do |
| 3: | Sample a minibatch |
| 4: | ; |
| 5: | for do |
| 6: | Sample |
| 7: | Sample |
| 8: | |
| 9: | |
| 10: | |
| 11: | |
| 12: | |
| 13: | |
| 14: | end for |
| 15: | |
| 16: | Compute through and |
| 17: | Update |
| 18: | end while |
| 19: | return |
Table 1 (PDF p. 20).
| Algorithm 2 | Sampling of DSSD |
| Require: | Observed frames ; frozen encoder and decoder ; trained ; prediction length ; schedule and mixing strength ; grid |
| Ensure: | Predicted future frames |
| 1: | ; number of frames in |
| 2: | Set and |
| 3: | for do |
| 4: | |
| 5: | |
| 6: | end for |
| 7: | Draw of shape |
| 8: | |
| 9: | for do |
| 10: | ; |
| 11: | |
| 12: | Solve for |
| 13: | Solve for |
| 14: | |
| 15: | |
| 16: | end for |
| 17: | return |
Table 2 (PDF p. 20).
Training details. The noise target is not detached from the dynamics parameters. The loss is written as a batch average of squared norms; the implementation additionally averages over latent coordinates. Operator checks are performed before training and at evaluation checkpoints as described in Appendix A.10.
Sampling details. All model-evaluation steps satisfy and . The parameters remain fixed, so the operators can be precomputed; no gradients or new noise draws are needed inside the sampling loop. At , , and the update returns . The initial colored Gaussian is generated from the terminal forward marginal at nonzero , as discussed in Appendix A.6. We use steps and the clean endpoint ; the implementation uses array indices and a clean-endpoint sentinel .
Experimental Details
Data and forecasting tasks
Table 4 specifies the input and prediction windows. All models forecast future frames from observed frames. We map SEVIR and RGB inputs from to and divide both Turbulence velocity components by four, without clipping.
| Dataset | Input | Resolution | Input transform | |
| SEVIR | VIL, 1 channel | |||
| Turbulence | Velocity, 2 channels | |||
| BAIR | RGB video | |||
| RoboNet | RGB video |
Table 4. Forecasting tasks. and are the numbers of observed and future frames.
Architecture, optimization, and sampling
Within each dataset, all methods use the same pretrained, frozen, framewise VAE ([23]), including its EMA weights. The VAE reduces each spatial dimension by a factor of four, outputs three latent channels, and uses a latent scale factor of 0.18215. During training, latent frames are sampled from the VAE posterior; evaluation encodes observations using its posterior mode. The U-Net takes clean observed latents and noisy future latents as conditioning and prediction inputs, respectively. We use a video U-Net [32] inspired by Stable Video Diffusion [4], configured identically for all three methods.
| Component | Setting |
| Video U-Net | 64 base channels; channel multipliers ; two residual blocks per level. |
| Attention and time mixing | Attention at downsampling factors ; four heads; transformer depth 1; dropout 0; temporal convolution kernel and learned spatial/temporal mixing. |
| Diffusion | 1,000 timesteps; cosine schedule ([26]) with ; prediction. |
| Optimization | Adam ([22]), , , weight decay 0; effective batch size 16; FP32. No denoiser-weight EMA. |
| Sampling | 50 deterministic DDIM steps on the same released_uniform timestep grid. |
Table 5. Shared model, optimization, and sampling settings.
| Method | Temporal operator | Learning rate |
| DPM | Identity (no temporal mixing). | . |
| DyDiff | Fixed EMA, . | . |
| DSSD | Learned stationary SSM operator, state size 12 (six conjugate pairs), | . |
Table 6. Method-specific temporal operator and learning-rate schedule.
The DSSD operator parameters form a separate optimizer group; the learning-rate schedule applies to both groups. DSSD’s stationary covariance and Cholesky factor are computed in double precision, with the parameter bounds and numerical operator checks detailed in Appendix § 10.10.
DiT backbone. For the SEVIR backbone ablation (Appendix § 13.§ 2), we replace the U-Net with a DiT-S/4 ([27]): 12 blocks of width 384 with six attention heads, MLP ratio 4 and no dropout. Each latent frame is split spatially into patches, giving 64 tokens per frame and 832 tokens for the 13 frames, and attention runs jointly over all frames with fixed sine–cosine spatial and temporal position embeddings. The inputs are the same as for the U-Net, the timestep enters through adaLN-Zero, and only the outputs at future-frame positions are used. The DiT has 32,529,840 parameters; all other settings, including the method-specific learning rates, match the U-Net runs on SEVIR.
Compute. Each model is trained on a single NVIDIA A100 80GB GPU, using the PCIe variant for SEVIR and Turbulence and the SXM4 variant for BAIR and RoboNet. Within each dataset, the three methods share the same GPU type and batch configuration. DSSD adds 18 trainable parameters for its six conjugate mode pairs to the 60,982,185-parameter denoiser shared by all methods, whereas DyDiff’s fixed EMA operator adds none. Table 7 reports the training cost of the first 50,000 updates. Relative to the faster baseline, DSSD’s median time per update is 1.4% higher on SEVIR and 2.0% higher on BAIR, is the lowest on Turbulence, and is 4.9% higher on RoboNet, computed from the values in Table 7. All three methods reserve the same GPU memory on every dataset. Learning the temporal operator therefore adds little training cost.
| Method | SEVIR | Turbulence | BAIR | RoboNet | ||||||||
| s/upd. | Total | Mem. | s/upd. | Total | Mem. | s/upd. | Total | Mem. | s/upd. | Total | Mem. | |
| DPM | 2.22 | 31.8 | 48.8 | 1.06 | 14.8 | 18.4 | 0.98 | 21.5 | 11.5 | 0.82 | 17.4 | 9.2 |
| DyDiff | 2.19 | 31.3 | 48.8 | 1.07 | 15.0 | 18.4 | 0.98 | 21.9 | 11.5 | 0.83 | 17.8 | 9.2 |
| DSSD (Ours) | 2.22 | 31.9 | 48.8 | 1.02 | 14.8 | 18.4 | 1.00 | 22.7 | 11.5 | 0.86 | 17.7 | 9.2 |
Table 7. Training cost over the first 50,000 updates on one A100 80GB GPU. s/upd.: median time per update (s); Total: wall-clock time (h); Mem.: reserved GPU memory (GiB).
Evaluation metrics
Evaluation protocol. The number of generated futures per input follows the evaluation protocol of DyDiff [16], which adopts the prevailing convention of each domain. For SEVIR and Turbulence, the generated futures form a probabilistic ensemble, as in Chen et al. (2024). The finite-ensemble CRPS estimator below accounts for ensemble size, and keeps sampling tractable on the SEVIR test set. For BAIR and RoboNet, we follow the video-prediction protocol of Gupta et al. (2023) and Wu et al. (2024), which selects among generated videos per input. Because this selection improves with , we retain for comparability. Within each dataset, all methods share the same and 50-step DDIM sampler, so method rankings are comparable. Scores are not comparable between the two dataset groups.
CRPS. For target and generated futures , the finite-ensemble probability-weighted-moment CRPS [12] (Zamo & Naveau 2018) at a pixel is
Unpooled CRPS averages this score over pixels, channels, and forecast times, and is reported on all four datasets. On SEVIR and Turbulence, we also report pooled variants: we pool the target and each generated future spatially, using either average or maximum pooling with stride 2, then compute the same CRPS. The subscripts in and identify these two pooling operations. To retain compatibility with the reference evaluation, the pooled Turbulence variants include only pooling windows whose target velocities are all nonnegative; unpooled CRPS uses the full field. The per-horizon BAIR curves in Figure 3 use frame-wise CRPS after spatial max pooling, averaged over the full BAIR test set.
CSI. CSI is reported on SEVIR and Turbulence. We threshold the fields before applying a maximum neighborhood operator with stride 2. SEVIR uses raw VIL thresholds 16, 74, 133, 160, 181, 219; Turbulence uses threshold zero for each velocity component. Each CSI is TP/(TP + FP + FN), and the reported score is macro-averaged over defined threshold–horizon–channel cells; cells without an observed or predicted event are excluded. We follow the implemented maximum-neighborhood convention, which differs from an average-pooling description of CSI in the DyDiff paper [16].
Temporal variogram score. The temporal variogram score [35] averages, over lags 1–4, the squared discrepancies between the target’s absolute temporal increments raised to power and the ensemble mean of the corresponding powered absolute increments. The per-horizon BAIR curves use only consecutive future-frame pairs and therefore start at the second future frame, because the first has no preceding future frame.
Energy score. Let denote the root-mean-square distance over a complete future video. We report the energy score [12]
RMSE. RMSE is defined differently for the two dataset groups. On SEVIR and Turbulence, it is the RMSE of the ensemble mean of the generated futures. On BAIR and RoboNet, it is the square root of the mean per-input minimum full-video MSE, so a generated future is selected for each input; the BAIR entries are computed as square roots of the corresponding MSE scores.
PSNR, SSIM, and LPIPS. These metrics are reported on BAIR and RoboNet. For each of the generated videos, we average framewise PSNR, SSIM (Wang et al., 2004), and VGG-based LPIPS (Zhang et al., 2018) over future time. We then report the maximum PSNR and SSIM and the minimum LPIPS for each input, selecting independently for each metric, so the selected video can differ across the three metrics. PSNR and SSIM are evaluated on the RGB scale; LPIPS uses RGB in .
FVD. Unlike the selection-based metrics above, FVD [38] compares the distribution of all 100 generated videos per input with the real videos. We prepend the observed frames to every generated future and to its real counterpart, then compute the Fréchet distance between I3D [5] feature distributions. LPIPS and FVD are computed offline from saved predictions. Their implementation follows the stated aggregation protocol, but exact numerical identity with an unspecified upstream feature extractor is not assumed.
Learned-mode diagnostics
Each temporal mode corresponds to one complex-conjugate pole pair, so the six residue weights index six modes. Figure 4(a) reports the absolute centered residue logits and their norm, the residue-logit norm, on SEVIR. Figure 7 additionally shows the residue shares for all four datasets. These quantities are computed from checkpoints saved every 10,000 updates from 10,000 to 50,000 updates. Centering removes the common logit shift that leaves the shares unchanged. The residue-logit norm measures the spread of the logits around their mean; absolute centered logits alone do not indicate whether a mode gains or loses weight. The shares directly show how relative weight is redistributed.
Figure 7. Mode-resolved residue diagnostics on the four datasets, measured every 10k training steps (10k–50k) from saved DSSD checkpoints. (a) Softmax residue shares of the six temporal modes, each corresponding to a conjugate pair. (b) Absolute centered residue logits of each mode (bars) and their total norm (diamonds).
Figure 4(b) reports normalized residue entropy, , computed from logged training values over the same update range. Values near one indicate nearly uniform shares, while lower values indicate greater concentration. These shares are mixture weights, not independent mode energies; their entropy does not measure pole decay or frequency. As a separate check on the near-constant Turbulence entropy, we reconstruct the discrete pole of its first mode at fixed : its radius is 0.937 at 10,000 updates and 0.889 at 50,000 updates, indicating faster decay at that dynamics level despite little redistribution of residue weight. These parameter changes do not establish physical mode identification or a causal contribution to forecasting quality.
Additional Results
Qualitative forecasts
Figures 5 and 6 show one test sequence per dataset. We select the sequence on which DSSD most improves on the better baseline in per-sequence average-pooled CRPS. For SEVIR, we take the highest-ranked sequence whose last two observations and target each have at most 5% saturated pixels and a standard deviation of at least 30 VIL units, because nearly saturated fields are hard to read. Each method row shows the generated future with the lowest full-video MAE among its samples, and the error rows show that sample’s absolute error.
| Backbone | Method | CRPS Raw | CRPS | CRPS | CSI | Temporal variogram | Energy score | RMSE |
| U-Net | DPM | <u>4.174</u> | <u>2.737</u> | <u>6.599</u> | <u>0.5518</u> | <u>2.339</u> | <u>10.862</u> | <u>15.420</u> |
| U-Net | DyDiff | 4.642 | 3.169 | 7.955 | 0.3911 | 4.492 | 13.024 | 25.540 |
| U-Net | DSSD (Ours) | 3.975 | 2.421 | 5.995 | 0.5707 | 2.129 | 10.263 | 14.585 |
| DiT | DPM | 5.425 | 3.862 | 7.453 | 0.4314 | 3.559 | <u>10.437</u> | <u>23.689</u> |
| DiT | DyDiff | <u>5.260</u> | <u>3.789</u> | <u>7.327</u> | <u>0.4383</u> | <u>3.326</u> | <u>10.777</u> | <u>24.905</u> |
| DiT | DSSD (Ours) | 5.211 | 3.706 | 7.188 | 0.5118 | 2.784 | 10.302 | 23.375 |
Table 8. Backbone ablation on SEVIR. Results with a U-Net and a DiT denoiser. Bold: best; <u>underline:</u> second best within each backbone.
In all four examples, errors concentrate in moving structures: storm cells in SEVIR, shear regions in Turbulence, and the robot arm in BAIR and RoboNet. The baselines’ errors spread more with lead time. This is most visible from to in Turbulence, and against DyDiff in the robot videos. DSSD’s errors stay more localized, although they also grow at the final RoboNet frame.
Backbone ablation on SEVIR
Table 8 compares the three methods with either a video U-Net or a Diffusion Transformer (DiT) denoiser (Peebles & Xie [27]), described in Appendix § 12.§ 2. Within each backbone and metric, bold and underlined values indicate the best and second-best methods, respectively.
DSSD obtains the best score on all seven reported metrics within both backbone groups. With DiT, it achieves a higher CSI w5 than DyDiff, the strongest baseline on this metric (0.5118 vs. 0.4383), and a lower temporal variogram score (2.784 vs. 3.326). Its raw CRPS is also lower (5.211 vs. 5.260). Thus, the favorable within-backbone ranking observed with the U-Net also holds for the transformer denoiser on SEVIR.
Mode-resolved temporal adaptation
Each temporal mode corresponds to one complex-conjugate pair of poles, so the six residue weights index six modes. Figure 7 tracks how the size and share of each mode change during training, revealing which modes the model comes to rely on and which it suppresses. Concretely, it shows the softmax residue shares at the saved 10,000–50,000-update checkpoints, together with for each mode and the norm of the centered six-vector. Centering removes the common logit shift, which leaves the residue shares unchanged.
The trajectories differ across datasets. SEVIR’s first-mode share increases from 0.255 to 0.664, BAIR’s second-mode share from 0.329 to 0.744, and RoboNet’s first-mode share from 0.234 to 0.667. Turbulence retains similar shares across the five checkpoints, and its residue-logit norm remains near 0.7. These observations explain the differing entropy trends in Figure 4; they measure residue reweighting, not independent mode energies. In particular, nearly fixed Turbulence residue shares do not imply fixed poles, as the pole-radius comparison in § 5.5 illustrates. Mode indices are tracked within each run and do not identify matching physical modes across datasets.
Multivariate time-series forecasting
We evaluate multivariate forecasting on six datasets distributed through GluonTS (Alexandrov et al. [2]), following TimeGrad’s dataset and prediction length setting (Salinas et al. [34]; Rasul et al. [29]). The datasets are Exchange, Solar, Electricity, Traffic, Taxi, and Wikipedia. To hold the representation and denoiser fixed across methods, DPM, DyDiff, and DSSD use the same temporal 1D dilated-convolution network: a pointwise channel projection followed by eight residual blocks with 128 hidden channels, which jointly processes observed values and noisy future values.
Figure 5. Example scientific-field forecasts and absolute errors on (a) SEVIR and (b) Turbulence. Turbulence shows the velocity component. Errors above the color-bar maximum saturate.

(a) (a) BAIR

(b) (b) RoboNet
Figure 6. Example robot-video forecasts and RGB-mean absolute errors on (a) BAIR and (b) RoboNet. Errors above the color-bar maximum saturate.
Each dataset is one multivariate sequence with channels. We compute a mean and standard deviation for each channel from the training sequence and apply the same normalization to test data. For each GluonTS test window, we forecast its future points from the preceding points. Table 9 gives these dimensions and window lengths.
| Dataset | Method | CRPS | RMSE | MAE | Energy | Variogram | |||
| Exchange | 8 | 30 | 30 | DPM | <u>0.37573</u> | <u>0.68591</u> | <u>0.56123</u> | 0.59404 | 0.78559 |
| DyDiff | 0.37640 | 0.76033 | 0.64038 | <u>0.54593</u> | <u>0.60261</u> | ||||
| DSSD (Ours) | 0.29896 | 0.61315 | 0.51080 | 0.46610 | 0.44125 | ||||
| Solar | 137 | 24 | 24 | DPM | <u>1.43163</u> | <u>2.31897</u> | 1.95040 | 1.89547 | 2.25310 |
| DyDiff | 1.48259 | 2.34964 | 1.97462 | <u>1.92896</u> | 2.37242 | ||||
| DSSD (Ours) | 1.41653 | 2.31678 | <u>1.95240</u> | 1.88563 | <u>2.26805</u> | ||||
| Electricity | 370 | 24 | 24 | DPM | <u>7.12656</u> | <u>7.08183</u> | <u>5.72460</u> | <u>6.72375</u> | 8.66073 |
| DyDiff | 7.17127 | 7.10774 | 5.74584 | 6.74831 | <u>8.67150</u> | ||||
| DSSD (Ours) | 7.04436 | 7.05699 | 5.70272 | 6.69563 | 8.87129 | ||||
| Traffic | 963 | 24 | 24 | DPM | <u>7.12199</u> | <u>10.90963</u> | 9.67041 | <u>8.47188</u> | <u>6.88153</u> |
| DyDiff | 7.14840 | 10.91133 | 9.66698 | 8.47718 | 6.85649 | ||||
| DSSD (Ours) | 7.08377 | 10.90212 | <u>9.66832</u> | 8.46117 | 7.08459 | ||||
| Taxi | 1,214 | 24 | 24 | DPM | <u>4.40064</u> | <u>5.75505</u> | 4.91774 | <u>4.65397</u> | <u>3.57764</u> |
| DyDiff | 4.41067 | 5.75627 | <u>4.91771</u> | 4.65638 | 3.57039 | ||||
| DSSD (Ours) | 4.37568 | 5.75023 | 4.91415 | 4.64667 | 3.68216 | ||||
| Wikipedia | 9,535 | 30 | 30 | DPM | 10.35604 | 16.95278 | 11.20462 | 12.66044 | 9.40709 |
| DyDiff | 10.41862 | 16.70496 | 11.03162 | 12.54705 | <u>9.47266</u> | ||||
| DSSD (Ours) | <u>10.41440</u> | <u>16.93813</u> | <u>11.19473</u> | <u>12.65975</u> | 9.52240 |
Table 9. Multivariate time-series test results at 50,000 updates. Bold and underlined values are best and second best within each dataset. All scores use training-set normalization; lower is better. is the number of jointly modeled series; and are observed and predicted time steps.
All methods are trained from scratch for 50,000 optimizer updates with batch size 16, Adam at a constant learning rate of , a 1,000-step cosine diffusion schedule, and seed 42. DyDiff uses the fixed EMA operator with . For DSSD, we use three conjugate SSM mode pairs (state size six), compared with six pairs in the video experiments. The time-series representation has no spatial latent feature maps, so we use fewer temporal modes as a modest capacity choice. All methods share the same denoiser. We use 50-step deterministic DDIM with a linearly spaced timestep grid.
Table 9 reports channel- and time-averaged finite-ensemble PWM CRPS, ensemble-mean RMSE and MAE, energy score, and temporal variogram score. Metric definitions follow Appendix C.3.
At 50,000 updates, DSSD has the lowest CRPS, RMSE, and energy score on five of six datasets, with DPM leading CRPS on Wikipedia and DyDiff leading its RMSE and energy score. The temporal variogram score is mixed: DSSD is best on Exchange and second on Solar, but has the highest score of the three methods on Electricity, Traffic, Taxi, and Wikipedia.
References
- [1]Michael Albergo, Nicholas M. Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. Journal of Machine Learning Research, 26(209):1–80, 2025.
- [2]Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. GluonTS: Probabilistic and neural time series modeling in Python. Journal of Machine Learning Research, 21(116):1–6, 2020.arxiv.org/abs/1906.05264
- [3]Grigory Bartosh, Dmitry Vetrov, and Christian A. Naesseth. Neural Flow Diffusion Models: Learnable forward process for improved diffusion modelling. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2404.12940
- [4]Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.arxiv.org/abs/2311.15127
- [5]João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the Kinetics dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.arxiv.org/abs/1705.07750
- [6]Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C. Azevedo. How I warped your noise: a temporally-correlated noise prior for diffusion models. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2504.03072
- [8]Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. RoboNet: Large-scale multi-robot learning. In Conference on Robot Learning (CoRL), volume 100 of PMLR, 2019.arxiv.org/abs/1910.11215
- [9]Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robot Learning (CoRL), volume 78 of PMLR, 2017.arxiv.org/abs/1710.05268
- [10]Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang, Xiaoyong Jin, Danielle Maddix, Yi Zhu, Mu Li, and Yuyang Wang. PreDiff: Precipitation nowcasting with latent diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2307.10422
- [11]Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2305.10474
- [12]Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007.
- [13]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2312.00752
- [14]Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022a.arxiv.org/abs/2111.00396
- [15]Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems (NeurIPS), 2022b.arxiv.org/abs/2206.11893
- [16]Xingzhuo Guo, Yu Zhang, Baixu Chen, Haoran Xu, Jianmin Wang, and Mingsheng Long. Dynamical Diffusion: Learning temporal dynamics with diffusion models. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2503.00951
- [18]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2006.11239
- [19]Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
- [20]Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.05557
- [21]Thomas Kailath, Ali H. Sayed, and Babak Hassibi. Linear Estimation. Prentice Hall, 2000.
- [22]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.arxiv.org/abs/1412.6980
- [23]Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations (ICLR), 2014.arxiv.org/abs/1312.6114
- [24]Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2107.00630
- [25]Juan Miguel Lopez Alcaraz and Nils Strodthoff. Diffusion-based time series imputation and forecasting with structured state space models. Transactions on Machine Learning Research (TMLR), 2023.arxiv.org/abs/2208.09399
- [26]Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2102.09672
- [27]William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2212.09748
- [28]Syama Sundar Rangapuram, Matthias W. Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
- [29]Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2101.12072
- [30]Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2206.13397
- [31]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2112.10752
- [32]Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.arxiv.org/abs/1505.04597
- [33]Salva Rühling Cachay, Bo Zhao, Hailey Joren, and Rose Yu. DYffusion: A dynamics-informed diffusion model for spatiotemporal forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.01984
- [34]David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank Gaussian copula processes. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1910.03002
- [35]Michael Scheuerer and Thomas M. Hamill. Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities. Monthly Weather Review, 143(4):1321–1334, 2015.
- [36]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021.arxiv.org/abs/2010.02502
- [37]Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2107.03502
- [38]Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018.arxiv.org/abs/1812.01717
- [39]Mark Veillette, Siddharth Samsi, and Chris Mattioli. SEVIR: A storm event imagery dataset for deep learning applications in radar and satellite meteorology. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- [40]Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal. MCVD: Masked conditional video diffusion for prediction, generation, and interpolation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
- [41]Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics-informed deep learning for turbulent flow prediction. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2020.arxiv.org/abs/1911.08655




