WorksComputer Science
The Clustering Occupation Kernel (CLOCK) Method for Time Series Arising from Dynamical Systems
Abstract
We introduce the Clustering Occupation Kernel (CLOCK) method, a novel representation-based framework for clustering time series arising from dynamical systems. CLOCK embeds observed sequences into a reproducing kernel Hilbert space (RKHS) via a time-averaged occupation kernel, enabling trajectory comparison through their RKHS norm. For systems governed by differential equations, this formulation rigorously approximates the kernel mean embedding of the invariant Sinai-Ruelle-Bowen (SRB) measure, meaning our dissimilarity metric approximates the maximum mean discrepancy (MMD) between attractors. To accommodate general time series, we introduce time lags via Takens' Embedding Theorem to reconstruct the phase space. Crucially, CLOCK natively handles trajectories of unequal lengths and irregularly-spaced observations. We validate the proposed metric using spectral clustering, demonstrating superior performance against benchmark methods on both synthetic datasets and a real-world intracranial large-scale electroencephalogram (EEG) dataset. Finally, ablation studies on the Lorenz-96 system are consistent with the theoretical convergence rates of the occupation kernels and highlight the method's strong robustness to zero-mean Gaussian noise.
Introduction
Time series, sequences of observations with accompanying timestamps [22], are ubiquitous objects in data science. The immense volume of available data makes annotating each time series infeasible, so unsupervised methods are needed to cluster data by shared characteristics. Time series clustering methods fall roughly into three categories: distance-based, model-based, and representation-based. Distance-based methods use a custom distance function defined on time series, so standard methods like K-means [19] can be adapted. The DTW barycenter-averaging method (DBA) [25] using dynamic time warping [30][4] is an example. Model-based methods cluster trajectories based on their similarity in the parameters of a model assumed to govern the data. For example, time-series-based Gaussian mixture models fall into this category [36]. Representation-based methods map the time series to latent feature vectors that capture the essential elements of each time series. Typical clustering methods in Euclidean space may be applied to these representations. Autoencoders [33][17] and contrastive-learning-based methods [40] are examples.
In this work, we adopt a representation-based perspective that views time series, after preprocessing, as solutions to ordinary differential equations (ODEs). Specifically, we embed each observed trajectory into a reproducing kernel Hilbert space (RKHS) by considering its time-averaged occupation embedding [28, 27]:
where is a positive-definite kernel defined on the state space .
This representation captures the time-averaged occupation measure of the trajectory in state space and provides a natural way to compare time series through Hilbert space distances. Its relevance becomes particularly clear when the observed trajectories are generated by dynamical systems. According to the Palis conjecture [23], a generic dynamical system defined as the flow of a () map on a compact manifold admits (i) finitely many attractors, (ii) each of which is supported by a Sinai–Ruelle–Bowen (SRB) measure, and (iii) the union of the basins of these attractors has full Lebesgue measure. These SRB measures characterize the long-term statistical behavior of trajectories.
The main contribution of this work is to (1) show that the embedding defined by (1) approximates the kernel mean embedding (KME)[32] of these SRB measures, (2) leverage this connection to propose the Clustering Occupation Kernel method (CLOCK): a principled clustering method that groups trajectories according to their associated SRB measures, and (3) evaluate CLOCK against the state of the art.
We compute dissimilarities between time series by approximating the maximum mean discrepancy (MMD) between their estimated SRB measures. The authors in [5] took a related approach, embedding time series in high-dimensional feature space using sliding windows. Their goal was to provide a kernel-based distance measure that estimates distances by matching feature distributions that only partially overlap in their support. However, for time series generated by a dynamical system, their method clusters trajectories based on transient dynamics governed by the initial distribution. By contrast, our proposed method clusters trajectories according to their long-term, asymptotic statistics, making the resulting distances invariant to initial conditions as long as the trajectories are long enough. Ultimately, the choice between these methods depends on whether the application demands sensitivity to short-term transients or long-term systemic invariants.
Whether the time series is generated by an unknown discrete-time system or a continuous higher-order ODE, we can reconstruct its multi-dimensional phase space by preprocessing the observations using time-delay embedding before using CLOCK. Provided the underlying states evolve on a smooth, compact manifold, Takens’ theorem [35] guarantees that these delay-coordinate trajectories provide a diffeomorphism to the true orbits of the underlying system.
We demonstrate the proposed metric’s effectiveness through a series of clustering tasks. Specifically, we apply spectral clustering to similarity matrices derived from the MMD distances between trajectory embeddings. We present numerical experiments in which the time series are the solutions of ODEs corrupted by noise. Different clusters correspond to different vector-field parameters. In this context, we compare CLOCK with standard benchmark approaches. We also evaluate CLOCK on a large medical dataset of electroencephalogram measurements with a gold standard.
Method
Problem setup
We are given a collection of time series , each composed of a finite collection of points and a corresponding collection of timestamps . In order to motivate the method, let the observations be noisy snapshots from the solutions to an initial value problem:
where , are i.i.d. and where is a finite collection of vector fields, typically within the same family but with different parameters. The goal is to find a cluster assignment that maps each time series in the collection to one of clusters without knowledge of the vector fields or the initial conditions. In this context, one could consider a model-based approach jointly estimating the cluster assignment, the vector fields, and the initial conditions using an Expectation-Maximization algorithm [6]. We present below a more direct approach: computing a similarity between trajectories that can be used as input to an existing clustering algorithm. Note that we do not assume that the time series are of equal length or that the observations are regularly spaced. In fact, the algorithm described below applies to any time-series clustering problem. However, it is justified for data generated by a dynamical system or data that can be preprocessed to exhibit the same statistical characteristics as if it were generated by a dynamical system.
Background
Let be a compact manifold, and let be a map (). A Sinai–Ruelle–Bowen (SRB) measure for is an -invariant Borel probability measure on with several important properties. We describe the property of interest as follows: 1 Let be a continuous function and let be a solution to the differential equation defined by . There exists a positive-Lebesgue-measure set, denoted and referred to as the basin of attraction for , such that if , then the following holds:
The Palis conjecture states that, for a dense set in the space of all () maps with the topology, each has a finite collection of SRB measures such that has total Lebesgue measure. That is, where denotes the Lebesgue measure.
Let be a reproducing kernel Hilbert space (RKHS) of functions defined on a topological space . Let be a probability measure on . Then the KME of is given by the image of the mapping:
The MMD between two probability measures defined on a topological space is the following quantity:
where are the KME’s of , respectively. The MMD quantifies the dissimilarity of the measures and . We say that is characteristic (see [32]) when the mapping (4) is one-to-one. If is characteristic, then the measures and are identical if and only if .
The clustering occupation kernel method (CLOCK)
Assume for now that each trajectory is observed continuously over the time interval . Let us then embed into an RKHS by computing its time-averaged occupation kernel. Thus, we select a continuous, positive-definite kernel with corresponding RKHS and compute the mapping:
where . The motivation for this embedding comes from the following result, proved in Appendix § 7:
Theorem A.1. Let be a continuous, positive-definite kernel defined on a compact manifold . Let be a solution to a first-order ODE satisfying the Palis conjecture. Then we have that the following holds almost surely:
where is the SRB measure of the attractor of the underlying dynamical system for which .
Proof sketch: We prove this theorem in two steps: first we prove weak convergence, and then we prove convergence of the norm, from which strong convergence follows. The convergence of the norm can be shown using the kernel’s Mercer property.
Now, according to Theorem A.1, we can approximate
where is the SRB measure of the attractor such that . Note that is the KME of .
We are now ready to propose the dissimilarity between time series. It is defined by:
In practice, the time series are observed at a finite number of time points, thus we estimate the integrals with trapezoid-rule quadratures using the samples and associated time points , see Appendix § 8. This dissimilarity is an estimate of , the squared MMD of and . Next, we convert dissimilarities to similarities in order to perform spectral clustering. We define the similarity according to:
where [10]. We obtain an matrix of similarities which we use as input for a standard spectral clustering algorithm. Note that other similarities may be derived from the given dissimilarity. All the experiments in this paper are presented with the similarity in (10).
Rate of convergence
We prove the following result in Appendix § 7:
Theorem A.2. Let be a compact forward-invariant region for the flow of a vector field, let be an invariant probability measure on , and let be a continuous positive-definite kernel with Hölder feature map. Suppose that time-averaged occupation kernels started from converge to at mean-square rate (e.g. under Lemma A.2), and that trajectories started from are exponentially asymptotic to trajectories on the attractor, with asymptotic phases distributed absolutely continuously with respect to (Assumption A.1). Then
Remark 2.1. Among the systems of Section 3.1, the assumptions of Theorem A.2 hold for the FitzHugh–Nagumo system (hyperbolic sinks) and the Van der Pol system (hyperbolic limit cycles), with every kernel used in our experiments; see Appendix A.1. For the chaotic systems the theorem is not established. The theorem concerns noise-free trajectories as , so it motivates, rather than certifies, the finite- experiments.
We also provide a visualization of this rate in Section 3.5.
Application to general time series
We use Takens’ embedding theorem [35] to extend our setting to more general time series. We augment the time-series dimension by adding time lags. That is, we use trajectories of the form where and are fixed. If and are chosen appropriately, the time-lagged trajectories will be the images of integral curves under a smooth embedding of a compact manifold into . Thus, we may approximate the KMEs of their SRB measures.
The optimal time-lag may be found by computing the first local minimum of the average mutual information of [9], while a large enough embedding dimension may be selected by minimizing the number of false nearest neighbors of the data points [12].
Scaling the method to large datasets
Like distance-based methods, this method requires computing dissimilarities for all possible pairs of time series in the dataset, so, by default, its complexity scales quadratically with the number of time series . However, by approximating an implicit kernel with an explicit one, e.g., random Fourier features [26] and using truncated (rank ) singular value decomposition, the time and space complexities are reduced to, resp., and .
Experiments
We compared the CLOCK method to a collection of standard methods on several synthetic datasets as well as a real, medical dataset.2 In addition, we performed ablation studies to determine the convergence rate of the time-averaged occupation kernels and the quality of the clustering as a function of trajectory length.
All experiments on synthetic data were conducted on a laptop with the Intel Core Ultra 9 Processor 185H, 64 GB RAM, and an NVIDIA GPU with 8GB of memory. The experiment on real data was conducted on a High-Performance Computing (HPC) cluster using a single NVIDIA A40 GPU with 45 GB RAM.
Synthetic datasets
For the synthetic datasets, we set the number of clusters to be 2 for simplifying the evaluation and generated 30 trajectories per cluster. Each cluster was generated by the same differential equation but with different parameters. Trajectories were integrated numerically using a standard fourth-order Runge-Kutta method with a fixed step size. White noise was then added. The datasets are visualized in Figure 1.

Figure 1. Shown are the synthetic datasets in the -plane with coloring given by the true cluster labels. We present the true trajectories along with the noisy samples. For higher-dimensional datapoints, we plot only the first two coordinates.
FitzHugh-Nagumo (FHN) oscillator: [8][21] This is a 2D system given by the equations:
where , , , are constants. For cluster 1, we used , , , . For cluster 2, we used , , , . We sampled initial conditions uniformly from the box and integrated trajectories over the time interval . We selected 101 regularly spaced points on each trajectory. We added zero-mean Gaussian noise with standard deviation 0.1.
Van der Pol oscillator:[37] A 2D system with governing equations:
where is a constant. We selected for cluster 1 and for cluster 2. Initial conditions were sampled uniformly in the box . We integrated trajectories over the time interval . We then sampled 101 points at irregular intervals on each trajectory. We added Gaussian noise with standard deviation 0.5.
Lorenz63:[14] This is a 3D dynamical system. The governing equations are given by:
where are constants. For cluster 1, we used . For cluster 2, we used . Initial conditions were sampled uniformly from the box . We integrated trajectories over the time interval and selected 41 points at irregular intervals on each trajectory. We then added Gaussian noise with standard deviation 1.5
Kuramoto:[13] We used a 10D Kuramoto model with governing equations:
for where and are constants. For both clusters, we set . For cluster 1, we set , and for cluster 2, we set . Initial conditions were sampled from the box . We integrated trajectories over the time interval and selected 51 regularly-spaced points. We then added Gaussian noise with standard deviation 0.5.
Lorenz96:[15] We used a 40D Lorenz96 system with governing equations:
where we use the convention , , and is a constant. For cluster 1, we used , and for cluster 2, we used . Initial conditions were sampled uniformly from the box . We integrated trajectories over the time interval and selected 41 irregularly-spaced points on each trajectory. We then added Gaussian noise with standard deviation 1.5.
Real dataset
Bonn EEG time series dataset:[3] This dataset consists of 1D intracranial electroencephalogram (EEG) recordings of the human brain used for epileptic seizure detection research. There were two clusters in the dataset, each with 100 time series. Cluster 1 consisted of baseline, seizure-free readings in the epileptogenic zone, while Cluster 2 contained measurements of seizure activity taken from all regions of the brain where such activity was found. Automatically identifying seizure activity in EEG recordings has the potential to provide insight into brain function, and to aid in diagnosis and seizure prevention. Each time series was 23.6 seconds long with 4097 equally-spaced samples.
For the CLOCK method, we used Takens’ embedding theorem to augment the time series (see § 2.5). We set and time-steps for this experiment.
Evaluation metrics
We used the class labels to define several clustering metrics. See Appendix § 10 for explicit formulas.
Adjusted Rand Index (ARI): The adjusted Rand index [34] counts the proportion of data points that are clustered consistently with the true labels after adjusting for random chance. It takes values in the range of with value 1 being a perfect clustering, 0 being entirely random, and negative values being worse than random.
Normalized Mutual Information (NMI): The normalized mutual information [18] is an information-theoretic metric which quantifies how much the knowledge of the predicted labels reduces the information content of the true labels. It is normalized to take values in with 1 being perfect correspondence and 0 indicating that the two sets of labels are independent.
Purity: Purity [2] measures the homogeneity of the predicted clusters in terms of the true cluster assignments. In the setting of two predicted clusters and two true clusters, the purity is just the accuracy of the clustering.
Comparators
We tested the method against several benchmark methods. We implemented each method in a manner similar to that of CLOCK. Thus, each implementation consisted of the following general steps:
(a) Map the time series to representative feature vectors in latent space (using the identity mapping by default). (b) Compute pairwise distances between the feature vectors using a distance metric (using Euclidean distance by default).
(c) Convert distances to similarities using a Gaussian kernel, choosing the median as the bandwidth.
(d) Obtain cluster assignments via spectral clustering on the matrix of similarities.
(e) By default, use linear interpolation to handle irregular spacing.
We tested against the following collection of methods:
Euclidean distance: This baseline method uses ordinary Euclidean distance to compute distances between time series. If the time series are given as matrices with shape , with the number of timestamps and the dimension, we compute . Thus, the similarity metric is given by a standard Gaussian kernel.
Dynamic Time Warping (DTW): We compute the distance between time series and with the DTW algorithm [30]. This algorithm aligns two time series by stretching and compressing segments with a dynamic one-to-many mapping before computing the distance between the aligned points. Time series with similar features that are out of phase along the time axis will be nearby in terms of DTW distance.
Shape-Based Distance (SBD): SBD is a distance metric for time series used in the KShape algorithm [24]. Like DTW, it aligns the time series before computing distance. However, unlike DTW, it aligns time series by shifting; not stretching and compressing. The cross correlation statistic , which is a similarity measure in the range , of the aligned, -normalized time series is computed, and the distance is given by . This distance is invariant to scaling and shifting.
Catch22: The Catch22 method [16] extracts 22 specially-chosen features from each time series to use as a low-dimensional representation. The 22 selected features were chosen from a pool of over 7000 candidate features and found to be optimal across a broad range of tasks (such as clustering and classification) using rigorous evaluation criteria.
Latent ODE: This is a deep learning method [29] which models the time series as trajectories generated by a learned vector field and initial conditions in a latent, high-dimensional space. Since dynamical information from the trajectories is encoded in the latent initial conditions, we used them as feature vectors for the time series. This method takes irregular spacing into account, and thus we don’t have to interpolate.
TS2Vec: TS2Vec [40] is another deep learning method using a contrastive framework to learn representations of time series. During training, positive pairs of context views are optimized via a contrastive loss. The context views are aligned across the time dimension, as well as across spatial dimensions. The contrastive loss is applied hierarchically at various scales to account for low- and high-resolution features. To handle irregular spacing, we formed a uniform grid of time-points and inserted samples at times corresponding to observations. We masked out the other points.
Latent ODE and TS2Vec have several hyperparameters to tune. We used an internal clustering metric, the density-based clustering validation (DBCV) score, for validating them. See Appendix § 11 for details. See Table 1 for the results of the numerical experiments.
| Dataset | Method | ARI | NMI | Purity |
| FHN | CLOCK GM | 1 | 1 | 1 |
| FHN | CLOCK quad | 0.6893 | 0.6615 | 0.9167 |
| FHN | Euclidean | 0.0638 | 0.1892 | 0.6333 |
| FHN | DTW | 0.0821 | 0.2102 | 0.65 |
| FHN | SBD | 0.1508 | 0.2746 | 0.7 |
| FHN | Catch22 | 0.634 | 0.619 | 0.9 |
| FHN | Latent ODE | 0.1209 | 0.1144 | 0.6833 |
| FHN | TS2Vec | 1 | 1 | 1 |
| Van der Pol | CLOCK GM | 0.6893 | 0.6615 | 0.9167 |
| Van der Pol | CLOCK quad | -0.0172 | 0 | 0.5 |
| Van der Pol | Euclidean | -0.0172 | 0 | 0.5 |
| Van der Pol | DTW | 0 | 0.0301 | 0.5167 |
| Van der Pol | SBD | 1 | 1 | 1 |
| Van der Pol | Catch22 | 0.7469 | 0.7084 | 0.9333 |
| Van der Pol | Latent ODE | -0.0124 | 0.0033 | 0.5333 |
| Van der Pol | TS2Vec | 1 | 1 | 1 |
| Lorenz63 | CLOCK GM | 1 | 1 | 1 |
| Lorenz63 | CLOCK quad | 0.8689 | 0.8214 | 0.9667 |
| Lorenz63 | Euclidean | 0.0969 | 0.0911 | 0.6667 |
| Lorenz63 | DTW | 0 | 0.0301 | 0.5167 |
| Lorenz63 | SBD | -0.0144 | 0 | 0.5 |
| Lorenz63 | Catch22 | -0.0124 | 0.0033 | 0.5333 |
| Lorenz63 | Latent ODE | 1 | 1 | 1 |
| Lorenz63 | TS2Vec | 1 | 1 | 1 |
| Kuramoto | CLOCK GM | 0.6893 | 0.6615 | 0.9167 |
| Kuramoto | CLOCK quad | 0.1197 | 0.1008 | 0.6833 |
| Kuramoto | Euclidean | 0.0958 | 0.0817 | 0.6667 |
| Kuramoto | DTW | 0.1508 | 0.2746 | 0.7 |
| Kuramoto | SBD | -0.0172 | 0 | 0.5 |
| Kuramoto | Catch22 | -0.0126 | 0.0032 | 0.5333 |
| Kuramoto | Latent ODE | -0.007 | 0.0072 | 0.55 |
| Kuramoto | TS2Vec | 0.0011 | 0.0132 | 0.5667 |
| Lorenz96 | CLOCK GU | 1 | 1 | 1 |
| Lorenz96 | CLOCK quad | 0.3922 | 0.449 | 0.8167 |
| Lorenz96 | Euclidean | 0 | 0.0301 | 0.5167 |
| Lorenz96 | DTW | 0 | 0.0301 | 0.5167 |
| Lorenz96 | SBD | -0.0105 | 0.0013 | 0.5167 |
| Lorenz96 | Catch22 | -0.0022 | 0 | 0.5 |
| Lorenz96 | Latent ODE | -0.0041 | 0.0037 | 0.5167 |
| Lorenz96 | TS2Vec | 0.8689 | 0.8214 | 0.9667 |
| EEG | CLOCK GM | 0.6873 | 0.5821 | 0.915 |
| EEG | CLOCK Laplace | 0.7383 | 0.6341 | 0.93 |
| EEG | Euclidean | 0.0006 | 0.0273 | 0.515 |
| EEG | DTW | 0.1199 | 0.242 | 0.675 |
| EEG | SBD | 0.0604 | 0.1787 | 0.625 |
| EEG | Catch22 | 0.0183 | 0.1081 | 0.57 |
| EEG | Latent ODE | 0.0421 | 0.1246 | 0.605 |
| EEG | TS2Vec | 0.6874 | 0.6159 | 0.915 |
Table 1. Results of Numerical Experiments. The CLOCK method is listed along with its accompanying kernel. Information on the kernels used can be found in Appendix C. The Gaussian Multiscale kernel (GM) with one bandwidth per dimension was the default choice. The Gaussian Uniscale kernel (GU) with a single bandwidth for all dimensions was chosen for high-dimensional data. The quadratic kernel (quad) and Laplace kernel were also used. Best results for each metric are in bold.
Ablation studies
We carried out two ablation studies to determine the the rate of convergence of time-averaged occupation kernels to the associated KMEs of SRB measures, and the quality of the clustering of the CLOCK method as a function of trajectory length. For both experiments, we used the Gaussian uniscale kernel (see section C for details).
The convergence rate of time-averaged integrals of observables over trajectories to the spatial integral of the observable with respect to the associated SRB measure is guaranteed to be for systems with sufficient regularity (see [38] for details). We computed the empirical convergence rate for the 40D Lorenz-96 system with . To do so, we integrated a single trajectory over the interval with random initial condition in the box . We computed a time-averaged occupation kernel for this trajectory, and used it as a proxy for the KME of the SRB measure of the Lorenz-96 attractor, denoted . For each of a collection of equally-spaced increasing time-points , , we integrated 1000 trajectories over the interval with random initial conditions drawn from the box and computed for , , where is the time-averaged occupation kernel of trajectory of length . A log-log plot of the resulting mean error over all trajectories as a function of trajectory length is plotted in Figure 2 (a). Note that after an initial "burn-in" period, we observe a straight line with slope , as expected.
Figure 2. In (a), the log-mean-squared error between the time-averaged occupation kernels and the empirical KME of the SRB measure of the Lorenz-96 attractor is plotted as a function of the log of the trajectory length , along with a 95% confidence interval (CI). The line with slope and intercept is plotted for reference. In (b), the ARI of clusterings with the CLOCK method on the Lorenz-96 system is plotted with a 95% CI as a function of .
For the next experiment, we generated two sets of trajectories for the 40D Lorenz-96 system, each with 30 trajectories. The first set corresponded to the parameter choice of , and the second to . Each trajectory was integrated over the interval with initial conditions drawn randomly from the box . For each of an equally-spaced increasing sequence of time-points , , we clustered the trajectories truncated to the interval with the CLOCK method. The mean ARI over 40 runs as a function of trajectory length is plotted in Figure 2 (b).
Discussion
As can be seen from Table 1, the CLOCK method achieved superior performance on all but one dataset. The quadratic kernel failed on the Van der Pol dataset most likely due to the degree mismatch between the underlying system and the kernel. However, no other method did as well as CLOCK overall, with the closest competitor being TS2Vec. TS2Vec experienced a significant degradation of performance on the Kuramoto dataset. This may be due to the trajectories in this dataset being locally similar but globally divergent (see Figure 1d), so that the hierarchical contrastive framework encoded dissimilar trajectories with similar feature vectors. We speculate that the overall good performance of the CLOCK method can be attributed to its capacity to accommodate the dynamical information contained in the trajectories. Although Latent ODE also incorporates dynamical information, it yields a single dynamical model for trajectories with multiple dynamics, which may be problematic. These results highlight the need for a method specifically-designed to cluster trajectories generated by differential equations.
In Appendix F, we demonstrate empirically how the CLOCK algorithm maintains good performance on extremely noisy datasets.
Limitations
While the proposed embedding and clustering framework can be applied to general time series without explicit assumptions, its theoretical motivation relies on the setting where trajectories are generated by dynamical systems. In practice, there are datasets for which this assumption is not satisfied. In such cases, it may be necessary to first transform the data into a representation that better reflects an underlying dynamical structure, for instance via delay-coordinate embeddings such as Takens’ embedding.
The method assumes that the time series are of sufficient length to capture the nature of the invariant measures of the attractors. Moreover, there exist trajectories generated by different dynamical systems that exhibit distinct short-term (transient) behaviors but share the same asymptotic behavior, i.e., are associated with the same SRB measure. In such cases, the proposed method may fail to distinguish between them, as it clusters trajectories based on their long-term statistical properties rather than their transient dynamics.
Conclusion
We have presented a clustering method using a novel dissimilarity metric for time series. The metric is given by the squared RKHS norm between time-averaged occupation kernels. The Palis conjecture and Theorem A.1 assert that this dissimilarity metric measures how distinct the associated SRB measures are, and thus is able to quantify the difference in the dynamics between two time series.
The method has displayed robust performance on a wide variety of synthetic datasets: low and high dimensions, regular and irregular spacing, and all with a significant amount of noise. This suggests that the RKHS distance between time-averaged occupation kernels is an effective metric whenever it is suspected that the data was generated by differential equations. In
addition, this method achieved the best performance on a real dataset consisting of EEG measurements.
Future work
There are a several directions for future research: Spectral clustering may be replaced with K-Means or community clustering. Additionally, different kernels may be selected to compute embeddings, and there are many ways to convert dissimilarities into similarities. A particularly promising direction is the extension of our framework to stochastic dynamical systems, including stochastic differential equations and continuous-time Markov chains where clustering could be performed based on invariant measures.
AI Use Statement
We were assisted by the Gemini 3.1 Pro LLM model in this research. Specifically, the LLM assisted in locating the foundational result that this work is based on: the Palis conjecture justifying the use of time-averaged occupation kernels as feature vectors for time series. Additionally, the LLM assisted with the design of the numerical experiments by recommending datasets and evaluation metrics to use. However, all theoretical proofs and numerical experiments were carried out by the authors themselves, and all theoretical results referenced by the LLM were verified in the original source material. We take responsibility for the final content of this work.
Supplementary Materials
Theoretical results
An SRB measure for an attractor of an ODE defined on a compact manifold has the property that
for every continuous function and trajectory such that , the basin of attraction for . The Palis conjecture states that, for a dense set in the set of all maps () on with the topology, there exists a finite collection of SRB measures such that has total Lebesgue measure. That is, where denotes the Lebesgue measure.
We wish to show that (17) extends to feature vectors in an RKHS. More specifically, we want to show that
This is the content of Theorem A.1 below. In Corollary ???, we show that the rate of convergence is .
Proposition A.1. Let be a continuous kernel defined on a compact manifold . Let be an increasing sequence of positive numbers such that . Let denote a trajectory generated by an ODE such that , the basin of attraction of an SRB measure . Define the linear functional according to
Then each is bounded, and their Riesz representers, denoted , converge weakly to the function defined by
Proof. The boundedness of the follows from the compactness of and the continuity of (see [27] for details). For the same reason, the operator
is bounded, and its Riesz representer is . The functions are given by
Because is continuous, every function in the corresponding RKHS is continuous. Thus, since is an SRB measure, we get
Lemma A.1. Let and be as in Proposition A.1. Then the following holds:
as .
Proof. We have
Since is continuous and is compact, is a Mercer kernel. Thus,
for some collection of non-negative eigenvalues and eigenfunctions . Let
We have that for any fixed , the sequence converges as since is an SRB measure. Likewise, for any fixed , the sequence has a limit as since the sequence of partial sums converges to the time-averaged double integral of the kernel over two copies of the trajectory (using the Dominated Convergence Theorem to pass the limit inside the integrals). We have that
where in the second equality we used the Dominated Convergence Theorem with the bound
We also have
using Dominated Convergence in the first equality. Thus, we need to show that the order of the limits is exchangeable. Let denote the limit . We have
where on the last line we used the uniform convergence of the Mercer kernel on . The bound on does not depend on and converges to zero. Thus, convergence of in is uniform, and we may apply the Moore-OsGood Theorem to show that the order of the limits is exchangeable. The result follows. □
Theorem A.1. Let the setting of Proposition A.1 hold. Then we have the following:
as .
Proof. By Proposition A.1 and Lemma A.1, we have
For systems with sufficient regularity (for example, hyperbolic systems; see [38] for details), the time-average of an observable over a trajectory converges to the spatial average with respect to the associated SRB measure at a rate of . We use this fact to show that the time-averaged occupation kernels with random initial conditions drawn from Lebesgue-absolutely-continuous distributions converge (in probability) to the KMEs of SRB measures at the same rate.
Definition A.1. Let be a () map on . Let be the flow of at time . Define the centered-kernel function according to:
Define the auto-correlation function for as:
First, we prove a convergence result for the case that the initial condition is distributed according to , an SRB measure for . We then extend the result to more general measures in the next result.
Lemma A.2. Assume we are in the setting of Definition A.1. Let , where is an SRB measure for . Suppose that the following regularity condition holds:
Then we have the following convergence in probability:
at the rate .
Proof. Denote by the following:
where . We have
where is the centered kernel in Definition A.1. We have
by integrating the right side of (39) with respect to over and using the -invariance of . Taking the change of variables , we get
Using Markov’s inequality in conjunction with the regularity condition, we get that
Set . Then, with probability at least , we have
giving us a convergence rate of .
Suppose that the system is exponentially mixing, meaning that correlations decay exponentially [39]. Then the auto-correlation function of the centered kernel will be integrable for kernels which are -Hölder continuous, for . This holds since, in this case, [7], and so . Standard kernels such as Gaussian, Laplace, Matérn, and polynomial satisfy this condition.
Setting. Let be a vector field on with flow , and let be compact and forward invariant, i.e. for all (e.g. a trapping region containing the attractor). Let be a -invariant Borel probability measure on , let be a continuous positive-definite kernel on with RKHS , and write and . For and define
Since and are continuous, is continuous from to ; in particular it is Borel measurable.
Hypotheses.
(K) Hölder feature map. There exist and such that for all .
(L) Rate under the invariant measure. There exist and such that, for ,
Assumption A.1 (Asymptotic phase with absolutely continuous phase distribution). Let , where is a Borel probability measure on . There exist a random variable with values in , defined on a common probability space with , a constant , and a random variable that is finite almost surely, such that
(A1) for all , almost surely;
(A2) the law of is absolutely continuous with respect to .
Assumption A.1 says that the orbit of a typical initial condition is exponentially asymptotic to the orbit of a point on the attractor (an asymptotic phase of ), and that the law of is not concentrated on a -null set. For hyperbolic attractors this is the mechanism by which SRB measures are physical.
Theorem A.2. Assume (K), (L), and Assumption A.1. Then
More precisely, for every there exists such that
Proof. By the triangle inequality,
Term (I). By (K) and (A1), almost surely,
Term (II). Let , which exists by (A2) and is finite -almost everywhere. For let , a Borel set. By Chebyshev’s inequality and (L), for . Hence, for any ,
Conclusion. Fix . Since and almost surely, choose with and with . Set . Taking above gives . On the event and for , term (I) is at most . With , a union bound gives for all .
Remark A.1 (Sufficient conditions for (L)). The proof of Lemma A.2 establishes , so (L) holds with whenever . A weaker sufficient condition is that the primitive is bounded, : integrating by parts in the proof of Lemma A.2,
This weaker condition covers periodic attractors, for which is periodic and .
Verification for the kernels and systems of Section 3.1
Kernels. All kernels of Appendix C satisfy (K). This follows from together with for , and, for the quadratic kernel, from its explicit feature map . With :
GU with bandwidth : , ;
GM with bandwidths : , ;
Laplace with bandwidths : , ;
quadratic: , .
The bandwidths are computed once from the data and then held fixed; the results apply conditionally on them.
FitzHugh–Nagumo. For both parameter sets the unique equilibrium is a hyperbolic sink (Jacobian eigenvalues and ), and numerically all initial conditions in converge to it. Hence and , so (L) holds with . Convergence to is exponential and uniform on compact subsets of the basin, so Assumption A.1 holds with .
Van der Pol. For there is a unique hyperbolic attracting limit cycle , of period , attracting every point except the origin. For , each full period contributes exactly , so with , and (L) holds with . (Here is periodic, so and Lemma A.2 does not apply as stated; cf. Remark A.1.) Take to be the asymptotic phase of on : (A1) holds by hyperbolicity of , and since the asymptotic-phase map is a submersion on the basin [11], (A2) holds when has a density.
Approximating the dissimilarity measure
We use integral quadratures to approximate the integrals in the dissimilarity measure in 9. Assuming ,
The trapezoidal quadrature weights for the i-th time series (and similarly for the j-th one) are defined as:
If the time series is sampled uniformly such that , the weights simplify to:
Kernels used
We used several different kernels in the numerical experiments in § 3. We list their definitions here.
Gaussian uniscale (GU):
where we set where ranged over each individual observation in the dataset. This kernel was used for high-dimensional data.
Gaussian multiscale (GM):
where was set to be the standard deviation of the coordinate of the dataset. This is the default kernel for CLOCK.
Laplace:
where was set to be the standard deviation of the coordinate of the dataset.
Quadratic (quad):
Formulas for evaluation metrics
We provide explicit formulas for the evaluation metrics in § 3.3.
Adjusted Rand Index (ARI): The formula is given by
where is the total number of time series in the dataset, is the number of trajectories that are in both the true cluster and the predicted cluster , is the total number of samples in true cluster , and is the total number of samples in predicted cluster .
Normalized Mutual Information (NMI): The formula is given by
where , , , are as above and by convention is set to 0.
Purity: If is a predicted clustering and a true clustering, then the purity is defined as
In the setting of two predicted clusters and two true clusters, the purity is just the accuracy of the clustering.
Hyperparameter tuning
We validated the hyperparameters of the deep learning methods in § 3.4 using an internal clustering metric, the DBCV.
Density-based clustering validation (DBCV): The DBCV [20] is an internal clustering metric, meaning it can be used in the absence of true cluster labels. This metric assigns a high value to partitions with highly-dense clusters with sparse between-cluster regions. It does so without assuming the clusters are globular, and so can measure the quality of clustering for highly-irregular collections of trajectories, which occurs frequently with chaotic systems, for example. The DBCV score is normalized to be in the range with 1 being a perfect clustering and -1 being exceptionally poor. We found it to be superior to the silhouette score [31] for the task of validating hyperparameters.
Noise robustness experiment
We speculate that the CLOCK method is highly resistant to zero-mean Gaussian noise since the feature vector is obtained by performing an integration along a trajectory. The integral effectively cancels out the perturbations caused by the noise. In order to gauge the impact of noise on the performance of CLOCK, we performed an experiment in which we gradually increased the noise level in the Lorenz96 dataset while leaving the true trajectories fixed. The CLOCK method with Gaussian uniscale and quadratic kernels (see Appendix § 9) was fit on each dataset, and we evaluated the quality of the clusterings. We ran this experiment with the baseline method TS2Vec to use as a comparison. The results are graphically represented in Figure 3.
Figure 3. Plots of evaluation metrics in the noise robustness test of Appendix § 12. The horizontal axis represents the standard deviation of the Gaussian noise added to each observation. The vertical axis represents one of the evaluation metrics: ARI, NMI or purity. In Figure 3a, we have ARI results, in Figure 3b, we have results for NMI, and in Figure 3c, we have purity results.
The results of the noise robustness experiment show that CLOCK is resistant to noise. It should be noted that the amount of noise added to the trajectories was extreme, even on the lower settings. In Figure 3, we see that CLOCK with Gaussian kernel fails as the noise level approaches 3.0, while CLOCK with quadratic kernel is affected at level 2.0. Thus, the choice of kernel is an important aspect of this clustering method. In practice, the kernel may be treated as a hyperparameter and validated using an internal clustering metric. TS2Vec generally performed worse than CLOCK with either kernel.
Assets used
The assets we used can be divided into methods, back-ends, and datasets.
Methods
Dynamic Time Warping (DTW): We used the
dtwfunction from the Python packagetslearn.metrics. This has the BSD-2-Clause license. We used version 0.8.0.
Shape-Based Distance (SBD): We used the package
scipy.signaland the functioncorrelate. This has the BSD 3-Clause license. We used version 1.17.1.
Catch-22: We used the package
pycatch22and the functioncatch22_all. This has the GNU General Public License v3 (GPLv3) license. We used version 0.4.5.
Latent ODE: We used the
pytorchpackage and thetorchdiffeqpackage. Thepytorchpackage has the BSD 3-Clause license. We used version 2.10.0. Thetorchdiffeqpackage has the MIT license. We used version 0.2.5.
TS2Vec: Code was downloaded from the GitHub repository at https://github.com/zhihanyue/ts2vec. This has the MIT license. We used the AAAI 2022 Official Release.
Spectral clustering: We used
sklearn.clusterpackage and the functionSpectralClustering. This has the BSD 3-Clause license. We used version 1.8.0.
Back-ends
JAX: We used the
jaxandjaxlibPython packages. These have the Apache 2.0 license. We used version 4.30.0 ofjaxand version 4.30.0 ofjaxlib.
Datasets
Bonn EEG dataset: Taken from the archive available at https://www.upf.edu/web/ntsa/downloads/-/asset_publisher/xvT6E4pczrBw/content/2001-indications-of-nonlinear-deterministic-and-finite-dimensional-structures-in-time-series-of-brain-electrical-activity-dependence-on-recording-regi. This data is in the public domain.
References
- [1]José F Alves. Nonuniformly hyperbolic attractors. Springer, 2020.DOI
- [2]Enrique Amigó et al. “A comparison of extrinsic clustering evaluation metrics based on formal constraints”. In: Information Retrieval 12.4 (2009), pp. 461–486.
- [3]Ralph G Andrzejak et al. “Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state”. In: Physical Review E 64.6 (2001), p. 061907.pubmed.ncbi.nlm.nih.gov/11736210
- [4]Donald J Berndt and James Clifford. “Using dynamic time warping to find patterns in time series”. In: Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining. 1994, pp. 359–370.
- [5]Srinjoy Das, Hrushikesh N Maskar, and Alexander Cloninger. “Kernel distance measures for time series, random fields and other structured data”. In: Frontiers in Applied Mathematics and Statistics 7 (2021), p. 787455.arxiv.org/abs/2109.14752
- [6]Arthur P Dempster, Nan M Laird, and Donald B Rubin. “Maximum likelihood from incomplete data via the EM algorithm”. In: Journal of the Royal Statistical Society: Series B (Methodological) 39.1 (1977), pp. 1–22.DOI
- [7]Dmitry Dolgopyat. “On decay of correlations in Anosov flows”. In: Annals of Mathematics 147.2 (1998), pp. 357–390.
- [9]Andrew M Fraser and Harry L Swinney. “Independent coordinates for strange attractors from mutual information”. In: Physical Review A 33.2 (1986), p. 1134.pubmed.ncbi.nlm.nih.gov/9896728
- [10]Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. “Large sample analysis of the median heuristic”. In: arXiv preprint arXiv:1707.07269 (2017).arxiv.org/abs/1707.07269
- [11]John Guckenheimer. “Isochrons and phaseless sets”. In: Journal of Mathematical Biology 1.3 (1975), pp. 259–273. DOI: 10.1007/BF01273747.DOI
- [12]Matthew B Kennel, Reggie Brown, and Henry DI Abarbanel. “Determining embedding dimension for phase-space reconstruction using a geometrical construction”. In: Physical Review A 45.6 (1992), p. 3403.pubmed.ncbi.nlm.nih.gov/9907388
- [13]Yoshiki Kuramoto. “Self-entrainment of a population of coupled non-linear oscillators”. In: International Symposium on Mathematical Problems in Theoretical Physics: January 23–29, 1975, Kyoto University, Kyoto, Japan. Springer. 2005, pp. 420–422.
- [14]Edward N Lorenz. “Deterministic nonperiodic flow”. In: Journal of Atmospheric Sciences 20.2 (1963), pp. 130–141.
- [15]Edward N Lorenz. “Predictability: A problem partly solved”. In: Proc. Seminar on Predictability. Vol. 1. Reading. 1996, pp. 1–18.
- [16]Carl H Lubba et al. “catch22: CAnoniCal Time-series Characteristics: Selected through highly comparative time-series analysis”. In: Data Mining and Knowledge Discovery 33.6 (2019), pp. 1821–1852.
- [17]Pankaj Malhotra et al. “TimeNet: Pre-trained deep recurrent neural network for time series classification”. In: arXiv preprint arXiv:1706.08838 (2017).arxiv.org/abs/1706.08838
- [18]Aaron F McDaid, Derek Greene, and Neil Hurley. “Normalized mutual information to evaluate overlapping community finding algorithms”. In: arXiv preprint arXiv:1110.2515 (2011).arxiv.org/abs/1110.2515
- [19]James B McQueen. “Some methods of classification and analysis of multivariate observations”. In: Proc. of 5th Berkeley Symposium on Math. Stat. and Prob. 1967, pp. 281–297.
- [20]Davoud Moulavi et al. “Density-Based Clustering Validation”. In: Proceedings of the 2014 SIAM International Conference on Data Mining (SDM). 2014, pp. 839–847. DOI: 10.1137/1.9781611973440.96.DOI
- [21]Jinichi Nagumo, Suguru Arimoto, and Shuji Yoshizawa. “An active pulse transmission line simulating nerve axon”. In: Proceedings of the IRE 50.10 (2007), pp. 2061–2070.
- [22]Aileen Nielsen. Practical time series analysis: Prediction with statistics and machine learning. O’Reilly Media, 2019.
- [23]Jacob Palis. “A global view of dynamics and a conjecture on the denseness of finitude of attractors”. In: Astérisque 261.xxiiixiv (2000), pp. 335–347.
- [24]John Paparrizos and Luis Gravano. “k-Shape: Efficient and Accurate Clustering of Time Series”. In: SIGMOD Rec. 45.1 (June 2016), pp. 69–76. ISSN: 0163-5808. DOI: 10.1145/2949741.2949758. URL: https://doi-org.proxy.lib.pdx.edu/10.1145/2949741.2949758.DOI
- [25]François Petitjean, Alain Ketterlin, and Pierre Gançarski. “A global averaging method for dynamic time warping, with applications to clustering”. In: Pattern Recognition 44.3 (2011), pp. 678–693.DOI
- [26]Ali Rahimi and Benjamin Recht. “Random Features for Large-Scale Kernel Machines”. In: Advances in Neural Information Processing Systems. Vol. 20. Curran Associates, Inc., 2007, pp. 1177–1184.
- [27]Victor William Rielly et al. “MOCK: an Algorithm for Learning Nonparametric Differential Equations via Multivariate Occupation Kernel Functions”. In: Transactions on Machine Learning Research (2025). ISSN: 2835-8856. URL: https://openreview.net/forum?id=fjVIp2Z9RS.
- [29]Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. “Latent ordinary differential equations for irregularly-sampled time series”. In: Advances in Neural Information Processing Systems 32 (2019).
- [30]H. Sakoe and S. Chiba. “Dynamic programming algorithm optimization for spoken word recognition”. In: IEEE Transactions on Acoustics, Speech, and Signal Processing 26.1 (1978), pp. 43–49. DOI: 10.1109/TASSP.1978.1163055.DOI
- [31]Ketan Rajshekhar Shahapure and Charles Nicholas. “Cluster quality analysis using silhouette score”. In: 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA). IEEE. 2020, pp. 747–748.
- [32]Bharath K Sriperumbudur, Kenji Fukumizu, and Gert RG Lanckriet. “Universality, Characteristic Kernels and RKHS Embedding of Measures”. In: Journal of Machine Learning Research 12.7 (2011).arxiv.org/abs/1003.0887
- [33]Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. “Unsupervised learning of video representations using lstms”. In: International Conference on Machine Learning. PMLR. 2015, pp. 843–852.arxiv.org/abs/1502.04681
- [34]Douglas Steinley. “Properties of the hubert-arable adjusted rand index”. In: Psychological Methods 9.3 (2004), p. 386.DOI
- [35]Floris Takens. “Detecting strange attractors in turbulence”. In: Dynamical Systems and Turbulence, Warwick 1980: Proceedings of a Symposium held at the University of Warwick 1979/80. Springer. 2006, pp. 366–381.
- [36]Ryohei Umatani et al. “Time series clustering with an EM algorithm for mixtures of linear Gaussian state space models”. In: Pattern Recognition 138 (2023), p. 109375. ISSN: 0031- 3203. DOI: https://doi.org/10.1016/j.patcog.2023.109375. URL: https://www.sciencedirect.com/science/article/pii/S0031320323000766.
- [37]Balth Van der Pol. “LXXXVIII. On “relaxation-oscillations””. In: The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2.11 (1926), pp. 978–992.DOI
- [38]Lai-Sang Young. “Recurrence times and rates of mixing”. In: Israel Journal of Mathematics 110.1 (1999), pp. 153–188.DOI
- [39]Lai-Sang Young. “Statistical properties of dynamical systems with some hyperbolicity”. In: Annals of Mathematics 147.3 (1998), pp. 585–650.DOI
- [40]Zhihan Yue et al. “Ts2vec: Towards universal representation of time series”. In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. 8. 2022, pp. 8980–8987.arxiv.org/abs/2106.10466

