betaFeedback

WorksComputer Science

MarathonTS: a benchmark of long synthetic time series from window assembly and long-series generators

Abstract

Many applications of synthetic time series require a series that is much longer than the windows on which a generative model is trained. Such a series can be assembled from a pool of short generated windows, which requires deciding which windows are taken, in what order, and how their seams are processed, or it can be produced directly by a long-series generator. This study presents MarathonTS, to our knowledge the first open benchmark that compares both families under one protocol. Every method produces a series of the same length on 9 open hourly datasets with 10 seeds. The series is evaluated by its fidelity to the real data (83 metrics in 12 groups, against three real references: train, validation and an out-of-time split), by its utility (models are trained on the synthetic series only and tested on real data, TSTR against TRTR, in 14 classification and regression tasks), by its memorisation of the training data and by its computational cost. The statistical analysis is carried out as a separate stage that reads the published tables and gives the rankings of every method in every slice, the answers to the research questions with tests and uncertainty, and the results of a pre-specified confirmatory family. In the main run, the methods differ in both confirmatory cells; the choice of windows matters more than the processing of their seams, which does not improve the series, and no method trains the downstream models as well as the real series. The top tier of the leaderboard is reproduced on other hardware, whereas the exact places within it are not. For the reproducibility of the study, the code, the data hashes and all results are publicly available (links below).

MarathonTS at a glance

9 — public datasets

30 — methods — 2 × 12 + 6

10 — seeds per method

83 — fidelity metrics

3 — fidelity refs — train / val / OOT

8 + 6 — TSTR tasks: classif. + regr.

2 — utility tests — val / OOT

10 — research questions

Introduction

Synthetic time series are used to train and test forecasting and classification models when real data are scarce, sensitive or cannot be shared. Many of these applications require a long series: a forecaster learns weekly and monthly patterns only from a series that contains them, a model of rare events needs a sufficient number of such events, and a stress test needs regimes that last longer than a few days. At the same time, generative models for time series are trained on windows of a fixed and short length [1, 2]. A window of one week at an hourly step contains 168 points, while the training series of a dataset covers months or years. This raises the question of how to obtain a synthetic series that is many windows long and remains realistic.

Two approaches to this problem can be distinguished. In the first approach, the series is assembled from a pool of short generated windows. Assembly involves three decisions: which windows are taken, in what order, and how the seams between neighbouring windows are processed. Each of these decisions has a long history outside time series, in concatenative speech synthesis [3], video textures [4] and image quilting [5], and the window pool can be produced by any generator. In the second approach, a long-series generator produces the whole series directly: a diffusion model extends the series window by window [2], or a pretrained forecasting foundation model [6, 7, 8] continues the training series for as long as needed. The two families make different trade-offs. Assembly keeps the quality of individual windows, but it can create seams and lose structure longer than a window. A long-series generator does not select from a fixed window pool, but chunked generation can introduce boundaries; it may also drift, smooth the series, or reproduce data seen in pretraining. Both families therefore require evaluation of local transitions and longer-scale structure.

These families have not been compared under one protocol. A recent study varies how overlapping windows are blended into a long series [9], but it does not vary which windows are taken and in what order, and it does not compare assembly with long-series generators. Evaluations of synthetic time series score sets of independent windows of the training length [10, 11]; therefore, they cannot capture the seams, the structure longer than a window, or the distance of the series to data from a later period. The real reference in such evaluations is usually the training data or a random split of the same windows. Without a common protocol, it remains unclear when assembly from windows is sufficient, which of its decisions matter, and whether a pretrained forecaster is a good source of long synthetic series.

This study presents MarathonTS, a benchmark of long synthetic time series. To our knowledge, it is the first open benchmark that compares assembly from a pool of generated windows with long-series generators, including pretrained forecasting foundation models, under one protocol, and the first to evaluate whole long series against a real reference from a later period (§ 3). The contributions of the study are as follows:

  • A common task and protocol (§ 4 and § 5). Every method receives only the training part of a series and returns one series of length KK times the training window, the same for every method and approximately as long as the training part, in the same channels. The main run compares 30 methods on nine open hourly datasets with ten seeds: nine assemblers and three seam joiners on each of two window pools, and six long-series generators (Diffusion-TS, Chronos, TimesFM 3.0, Timestral and two sampling modes of zEDGE_TS). Within one pool, all assemblers read the same windows, so they differ only in selection, order and seam processing.

  • Evaluation of the whole series (§ 6). Fidelity is measured by 83 metrics in 12 groups, including groups for the seams, for structure longer than a window and for tail events, against three real references: train, a validation split and an out-of-time split. Utility is measured by training downstream models on the synthetic series only and testing them on real data (TSTR against TRTR) in 14 classification and regression tasks. Memorisation and cost are measured separately.

  • A pre-specified statistical analysis (§ 7). The analysis consists of rankings with uncertainty and robustness checks, research questions with an empirical noise level as the smallest effect of interest and tests of both difference and equivalence, and a small confirmatory family that carries the main conclusion. The analysis was frozen before the main run was carried out.

  • Open, reproducible and extensible software (§ 8). Every component is registered by name, so a new dataset, generator, assembler, metric or test enters the comparison through a plugin. Incremental recomputation and run profiles make it possible to run the benchmark on a single GPU.

The main run gives the following picture (§ 9). The methods differ in both cells of the confirmatory family, whereas the confirmatory comparisons of long-series generation with assembly, of window selection with random selection and of seam processing with no processing are inconclusive under the pre-specified joint rule of statistical and practical significance (§ 7.4). Descriptively, window selection often moves the series towards the real data compared with a random choice of windows, whereas the seam joiners, tested on the zseries assembly of the VAE pool, often reduce utility compared with the same assembly without them. The top tier of the leaderboard, the methods not shown to be worse than the observed leader, holds 15 of 24 methods. Its first eleven places are taken by the methods of the proprietary zEDGE pool and by zEDGE_TS1, and the best open method is the kernel-herding assembler on the VAE pool. Among the long-series generators, again descriptively, Diffusion-TS and TimesFM are competitive with assembly from the open VAE pool, Chronos and Timestral fall behind it, and assembly from the zEDGE pool is ahead of all four. No method reaches the utility of the real series: the transfer ratio is significantly below 1 for every method, although for the best methods the gap is small. A replication run on a different hardware and software stack, covering the 13 ranked methods of the VAE pool and the open long-series generators (the zEDGE methods were not rerun), gives similar rankings of these methods with one change in top-tier membership; the order within the tier differs between the runs.

The rest of the paper is organised as follows. § 3 is devoted to related work; § 4 defines the task and the data; § 5 describes the compared methods; § 6 presents the evaluation; § 7 describes the statistical analysis; § 8 describes the software; § 9 shows the results; § 10 discusses the limitations; § 11 contains the conclusions.

Related work

Four directions of existing work are relevant to this study: generative models for time series, the evaluation of synthetic time series, the assembly of long content from pieces in other fields, and the resampling of real series in statistics.

Generative models for time series

Window generators. Most generative models for time series learn the distribution of windows of a fixed length. TimeGAN [1] trains a GAN in a learned latent space with an additional supervised loss on one-step-ahead transitions. Variational autoencoders, including the β\beta-VAE [12] that is used as the pool generator in this study, encode a whole window into one latent vector. Diffusion models such as CSDI [13] and Diffusion-TS [2] denoise a window as a whole and can also fill in its missing part given the observed rest. In all of these models, the window length is fixed at training time, so the length of a sample is bounded by what fits into the network and into the training windows. Few models target long series directly: PSA-GAN [14] grows the generated length progressively during training, and a model with an imputation mode can extend a series window by window, conditioning each new window on the tail of the previous one.

Pretrained forecasting models. A second source of long series is pretrained forecasting foundation models used as samplers. Chronos [6] tokenises scaled values and samples trajectories autoregressively; TimesFM [7] is a decoder-only model that forecasts patches of the series; Timestral [8] is a recent family of probabilistic forecasting foundation models. Conditioned on the end of the training series, such a model produces a continuation of any length without being trained on the dataset. Its output is a forecast rather than a sample of the training distribution: the median forecast is a single deterministic path, smoother than any sampled trajectory. Benchmarks of these models, such as TSFM-Bench [15], evaluate them as forecasters, by the error of the forecast, and not as sources of synthetic data. Whether these models are good sources of long synthetic series is an empirical question, which the benchmark answers on equal terms with the trained generators.

Evaluation of synthetic time series

Evaluation axes. Synthetic time series are evaluated along three axes [10]. Fidelity compares the synthetic and real distributions: through two-sample statistics such as the maximum mean discrepancy [16], through classifier two-sample tests [17] and their recurrent variant, the discriminative score of Yoon et al. [2019], through the Fréchet distance [18] between window embeddings (Context-FID uses TS2Vec [19] embeddings [14]), through precision, recall, density and coverage in a feature space [20], and through hand-crafted statistics of the series such as catch22 [21]. Utility measures whether the synthetic data can replace the real data in a downstream task: train on synthetic, test on real (TSTR) against train on real, test on real (TRTR) [22], and the closely related predictive score of Yoon et al. [2019]. Privacy measures whether the generator reproduces its training data, usually through the distance from a synthetic sample to its nearest training sample.

Benchmarks and libraries. TSGBench [10] compares generators on a common set of datasets, metrics and a ranking procedure; TSGM [11] provides generators, augmentations and evaluation metrics in one framework. Recent benchmarks narrow the question. ConTSG-Bench [23] evaluates generation under a condition, by fidelity and by adherence to the condition. CTBench [24] evaluates generators of cryptocurrency series in a walk-forward design, in which TSTR utility is measured on the real days that follow each training period. Amorim et al. [2026] train forecasters on synthetic series that fully replace the real ones (TSTR), test them on the final observations of each series and weigh their error against a distance-based privacy risk. Seq2Synth [26] shows for sequential tabular data that rankings by static fidelity and by temporal fidelity diverge. In all of these works, as in the papers that introduce the generators, the evaluated object is a set of independent windows or sequences of the training length, and the fidelity reference is the data the generator was trained on or a random split of the same data; a later period, where it is used, enters only as the test set of the utility tasks. Therefore, the properties that appear only in a long series are left out: the seams between consecutive windows, structure longer than a window (slow trends, regimes, the distribution of long excursions), and the distance to data from a later period. MarathonTS evaluates whole series of one common length, adds metric groups for seams and long-range structure, and scores every series against three real references, including an out-of-time split (§ 6).

Assembly of generated windows. The closest study to ours is that of Long et al. [ [9], which starts from the same problem: differentially private generators produce windows of a fixed length, while forecasters need a long series. In that study, the windows of a differentially private diffusion model are placed in their generated order, overlapping windows are blended with different overlap rates and weighting schemes, and the result is evaluated by the continuity at the boundaries, by fidelity and by TSTR utility against TRTR on four datasets. The authors find that the best assembly depends on the forecaster and that better continuity or fidelity does not reliably give a lower forecast error, which agrees with our premise that the series must be judged by several criteria at once. The study varies how overlapping windows are blended; it does not vary which windows are taken and in what order, and it does not compare assembly with long-series generators.

Position of this study. To our knowledge, no other open benchmark compares the selection, order and seam processing of generated windows with long-series generators, including pretrained forecasting foundation models, under one protocol, and scores the fidelity and utility of whole long synthetic series against an out-of-time real reference. Each part of this protocol has precedents, cited above and in § 6; the combination is new. The statement rests on searches of arXiv and the web in October and November 2026 for benchmarks of time series generation, for the assembly of generated windows into long series, and for forecasting foundation models used as generators of synthetic data.

Assembling long content from pieces

Selection and order of pieces. The assembly of a long output from a library of short pieces has a long history outside time series. Concatenative speech synthesis selects and orders recorded units by minimising the sum of a target cost and a concatenation cost [3]. Video textures [4] and motion graphs [27] build a graph of transitions between similar frames or poses and produce an unbounded sequence by walking it.

Processing of seams. Image quilting [5] places overlapping patches and cuts each seam along a minimum-error boundary, and graphcut textures [28] find the seam by a minimum cut. WSOLA [29] joins overlapping audio segments at the shift that maximises waveform similarity, which is a refinement of plain overlap-add [30]. These four fields (speech, video, images and audio) separate the same three decisions that define assembly from a window pool: which pieces are used, in what order, and how the seams are processed. Several assemblers and seam joiners of the benchmark (§ 5) are one-dimensional versions of these methods, each with the seam criterion of its own literature.

Reconciliation during generation. Diffusion models offer a different route to long content: overlapping pieces are reconciled during generation instead of after it. RePaint [31] fills in a missing region by resampling the known part at every denoising step; MultiDiffusion [32] averages overlapping windows at every step; DiffCollage [33] generates the pieces of a factor graph in parallel. The analogue of these methods for time series is the window-by-window extension of a diffusion generator described above.

Resampling real series

Bootstraps and surrogates. Statistics has its own long tradition of producing new series from an observed one. Block bootstraps concatenate blocks of the real series: of fixed length [34], wrapped into a circle [35], of random length [36], with an automatically chosen length [37], or aligned with the seasonal phase [38]. The sieve bootstrap [39] fits an autoregression and resamples its residuals, the Markov bootstrap [40] resamples from an estimated transition density, and AAFT and IAAFT surrogates [41, 42] randomise Fourier phases while keeping the spectrum and the distribution of values.

Matching of the next piece. Two schemes choose the next piece by how well it continues the current one. The matched-block bootstrap [43] draws the next block with a kernel of how well its beginning matches the end of the current block, and the nearest-neighbour bootstrap [44] takes the successor of one of the kk nearest neighbours of the current state.

These methods resample the real training series itself, so their blocks are copies of real data by construction; they answer a different question from the one studied here. In this study, their matching and resampling ideas are used only on top of a pool of generated windows: the matched-block and kk-nearest-neighbour assemblers of § 5 apply these two selection rules to the pool instead of to the real series.

The task, the data and the splits of the benchmark are described in the next section.

Benchmark design

Task: a long synthetic series

Let x1,…,xnx_1,\ldots,x_n be an hourly univariate series, split in time into a training part of length ntrainn_{\mathrm{train}}, a validation part (val) and an out-of-time part (OOT). A generative model is trained on windows of length TT. The task is to produce one synthetic series of length K⋅TK \cdot T, where

T=168,K=round⁡(ntrain/T).(1)T = 168,\qquad K = \operatorname{round}(n_{\mathrm{train}}/T). \tag*{(1)}

That is, the synthetic series is approximately as long as the real training series and KK times as long as a training window. The window length is one week of hours for every dataset, so a window contains the full weekly cycle, but nothing longer than a week.

Every method receives the training part only and returns its series in the same three channels: the value scaled with the bounds of train and the hour of the week encoded on a circle (§ 4.3). For an assembly method, the calendar of the series is decoded from the two calendar channels of its windows: their angle is rounded to the nearest hour of the week. A window generator produces these channels together with the value, so the decoded calendar can deviate from a valid one inside a window as well as at a seam. The open long-series generators generate the value only and receive the calendar: Diffusion-TS starts it at a random hour of the week, and the pretrained forecasters continue the calendar of train; zEDGE_TS generates it as part of its windows. The calendar of a series enters the weekly and daily profiles of the fidelity metrics (F3.8, F4.1–F4.3, F8.4) and the hour-of-week feature of every downstream task (§ 6.2). The series is then converted back to real units and evaluated (§ 6). Each method is run on every dataset with 10 seeds, which gives one series per (dataset, method, seed).

The compared methods are divided into two families:

  • an assembly method takes a pool of N≥KN \ge K windows sampled from a window generator, selects KK of them without repetition, orders them, and optionally processes the seams between neighbours;

  • a long-series generator produces the series of length K⋅TK \cdot T directly, without a pool.

All assembly methods on one pool read the same windows, so they differ only in selection, order and seam processing. The methods of both families are described in § 5.

Datasets

Table 1 lists the nine open datasets used in the benchmark. Each dataset yields a single target series, brought to an hourly step, and every dataset is used in every downstream task, so the whole benchmark runs on one set of datasets.

The datasets are chosen so that they differ in the properties that make long series hard to generate:

  • Electricity, PJME, Metro and Bike have strong weekly seasonality;

  • Jena and ETH1 have strong daily but weak weekly seasonality, which checks that the hour-of-week calendar does not impose spurious structure;

  • PM2.5, Household and Bitcoin have heavy tails and spikes;

  • Bitcoin has volatility clustering and almost no seasonality;

  • Bike, ETTh1 and Bitcoin drift in level.

Household, PM2.5, Metro, Bike and PJME contain gaps or duplicate hours that need preprocessing. The datasets also differ in length: on the full series, KK ranges from about 60 (ETTh1, Bike) to 519 (PJME). Sources and licences are listed in section A.

Leakage into pretrained models. Pretrained foundation models may have seen a benchmark dataset during pretraining or post-training, and in this case their continuation of the training series is not out-of-sample. Therefore, every pair of a pretrained model and a dataset is audited against the primary descriptions of the training corpora and, where the corpus is public, against the data themselves. A model is left out on a dataset if its corpus contains the same data (an exact pair) or the same object at other stations, years or resolution (a related pair); pairs that share only the phenomenon are kept. By this rule, Chronos is left out on Electricity (exact) and Beijing PM2.5 (related), and TimesFM 3.0 on Household (exact), Beijing PM2.5 and Bitcoin (related). Timestral is pretrained on synthetic data only, but the real data of its post-training are not listed [8], so its pairs are marked unknown and kept (§ 10). The exclusion applies to that method on that dataset in every ranking and test, while the other methods keep every dataset. The full audit is given in section A.

Preprocessing and temporal split

Figure 1 shows the path from the raw files to the model inputs. The raw files are converted losslessly to Parquet. Each dataset is then reduced to an hourly series as follows:

Figure 1. From raw files to model inputs. Each dataset becomes one hourly univariate series. The series is split in time with embargoes. All statistics (scaler, window length, number of windows, task thresholds) are fitted on the training split only. The series is then encoded as three channels in [0,1][0, 1] and cut into windows (red box: the leakage boundary).

  • for the sub-hourly sources (Electricity, Household, Jena), the hourly mean is taken;

  • for Bitcoin, the log of the last price of the hour is taken;

  • for the hourly sources, the values are used as they are.

Gaps of up to 6 hours are interpolated; longer gaps stay missing, and windows that contain them are dropped. Metro is taken after its last gap longer than 30 days, and a missing hour of Bike means zero rentals.

The series is split in time into train, val and OOT, with an embargo of 168 hours between neighbouring parts (Figure 2). The embargo is as long as the longest label horizon and the longest feature lag of the downstream tasks (§ 6.2), so no label or feature of a test row reaches into train. Profiles min and mid take the last 10 000 hours of each dataset and split them 50/25/25. After the embargoes, every dataset has 5 000 training hours, 2 332 val hours and 2 332 OOT hours, and K=30K = 30. Profile max takes the full series and splits it 60/20/20, which gives the values of KK in Table 1. The results section § 9 are obtained for profile min.

DatasetTargetSource stepHoursKK
ETTh1OT1 h17 42062
Electricity Load Diagramssum over 370 clients15 min35 065125
Household Electric PowerGlobal_active_power1 min≈34 590\approx34\,590124
PJM Hourly Energy (PJME)PJME_MW1 h145 392519
Beijing PM2.5pm2.51 h43 800156
Jena Climate 2009–2016T (degC)10 min≈70 000\approx70\,000250
Metro Interstate Traffictraffic_volume1 h≈29 000\approx29\,000103
Bike Sharing (hourly)cnt1 h17 54463
Bitcoin Historical DataClose (log price)1 min≈85 000\approx85\,000305

Table 1. Datasets. Hours: length of the target series after conversion to an hourly step, from the first to the last valid value. KK: number of windows on the full series (60% train, T=168T = 168, profile max); profiles min and mid take the last 10 000 hours of every dataset, which gives K=30K = 30 everywhere.

Figure 2. Temporal split (profile min, drawn to scale). Hatched bars are embargo gaps of 168 h, wide enough to cover every label horizon h≤168h \le168 and the longest feature lag (168). Inside train, the last 10% is held out to stop generator training early. Every statistic the methods may use comes from train. Val is the primary fidelity reference; train and OOT are also references. Val and OOT are the utility test sets.

Everything a method may use is fitted on train only: the min–max bounds of the value, TT, KK, the pool size N=max⁡(300,2K)N = \max(300, 2K), the stride of the training windows, and the thresholds of the downstream tasks. The task thresholds are computed once per split, reviewed and committed to the repository. A run only reads them, and a file computed on another split is refused, so a threshold can never come from the val or OOT part of a run. The series is encoded as three

channels in [0,1][0,1]:

(xt−ℓu−ℓ,1+sin⁡(2πwt/168)2,1+cos⁡(2πwt/168)2),(2)\left(\frac{x_t-\ell}{u-\ell},\frac{1+\sin(2\pi w_t/168)}{2},\frac{1+\cos(2\pi w_t/168)}{2}\right), \tag*{(2)}

where ℓ\ell and uu are the minimum and maximum of train and wt∈{0,…,167}w_t \in\{0,\ldots,167\} is the hour of the week. Training windows of shape (T,3)(T,3) are cut with the smallest stride that gives at most about 5 000 windows (stride 1 in profile min). The last 10% of train is a holdout for early stopping of the generators. It is separated from the rest of train (the core) by another embargo of 168 rows, so that core and holdout windows do not overlap.

Each part of the split has a fixed role:

  • core windows train the generators, and holdout windows stop their training;

  • val is the primary real reference for fidelity and one of the two utility test sets;

  • train and OOT are the other two fidelity references: train shows how well a method reproduces the data it was fitted on, val measures generalisation, and OOT measures generalisation under a shift in time;

  • OOT is also the second utility test set.

Window-pool generator

The benchmark uses two pool generators. The open one is a convolutional β\beta-VAE [12] from the TSGM library [11], shown in Figure 3. The other one, zEDGE, is a proprietary window generator of zypl.ai, evaluated as a black box under the same protocol: it is trained on the core training windows only and returns a pool of windows in the same three channels.

Figure 3. The window-pool generator. A convolutional β\beta-VAE is trained on core training windows and stopped early on the holdout loss. After training, NN windows are sampled from a shrunk prior and written to disk as the pool that every same-pool method shares.

The VAE encodes a window of shape (T,3)(T,3) into a 16-dimensional latent vector and is trained with the loss “reconstruction + β⋅KL+\ \beta\cdot\mathrm{KL}” with β=2\beta=2 (Adam, learning rate 10−310^{-3}, batch 64, up to 400 epochs). Training is stopped early on the holdout loss with a patience of 50 epochs, but the first 150 epochs are not eligible for stopping. The pool consists of N=max⁡(300,2K)N=\max(300,2K) windows decoded from latent vectors drawn from a shrunk prior N(0,0.52I)\mathcal{N}(0,0.5^{2}I), that is, at a reduced sampling temperature, which is known to trade the diversity of samples for their quality [45]. The learning rate and the batch size follow TSGM; β\beta, the latent size, the epoch limits and the prior scale were taken from earlier work of the authors on other data and were not tuned on this benchmark (§ 10). The decoder ends with a sigmoid, so every window stays within the range of train. The pool is drawn anew for every seed and written to disk, and every assembly method of that seed reads the same pool. A zEDGE pool is used in the same way.

In profile min, one VAE is trained per dataset, and the seeds change only the sampling of the pool. In profiles mid and max, a VAE is trained on every seed, so the seeds also capture the variance of training. When a model is shared by several seeds, its training time is still charged to every seed in the cost comparison. The pool generator is a component like any other: any generator that outputs windows in the same three channels can replace the VAE, and every assembly Diagram labels: train core windows (n≤5000n \le5000, TT, 3); fit; β\beta-VAE (tsgm vae_conv5 + BetaVAE); Conv encoder →μ\to\mu, log σ2\sigma^2 (latent 16); z=μ+σ⋅εz = \mu+ \sigma\cdot\varepsilon; Conv decoder →(T,3)\to(T, 3); loss = reconstruction + β⋅\beta\cdot KL; β=2\beta= 2 · Adam 1e-3 · batch 64; holdout windows (last 10% of train); Early stopping after every epoch; holdout loss with z=μz = \mu; epochs 1–150 ignored (min_epochs); patience 50 · ≤\le 400 epochs · keep best weights; best weights; generators/<ds>/tsgm_vae/ replica<r> (min) · seed<k> (mid, max); Sampling (per seed); z∼0.5⋅N(0,I)→z \sim0.5 \cdot\mathcal{N}(0, I) \to decoder; NaN →0.5\to0.5, clip to [0, 1]; Model sharing across seeds (pool_train_replicas); min: 1 model / dataset mid, max: 1 model per seed. Seeds always change sampling. Fit and sampling have separate fingerprints: new NN re-samples without re-training.; pool (NN, TT, 3); NN = max(300, 2K); pools/<ds>/tsgm_vae/seed<k>; every assembler on the VAE pool.

method then runs on the new pool unchanged (§ 8). The methods of both pools enter the same rankings and research questions as every other method (§ 7.5).

The compared methods are described in the next section.

Compared methods

A compared method is either a pool generator combined with an assembler (and optionally a seam joiner), or a long-series generator. Figure 4 shows both families. The benchmark uses two pool generators: the β\beta-VAE of § 4.4 and zEDGE, a proprietary window generator of zypl.ai. Every assembler is run on the pool of each generator, and the name of a method is the pool prefix followed by the assembler. For example, vae_greedy and zedge_greedy apply the same assembler to the two pools. Within one pool, all assemblers read the same windows, so they differ only in which windows they select, in what order, and how they process the seams.

Figure 4. The compared methods. Groups A and B take windows from the same pool, so they differ only in which windows they pick, in what order, and how seams are handled. Group C generates long series directly. Every method returns one series of length K⋅TK \cdot T per (dataset, seed).

The main run (profile min) compares 30 methods:

  • on each pool, nine assemblers and the three seam joiners of section 4.2;

  • six long-series generators: four open ones and two sampling modes of zEDGE_TS.

The methods and their settings are listed in section B.

The comparison is kept neutral by two principles. First, each alternative assembler uses the seam criterion of its own literature. Second, the three solvers that have no criterion of their own (greedy, ATSP and PCTSP) all minimise the same generic cost, the jump at the seam: the Euclidean distance between the last point of a window and the first point of the next, over all three channels. This is the frame distance of video textures [4]. Every generated series is stored with the source of each point and a flag for the seams, so that the seam metrics (§ 6.1) apply to every method. For a long-series generator, the boundaries between its generation chunks count as seams.

Window selection and order

Every assembler of group A draws KK windows from the pool of NN without repetition. The assemblers of the group are described below.

Random path. KK random windows are taken in random order (rand). This is the control of the group: any effect of an assembler is measured against what “just other windows of the same pool” give. In profiles mid and max, a second random path with an independent seed (rand_b) is used to estimate the noise level of the comparison.

Solvers by the jump at the seam. Greedy nearest neighbour starts from a random window and always appends the window with the smallest jump. ATSP solves the asymmetric travelling salesman problem over the whole pool with the Lin–Kernighan–Helsgun heuristic [46] and takes the cheapest run of KK consecutive windows from the tour. Prize-collecting TSP [47] may skip windows at a penalty, so it chooses at once which KK of the NN windows to use and their order.

Resampling rules on the pool. The matched-block bootstrap [43] is a Markov chain over the pool: the next window is drawn with probability proportional to exp⁡(−12(dij/h)2)\exp\left(-\frac{1}{2}(d_{ij}/h)^2\right), where dijd_{ij} is the jump from window ii to window jj and the bandwidth hh is the median jump. kk-nearest-neighbour resampling [44] draws the next window among the k=Nk=\sqrt{N} windows whose first point is nearest to the last point of the current one, with weights inversely proportional to the rank.

Image quilting. Following Efros and Freeman [ [5], the next window overlaps the current one by 12 points and is drawn at random among the windows whose overlap error is within 1.1 times the best. The overlap is then cut at the point of minimum error.

Kernel herding. KK windows are selected greedily to minimise the MMD to the training windows [48, 16] and are put in random order. This assembler changes only the selection, not the order, and therefore makes it possible to separate the two effects.

zseries. zseries is a proprietary assembler of zypl.ai under development (vae_zseries, zedge_zseries). It is evaluated as a black box: its results are reported and ranked like those of any other method, but its code and description are not released. It uses the thresholds of task U2.1, which are fixed on train (§ 4.3); no open assembler uses them (§ 10).

Seam processing

A seam joiner takes the ordered windows of an assembler and processes the seams between them. Neighbouring windows overlap by 12 points. The following joiners are used:

  • crossfade blends the overlap with a Hann window, as in overlap-add synthesis [30];

  • WSOLA [29] first shifts the next window by up to 12 points to maximise the correlation in the overlap and then blends;

  • the boundary cut of image quilting [5] switches from one window to the next at the point of smallest difference, which in one dimension is the minimum-error cut.

Because of the overlap, the joined series is shorter than K⋅TK \cdot T. Therefore, the assembler supplies more than KK windows, and the result is truncated to K⋅TK \cdot T. In profiles and mid, the joiners are applied to the path of zseries; in profile max, to the path of every assembler except the random path and image quilting, which has its own cut. A seam-processing variant is ranked only when every assembler has it (§ 7).

Long-series generators

The methods of group C produce the series of length K⋅TK \cdot T without a pool, as shown in Figure 5.

Figure 5. The baselines that generate long series directly. Diffusion-TS grows the series by autoregressive infilling: each new window takes the last T/2T/2 generated values as known context and fills in the rest. Chronos samples continuations of the end of the training series in chunks of 64 points. TimesFM and Timestral give their median forecast of the K⋅TK \cdot T steps after train, with the whole train as context (up to the model’s limit). zEDGE_TS builds a sequence of latent vectors in blocks (zedge_ts_long) or one vector at a time (zedge_ts_ar) and decodes it into overlapping windows. Boundaries between chunks or blocks count as seams in the seam metrics.

Diffusion-TS. The official implementation of Diffusion-TS [2] is trained on the same core windows as the pool generators and stopped early on the holdout windows. The series is grown by autoregressive infilling: the first window is generated unconditionally, and each next window takes the last T/2T/2 generated values as known context and fills in the remaining T/2T/2 (50 DDIM steps). The calendar channels are given at every position, and only the value is generated. One model per dataset is trained in profiles min and mid, and one model per seed in profile max.

Chronos. Chronos [6] (chronos-t5-small; chronos-t5-base in profile max) is used zero-shot with the default sampling of its library. It samples one trajectory that continues the last 512 training values, 64 points at a time. Each chunk is appended to the context, which is then cut back to the last 512 points.

TimesFM and Timestral. We evaluate median-output configurations of TimesFM 3.0 [7] and Timestral-209M [8], used zero-shot as documented, with pinned model revisions. Both models forecast quantiles, and the benchmark takes their median, a deterministic configuration (§ 10). The context is the whole training series, up to the limit of the model (15 360 and 32 768 points), and the series is the median forecast of the K⋅TK \cdot T steps after train. TimesFM produces it in one pass, without seams. Timestral produces it in blocks of 1 024 steps and feeds the median of each block back as context, so the block boundaries are seams. A median forecast contains no sampling noise. Therefore, these two methods return the same series for every seed and have no between-seed variance; this is taken into account in the tests (§ 7).

zEDGE_TS. zEDGE_TS is a proprietary long-series generator of zypl.ai under development. It works on the latent vectors of the zEDGE pool generator: one sequence model is trained per dataset and shared by the seeds, which change only the sampling. It is evaluated in two sampling modes, which use the same trained model. zedge_ts_long builds the long latent sequence in blocks of 12 latent vectors; each next block is generated by inpainting, conditioned on the last three latent vectors of the previous block. zedge_ts_ar is autoregressive: it adds one latent vector at a time, conditioned on the preceding 11; this is slower, but every step agrees with a longer context. In both modes, every latent vector is decoded into a window, and neighbouring windows, which overlap by T/2T/2 points, are joined by a cosine crossfade; the starts of the decoded windows count as seams.

The calendar is obtained in three ways: the pool generators and zEDGE_TS generate it as part of a window, Diffusion-TS has it imposed, and the pretrained forecasters continue the real calendar of train. Therefore, the metrics of calendar continuity are left out of every ranking and research question. A difference between methods can still come from the calendar as well as from the values; section 8.6 gives a check. The pretrained forecasters are left out on the datasets that leaked into their pretraining data (§ 4.2).

Methods not included

Seam inpainting with CSDI [13], which would fill in the zone around each seam conditioned on both neighbours, needs a separately trained diffusion model and is not included. TimeGAN [1] and other window generators can replace or extend the pool generators without changes to the code (§ 8), but they are not part of the main run. The block bootstraps and surrogates of § 3 resample the real training series and copy it by construction, so they answer a different question. Their selection rules enter the benchmark only on top of a generated pool, through the matched-block and nearest-neighbour assemblers.

The evaluation of the generated series is described in the next section.

Evaluation

In this study, every series is evaluated along three axes: its fidelity to the real data (§ 6.1), its utility for models trained on it (§ 6.2), and the cost of producing it (§ 6.3). Memorisation of the training data is part of fidelity, but it is ranked separately. All series have the same length K⋅TK \cdot T and are evaluated in real units by the same code.

Fidelity

Three real references. Every series is scored against three real references, as shown in Figure 6. Val is the primary reference: it follows train in time, and it has not been used by any method. Train shows how well a method reproduces the data it was fitted on; every method has seen it, and some methods take reference statistics from it. OOT is the latest part of the series and measures fidelity under a shift in time. A method that copies train well but drifts away from the later data scores well against train and worse against val and OOT. Scoring against all three references therefore separates the two cases.

Figure 6. Fidelity evaluation. Every generated series, the real OOT series (real-versus-real reference), and length-matched truncations of the generated series are scored by the same code against three real references: val (primary), train and OOT (the real OOT series against val and train). The metrics are organised in 12 groups (83 metrics); memorisation (F10) is computed once, against train.

Real-versus-real reference. As a real-data reference, the OOT series is evaluated against the other reference splits, val and train, using length-matched comparisons: it is scored as if it were a synthetic series (the pseudo-method real). OOT is shorter than K⋅TK \cdot T, and many metrics depend on the length of the series; therefore, every method is also scored, on its first seed, on its first ∣OOT∣|\mathrm{OOT}| points, and this score is compared with that of the real series at equal length. Since OOT is the latest part of the series, a shift in time can place it farther from val or train than a synthetic series that reproduces them; a method can therefore score better than the real series on a metric.

Metric groups. The 83 metrics are organised in 12 groups. Table 2 lists the groups, and the full list of metrics is given in section C. Groups F1–F4 cover the properties that most evaluations of synthetic time series check:

GroupnnWhat it measuresExamples
F115Marginal distribution of valuesKS, Wasserstein-1, quantile errors, Hill tail index
F29IncrementsKS of Δk\Delta_k, k∈{1,6,24,168}k \in\{1, 6, 24, 168\}; volatility ratio
F310Temporal dependenceACF within and beyond a window, PACF, spectrum, DFA
F45Seasonality and calendarweekly and daily profiles, calendar continuity
F58Seamsjumps at seams, change points at seams, seam detector
F67Distribution of windowsdiscriminative score, C2ST, MMD, feature FID
F78Long-range structurerolling statistics, level crossings, drawdowns, regimes
F86Tail events of task U2.1event share, durations, inter-arrival times
F94Diversity and coverageprecision, recall, density, coverage
F102Memorisation (ranked separately)DCR ratio, share of near copies
F114Class balanceclass-share errors over all classification tasks
F125Outliers and extremesexceedance rates, spikes, weekly block maxima

Table 2. Fidelity metric groups. nn: number of metrics in the group. Every group except F10 is computed against val, train and OOT.

  • the marginal distribution of values (F1);

  • the distribution of increments at horizons from one hour to one week (F2);

  • temporal dependence through the autocorrelation, the spectrum and detrended fluctuation analysis (F3);

  • the daily and weekly profiles (F4).

Group F6 compares the distribution of windows with the standard measures of the literature: a post-hoc recurrent discriminator [1], a classifier two-sample test [17], a predictive score, the MMD [16] for windows of length TT and 2T2T, a Fréchet distance between window features, and the nearest-neighbour distance from synthetic to real windows. Group F9 measures diversity and coverage through precision, recall, density and coverage [20].

Three groups examine properties of the assembled or generated series. Group F5 measures the seams: the distribution of the jumps at the seams against the real increments and against the increments elsewhere in the same series, the local autocorrelation around seams, the share of detected change points that fall on a seam, and how well a classifier distinguishes a snippet around a seam from a snippet elsewhere. For a series without seams these metrics are undefined, except the rate of change points, which compares the series as a whole. The share of change points at seams depends on how dense the seams are: with seams every 64 to 1 024 points, 0.6 to 11 % of the positions lie within three points of a seam, so the share is compared with this chance level in a supplementary check (§ 9.6 and J). Group F7 measures structure longer than a window: the distributions of rolling means and standard deviations over one week and one month, the rates of crossing the real quantile levels, the lengths of excursions above and below them, drawdowns from the rolling weekly maximum, the trend, and the durations of high-volatility regimes of a two-state hidden Markov model. F3 also includes the autocorrelation at lags from TT to 3T3T, beyond the length of a training window.

The remaining groups are related to what the downstream tasks depend on. Group F8 measures the structure of the events of task U2.1, an increase over 24 hours above the 0.90 quantile of train: their share, durations, inter-arrival times, weekly profile, and conditional rates given the recent past. Forecasting such tail events is a common classification problem, and a synthetic series that reproduces the marginal distribution can still get the timing and clustering of the events wrong. Group F11 measures the class balance over every classification task, with relative errors, so that a rare class weighs as much as a frequent one. Group F12 measures outliers and extremes: exceedances of the real 0.5% and 99.5% quantiles, local spikes, and the distributions of weekly maxima and minima.

Memorisation. Group F10 measures how close the synthetic windows are to the training windows. It consists of two metrics: the absolute logarithm of the ratio of the median distance to the closest training window (DCR) of the synthetic windows to that of the real val windows, and the share of synthetic windows that are closer to train than 95% of the val windows (near copies). The first metric is unsigned; the signed ratio, which separates windows closer to train than the val windows from windows farther from it, is reported in section K. This group is computed once, against train, and is ranked separately from fidelity, since a series can be faithful by copying train, and a ranking that mixes the two would reward such a series.

Conventions. Each metric has a fixed orientation (lower or higher is better). Metrics that compare levels, windows or durations are computed after z-scoring both series with the mean and standard deviation of the reference. Deviations of a scale (standard deviations, rates, durations) are reported as ∣log⁡(synthetic/real)∣\lvert\log(\text{synthetic}/\text{real})\rvert, and accuracies of a classifier as ∣AUC−0.5∣\lvert\text{AUC} - 0.5\rvert, so that 0 is ideal for both. Windows for F6, F9 and F10 are cut from the z-scored series with length TT and step T/4T/4. The window features of F6 and F9 are 16 simple statistics of a window, such as moments, quantiles, lag-1 and lag-24 autocorrelation, and the positions of the maximum and the minimum.

Utility: TSTR versus TRTR

Utility measures whether a model trained only on the synthetic series performs on real data as well as the same model trained on real data. The protocol is shown in Figure 7. Each model is trained in two regimes: on the synthetic series of a method (TSTR, train on synthetic, test on real) and on the real training series (TRTR). Both are tested on the same real rows of val and of OOT, and synthetic data never enter a test set. The closer TSTR is to TRTR, the more of the information useful to the model the synthetic series carries. The benchmark measures the quality of the synthetic data, not its value as augmentation; therefore, regimes that mix real and synthetic training data are not part of the benchmark.

Figure 7. Utility protocol. Features and labels are computed separately on each series: real train, the synthetic series, and the real test parts. Thresholds are frozen on real train. Each model is trained either on real train (TRTR) or on the synthetic series only (TSTR) and evaluated on the same real val and OOT rows. Synthetic data never enters a test set.

Tasks. The benchmark contains 14 tasks in nine groups U2.1–U2.9: eight classification tasks and six regression tasks. Figure 8 shows the tasks, and their details are given in section E. The binary tasks ask whether the change over a horizon hh exceeds a quantile of the changes in train:

The 14 downstream tasks and the models trained on eachVector (PDF)
IDtask name(s)kindh (hours)targetmodel outputlinearMLPGBDT
U2.1clf_up_h24_q90clf24Δ24>q0.90\Delta_{24} > q_{0.90}P(y=1)✓✓✓
U2.2clf_up_h{1,6,168}\{1,6,168\}_q90clf1, 6, 168other horizonsP(y=1)✓✓ (max)✓
U2.3clf_up_h24_q{95,99}\{95,99\}clf24rare eventsP(y=1)✓✓ (max)✓
U2.4clf_down_h24_q10clf24Δ24<q0.10\Delta_{24} < q_{0.10}P(y=1)✓✓ (max)✓
U2.5mc3_h24mc24down / neutral / up3 probs✓✓ (max)✓
U2.6reg_h{1,24,168}\{1,24,168\}reg1, 24, 168value xt+hx_{t+h}scalar✓✓ (max)✓
U2.7qreg_h24qreg24quantiles 0.1, 0.5, 0.93 values✓✓ (max)✓
U2.8ms_h24ms1...24path xt+1:t+24x_{t+1:t+24}24 values✓✓ (max)✓
U2.9vol_h24vol24std of next 24 changesscalar✓✓ (max)✓
14 tasks in total; all share one feature set (11 features). Labels are computed on each series with thresholds frozen on real train. MLP runs on U2.1 only in profiles min/mid and on all tasks in max. Every (task, model) is fitted in regime S (TSTR) and R (TRTR) and evaluated on real val and OOT.

Figure 8. The 14 downstream tasks (U2.1–U2.9) and the models trained on each. All tasks use the same 11 features. Labels are functions of the series, with thresholds frozen on real train.

  • the upper tail at h∈{1,6,24,168}h \in\{1, 6, 24, 168\} with q=0.90q = 0.90;

  • rare events with q∈{0.95,0.99}q \in\{0.95, 0.99\};

  • the lower tail with q=0.10q = 0.10.

A three-class task combines the two tails. The regression tasks forecast the value at h∈{1,24,168}h \in\{1, 24, 168\}, the quantiles 0.1, 0.5 and 0.9 at h=24h = 24, the whole next day as a vector, and the volatility of the next day. All thresholds are fixed on real train before any run (§ 4.3).

Features and labels. Every task uses the same 11 features: the value, its differences at lags 1, 2, 3, 6, 12, 24, 48 and 168, and the hour of the week on a circle. Features and labels are computed on each series separately, so no lag crosses the boundary between a synthetic and a real series. The labels of a synthetic series are computed from the series itself with the thresholds of real train. Thus, a method that produces too few tail events also produces too few positive labels.

Models. Three model families are trained:

  • linear models (logistic or softmax regression, ridge regression, linear quantile regression);

  • gradient-boosted trees (XGBoost);

  • a small fully connected network.

Their hyperparameters are fixed in advance in a configuration file and are not tuned on the data. Profile max adds a grid search on the tail of each regime’s own training data, so that neither val nor OOT is used for tuning. To limit the cost, profiles min and mid run the network on task U2.1 only. Classifiers are trained with balanced class weights, and their probabilities are mapped back to the class shares of their own training data. This makes it possible for the Brier score, the log-loss and the calibration error to measure calibration.

Metrics. Classification is scored by ROC AUC, PR AUC, the Brier score, the log-loss, the expected calibration error, F1 and lift; regression is scored by MAE, RMSE, MASE, sMAPE and the error on extreme targets; quantile regression is scored by the pinball loss, the coverage of the 10–90% interval and the rate of crossing quantiles. MASE, the pinball loss and the error on extreme targets are divided by the error of the seasonal naive forecast (lag 168) on real train. Each task type has one primary metric for the rankings: AUC for binary classification, log-loss for the three-class task, MASE for regression, volatility and the multi-step forecast, and the pinball loss for quantiles. The transfer ratio relates TSTR to TRTR on the same test rows; a value of 1 means that the synthetic series is as useful as the real one.

Cost

The cost of a series is recorded per (dataset, method, seed). It includes the wall-clock and CPU time of training the generator, sampling and assembly, the peak GPU memory, and the increase of resident memory. When one generator model is shared by several seeds, its training time is charged to every seed, so that a method whose model is trained once is not favoured. For the pretrained forecasters, model loading counts as training time. Every record stores the hardware and library versions, and the comparison of costs is valid only within one type of hardware; all costs in this paper come from the main run on one NVIDIA H200 (§ 9.1).

These measurements are the input of the statistical analysis described in § 7.

Statistical analysis and research questions

The statistical analysis is carried out on the benchmark tables without generating anything; its protocol and the date at which it was frozen are given in § 7.6. The pipeline of the analysis is shown in Figure 9. It consists of three parts:

Figure 9. From benchmark tables to conclusions. The long-format tables feed three analyses applied to every method: a ranking of every method in every slice, paired comparisons A vs B measured in units of an empirical noise level, and associations across methods. A pre-specified confirmatory family (C1–C4) is corrected jointly.

  • rankings of every method (§ 7.2);

  • research questions about families of methods and about the benchmark itself (§ 7.3);

  • a small pre-specified confirmatory family that carries the main conclusion (§ 7.4).

All parts use the same rules for pairing, noise and verdicts (§ 7.1). Every method is treated in the same way: the family of a method (assembly or long-series generator) is read from its description, so a method added to the benchmark enters the rankings and the questions without changes to the analysis.

Figure 10 shows the research questions and their relation to the confirmatory family.

Pairing, noise and verdicts

Units of replication. Within a dataset, the replicates are the 10 seeds. A seed fixes the pool, the random paths and the training of the generators and the models, so methods with the same seed form a pair. Across datasets, the unit is the dataset: a global conclusion is drawn from the means over seeds of the nine datasets, since the seeds of one dataset are not independent. In profile min, the VAE and Diffusion-TS are trained once per dataset; therefore, a per-dataset conclusion is conditional on the trained model, while the global conclusion generalises over datasets.

Direction. A comparison of AA with BB is carried out on one unit of analysis: a fidelity metric against one reference, or a task ×\times model ×\times split of TSTR. The paired difference is calculated as d=o⋅(A−B)d = o \cdot(A - B), where the orientation oo is +1+1 if higher is better and −1-1 otherwise. That is, d>0d > 0 always means that AA is better.

Noise as the smallest effect of interest. A difference is counted only if it exceeds the noise, that is, how much the score of a method changes for reasons that have nothing to do with the method. For each dataset and unit, the noise of comparisons on one pool is calculated as the median over seeds of ∣vae_rand(k)−vae_rand(k+1)∣|\mathrm{vae\_rand}(k) - \mathrm{vae\_rand}(k+1)|, the difference between random assemblies of neighbouring seeds. This difference contains the sampling variance of the pool. The noise is therefore large, which makes superiority harder to establish and the equivalence region wider; equivalence is defined relative to it, that is, to the stochastic variability of the benchmark, not to a tolerance of an application. For a comparison with an independently trained method, such as a long-series generator, the noise is the same between-seed spread of whichever of the two methods has the larger one. TimesFM and Timestral have no spread of their own (section 4.3), so in their comparisons the margin is set by the spread of the method they are compared with. The noise is fixed before any comparison and serves two purposes [49]: it is the threshold of practical significance and the margin of equivalence. Global effects are expressed in noise units, z=d/noisez = d/\mathrm{noise}, which makes metrics and datasets of different scales comparable.

idquestionevidenceanalysis /compared withnote
RQ0benchmark validity: noise, real-versus-real reference, which metrics discriminate, redundancyall F (3 references), TSTRrand vs rand_b (mid, max); real; Friedman
RQ1ranking of every method in every sliceall leaves: F × reference, U × split, costall methodsleaderboard
RQ2assembly from a window pool vs generators of long seriesF (val, OOT), TSTR all taskslong gen. vs rand, assembly median
RQ3inside assembly: window selection × seam processingF (val, OOT), TSTR all tasksvs random path; seam-processing contrasts
RQ4does fidelity predict utility (which metrics, which reference)F (3 references) ↔ transfer ratioSpearman across methods
RQ5gap to real data; stability over time and model class; decision transferTSTR / TRTR, U6 real (TRTR)Kendall τ(c) max only
RQ6speed and cost: training, generation, memory; quality–time Paretotime, memoryall methods
RQ7moderators: dataset properties and family differencesdataset descriptorseffects of RQ2, RQ3explor.
RQ8class balance: shares of every classification task, rare eventsF8.1, F11; TSTR ECEvs random path; long gen. vs assembly median
RQ9extremes: tails, outliers, block maxima / minimaF1 tails, F2.4, F7.4, F12; extreme error, rare PR AUCvs random path; long gen. vs assembly median
C1–C4confirmatory: methods differ; long vs assembly; selection; seam processingU2.1 AUC, U2.6 MASE (TSTR, linear, OOT)Friedman; assembly median; contrastsconfirm.
Evidence colour: utility TSTR/TRTR (green), cost and descriptors (grey), fidelity and utility together (white). F = fidelity metric groups (Fig. 12); A, B, C = method groups of Fig. 8.
Confirmatory family: joint Holm over C1–C4 in both cells. Other questions: BH within a question; RQ7 exploratory.

Table 10. The research questions of the benchmark. RQ0 validates the benchmark itself, RQ1 ranks every method in every slice, RQ2–RQ9 compare families of methods and properties of the synthetic data. C1–C4 (red) form the pre-specified confirmatory family.

Verdicts and tests. The following tests are used.

  • Per dataset, two one-sided Wilcoxon signed-rank tests [50] over seeds, each at level α/2\alpha/2, test the hypotheses “AA is better” and “AA is worse”; TOST [51] with two one-sided Wilcoxon tests is used for equivalence within ±\pmnoise; the effect is the Hodges–Lehmann estimate.

  • Globally, the tests are applied to the dataset means: Wilcoxon, the sign test and the tt-test, a two-level bootstrap (datasets, then seeds) for the confidence interval, and TOST within ±1\pm1 noise unit. The conclusion is robust when the tests agree.

The verdict is better or worse if the one-sided test is significant after correction and the effect exceeds the noise (per dataset, the Hodges–Lehmann estimate; globally, the median z>1z > 1). The verdict is same if equivalence is shown by TOST, and inconclusive otherwise. “Same” means demonstrated equivalence, not the absence of a significant difference. If the noise of a unit on a dataset is zero, the margin is undefined: in this case the per-dataset verdict is inconclusive and the dataset is left out of the global test.

Intervals. The interval of a global effect is a two-level percentile bootstrap with 2 000 replicates: datasets are drawn with replacement, and then seeds within each drawn dataset; the paired difference dd of two methods is formed on the same dataset, seed and unit before resampling, so the pairing of methods and seeds is kept. The interval of a mean rank uses the same scheme with 1 000 replicates, the ranks being formed within each dataset and seed. A method without sampling, such as TimesFM or Timestral, contributes no variation of its own, and in profile min the seeds of the VAE and of Diffusion-TS vary the sampling, not the training (§ 10). The test rows are fixed, so the uncertainty within a series is not resampled. The intervals are not adjusted for multiplicity.

Missing utility cells. A synthetic series may be unable to train a task. For example, a smooth median forecast may never move above the threshold, so its labels have one class. Leaving such a cell out would rank the method only on the datasets where it succeeded. Instead, the cell enters as a model that learned nothing: each metric takes the worse of two values, the chance level and the worst value of any method on the same dataset, seed, task, model and split. The value of an imputed cell therefore depends on the other methods of the block, so a poor method can lower it. The chance levels of every metric are listed in Table 13; a metric without one takes the worst value of its block. A method excluded for leakage (§ 4.2) is not imputed; it is absent on that dataset.

MetricChance level
ROC AUC (binary; macro for three classes)0.5
PR AUCpp
Log-lossthe entropy −pln⁡p−(1−p)ln⁡(1−p)-p \ln p - (1-p) \ln(1-p) for a binary task; ln⁡K\ln K for KK classes
Brier scorep(1−p)p(1-p)
F1, ECE, lift at 10 %, macro F1none: the worst value of the block
Losses of regression, quantile, volatility and multi-step tasksnone: the worst value of the block (not needed in the published runs, in which no such cell failed)

Table 13. Chance levels of the utility metrics. pp: the positive share of the test rows; KK: the number of classes. A metric without a chance level takes the worst value of its block.

Multiplicity. Corrections for multiple testing are applied at three levels:

  • the confirmatory family C1–C4 (§ 7.4) is corrected jointly by Holm [52], and only this family gives the main conclusion;

  • secondary questions (RQ0, RQ2–RQ6, RQ8, RQ9) are corrected by Benjamini–Hochberg [53] within each question, both for the global pp-values and for the per-dataset verdicts of each question and dataset; they give a map of where a family of methods is better, worse or the same, with the false discovery rate controlled;

  • exploratory analyses (RQ7) report pp-values, and their results are stated as hypotheses.

With 10 seeds, the smallest one-sided Wilcoxon pp-value is about 0.001, so a per-dataset verdict is possible. With nine datasets it is about 0.002, so a global verdict after correction requires a large and consistent effect. Thus, the main lever of power is the number of datasets.

Rankings

RQ1 asks how the methods rank and with what uncertainty. The seam-processing variants are not ranked, since in profile min they exist only on the path of zseries; ranking them would let one assembler enter four times and every other assembler once. Image quilting selects windows and cuts the seams in one method and is always ranked. Methods left out of the rankings take part in every other question.

From leaves to rankings. A leaf is one fidelity metric against one reference, or one task × model × metric × split of TSTR. Within a leaf, the methods are ranked separately for each (dataset, seed), since methods of one seed share the pool and the test rows; ties receive the average rank. The ranks are then averaged along a fixed hierarchy with equal weights at every level, so that a group with many metrics does not outweigh a group with few:

  • leaves form groups;

  • groups form criteria (Table 3), and each criterion is averaged over its references or test splits;

  • criteria form the fidelity and utility domains.

DomainCriterionContent
FidelitydistributionF1 values, F2 increments, F12 outliers and extremes
dependenceF3 temporal dependence, F4 seasonality
windowsF6 distribution of windows, F9 diversity and coverage
long structureF7
targetF8 tail events of U2.1, F11 class balance
seamsF5
UtilityclassificationU2.1–U2.5, TSTR on val and OOT
regressionU2.6–U2.9, TSTR on val and OOT
SeparatememorisationF10, against train
costtotal time, training, generation, memory

Table 3. Ranking criteria. Fidelity criteria are averaged over their references, utility criteria over the two test splits. The calendar-continuity metrics F4.4–F4.5 are left out, since the pretrained forecasters take the real calendar (section 4.3).

A ranking is the mean of the two domains. Memorisation (F10) and cost are ranked separately and do not enter the rankings of fidelity and utility. Table 3 lists the criteria and the metric groups that form them. A metric that is undefined for a method (for example, the seam metrics of a series without seams) is not imputed: the method is not ranked in that leaf, and a criterion without any defined leaf is left out of the method’s domain, whose remaining criteria are averaged. In the overall leaderboard this concerns the seam criterion of TimesFM and Timestral; section J gives the leaderboard without the seam criterion for every method.

The overall leaderboard and four rankings. The headline ranking is the overall leaderboard. Its fidelity domain uses a core set of one or two canonical metrics per criterion, each averaged over train, val and OOT: for example, the Wasserstein distance of values, the power spectrum, the discriminative score and the MMD of windows, the class shares, and the detectability of seams. Its utility domain averages classification and regression on both test splits. The core set was fixed by what the metrics measure, not by how well they separated the methods: choosing the discriminative metrics on the same data as the ranking would select noise that happens to favour some method. Within a leaf, the ranks are noise-aware: the rank of a method is one plus the number of methods that beat it by more than the noise of the leaf on that dataset, plus half the number of other methods within the noise, so near-identical methods share a place instead of pushing the others apart. The margin is the same for every pair of methods in a leaf, no groups are formed, and the ranks of a block sum to k(k+1)/2k(k+1)/2 for kk methods; when a method is missing on a dataset because of leakage, the ranks of the others are rescaled to 1,…,k1,\ldots,k. Closeness within the noise is not transitive, so a method can tie with two others that do not tie with each other (§ 25). Four further rankings combine fidelity against val and OOT, or against train, with utility in classification or in regression, over every fidelity metric rather than the core set. These rankings show whether the order depends on the reference and on the task type.

Uncertainty and robustness. A two-level bootstrap of the mean ranks (datasets, then seeds) gives an interval for each rank and the probability of being first and in the top three. The Friedman test [54] with datasets as blocks tests whether the methods differ at all. The top tier consists of the best method and every method that is not significantly worse than it by a one-sided Wilcoxon test over datasets with Holm correction [55]. The top tier is thus a non-rejection set, not an equivalence class: with limited power, a method can remain in it without being close to the leader. The comparisons are made with the observed leader and are not adjusted for its selection (section M). The Nemenyi critical difference [56] is reported for reference only, since it grows with the number of methods. The robustness of the leaderboard is checked by variants fixed in advance, each of which is compared with the leaderboard by Kendall’s τ\tau:

  • ordinary average ranks;

  • fidelity against train only, and against val and OOT only;

  • every fidelity metric instead of the core set;

  • only the datasets where every method is ranked;

  • a field of one method per generator, so that many variants of one generator cannot move the others;

  • leaving out one dataset at a time;

  • without the seam detector and the discriminative score against val, the two metrics whose separation of real data from a random assembly could not be checked;

  • 2 000 random weights of the criteria (Dirichlet(1) within each domain).

Next to the rank. Ranks show which method is better, but not by how much. Therefore, each core metric is also reported on a normalised scale on which the random assembly is 0 (of the method’s own pool; of the VAE pool for a long-series generator) and the real data are 1; a value above 1 means closer to the reference than the real data. Cost is reported next to each ranking as a Pareto front of rank against median time and as the same ranking with cost as a third domain. Memorisation does not change the rank but is flagged: a method is flagged when its share of windows very close to train is at least twice that of the real OOT series, significantly over datasets.

Research questions

RQ0: validity of the benchmark. This question asks whether the benchmark measures what it should. It consists of four checks:

  • (a) two random assemblies of one pool should be equivalent on every metric; this check needs the second random path and is carried out in profiles mid and max only;

  • (b) the real OOT series, scored as a method, is compared with the methods on each fidelity metric at equal length (§ 6.1); a metric on which the real OOT reference does not outperform most synthetic methods is flagged for inspection; the cause can be a shift between OOT and the reference, sampling variability or the sensitivity of the metric;

  • (c) a metric is discriminative if the Friedman test across methods is significant after correction and the spread of the methods exceeds the noise. The check covers 297 units: 79 fidelity metrics against each of the three references (the 83 metrics less the two calendar-continuity metrics, which are left out of every question, and the two memorisation metrics), the two memorisation metrics once, against train, and 58 utility units (each of the 14 tasks with two downstream models on val and OOT, and task U2.1 with the third model as well);

  • (d) pairs of metrics whose rank correlation across methods is at least 0.9 in absolute value duplicate each other, and the agreement of a metric against different references shows how much the conclusion depends on the reference.

RQ2: assembly versus long-series generation. Each long-series generator is compared with the random path of the VAE pool and with the assembly median, the median over all assembly methods of the pool for each dataset, seed and unit. The median is used rather than the best assembler, because the best one would be selected on the same data on which it is compared.

RQ3: within assembly. This question asks whether the selection and order of the windows and the processing of the seams move the series towards the real data, each against its own control, when a series is assembled from one pool. Each assembler is compared with the random path, and each seam-processing variant with the same path without processing. The main effect of each processing is averaged over the paths that have every processing.

RQ4: does fidelity predict utility? For each fidelity metric and reference and each utility criterion, the Spearman correlation across methods within a dataset between the metric and the transfer ratio is calculated and tested over datasets. A metric is called predictive, the term of the protocol, when this correlation is positive over the datasets after correction. The correlation is an association, not a causal statement, across the methods of the benchmark, several of which share a pool or a generator; whether a metric selects a generator outside the benchmark is not tested.

RQ5: gap to real data and stability. This question consists of three parts:

  • (a) the transfer ratio of each method and criterion is tested against 1, the value at which the synthetic series trains the models as well as the real one. For ROC AUC, transfer is measured as skill above chance: (AUCTSTR−0.5)/(AUCTRTR−0.5)(\mathrm{AUC}_{\mathrm{TSTR}} - 0.5)/(\mathrm{AUC}_{\mathrm{TRTR}} - 0.5), and is undefined when AUCTRTR≤0.52\mathrm{AUC}_{\mathrm{TRTR}} \le0.52. For lower-is-better losses, transfer is lossTRTR/lossTSTR\mathrm{loss}_{\mathrm{TRTR}}/\mathrm{loss}_{\mathrm{TSTR}}; for other higher-is-better scores it is scoreTSTR/scoreTRTR\mathrm{score}_{\mathrm{TSTR}}/\mathrm{score}_{\mathrm{TRTR}}. Absolute scores are reported in Table 15.

  • (b) Kendall’s τ\tau between the rankings on val and on OOT, and between model classes, shows whether the order of the methods survives a shift in time and a change of the model;

  • (c) in profile max only, a grid of model configurations is ranked on real and on synthetic data, to test whether a choice made on synthetic data transfers to real data.

TaskModelSplitTRTRTSTR, medianTSTR, bestBest method
U2.1U_{2.1} (AUC)linearval0.7470.6640.749zedge_pctsp*
linearOOT0.7430.6570.742zedge_ts_ar*
gbdtval0.7390.6440.703diffusion_ts
gbdtOOT0.7640.6510.758zedge_ts_ar*
mlpval0.7570.6520.727zedge_quilting*
mlpOOT0.7890.6720.757zedge_quilting*
U2.6U_{2.6} (MASE)linearval0.5810.9570.573timestral
linearOOT0.8320.8880.841zedge_quilting*
gbdtval0.7931.3870.810timesfm
gbdtOOT0.7961.1010.796timesfm
MethodSeamsNear a seam, %F5.6Chance levelRatio
timestral40.60.0000.0050.00
vae_zseries*294.00.0220.0400.57
zedge_zseries*294.00.0240.0400.60
vae_pctsp294.00.0270.0400.67
vae_atsp294.00.0280.0400.67
zedge_ts_ar*588.10.0290.0810.38
vae_greedy294.00.0290.0400.72
diffusion_ts588.10.0360.0800.45
vae_quilting324.40.0410.0450.93
zedge_atsp*294.00.0450.0401.15
zedge_pctsp*294.00.0450.0401.15
zedge_quilting*324.40.0450.0441.02
vae_knn294.00.0490.0391.18
vae_zseries_quilt*324.40.0510.0441.16
vae_zseries_xfade*324.40.0530.0441.13
zedge_zseries_quilt*324.40.0550.0441.27
zedge_ts_long*588.10.0550.0790.69
zedge_zseries_xfade*324.40.0590.0441.34
zedge_greedy*294.00.0620.0391.56
vae_zseries_wsola*334.60.0810.0461.75
zedge_knn*294.00.0810.0392.00
zedge_zseries_wsola*334.60.0860.0461.87
chronos7810.80.0910.1090.83
vae_mbb294.00.0950.0402.49
vae_rand294.00.1000.0392.51
vae_herding294.00.1050.0392.65
zedge_mbb*294.00.1400.0393.15
zedge_herding*294.00.1420.0393.67
zedge_rand*294.00.1510.0393.89
timesfm00.0–––

Table 15. Absolute scores of the confirmatory tasks, main run: TRTR, the median TSTR over the 30 methods and the best TSTR with its method. Supplementary. * proprietary (zypl.ai).

RQ6: cost. The following characteristics are reported: the median time of each method over datasets, the time relative to the fastest method, the separate costs of training and of generating a new series, the peak memory, and the slope of log time on log KK, which shows how the cost grows with the length of the series. The slope needs different KK and is not available in profile min, where K=30K = 30 for every dataset. Generation is reported separately from training, because a generator is trained once and sampled many times.

RQ7: moderators (exploratory). The dataset effects of RQ2 and RQ3 are correlated with descriptors of the datasets: autocorrelation at 24 and 168 hours, the kurtosis of the increments, volatility clustering, trend and the share of missing values. With nine datasets, the results of this question are stated as hypotheses only.

RQ8 and RQ9: class balance, outliers and extremes. Two thematic slices collect metrics from different groups:

  • class balance: the class shares of every task and the calibration of the TSTR classifiers;

  • outliers and extremes: tail quantiles, the tail index, large jumps, the metrics of F12, the regression error on extreme targets and the PR AUC of rare events.

Inside the fidelity aggregates, these metrics are averaged with many others. Therefore, each slice is ranked on its own, the families of methods are compared on it as in RQ2 and RQ3, and the fidelity metrics of the slice are correlated with the corresponding utility.

Confirmatory family

The main conclusion of the paper rests on ten tests in two cells fixed in advance: a classification cell, task U2.1 with ROC AUC, and a regression cell, task U2.6 at h=24h = 24 with MASE. Both cells are TSTR with the linear model, tested on OOT, the split furthest from train. Each cell contains the following comparisons:

  • C1 asks whether the methods differ at all: an omnibus Friedman test over the ranked methods with datasets as blocks. The design is complete: a method excluded for leakage on any dataset drops out of the test, and every dataset stays. In both runs these are Chronos and TimesFM, so C1 compares the other 22 ranked methods of the main run, those of the zEDGE pool and zEDGE_TS included, and the other 11 of the replication run, on all nine datasets (section H);

  • C2 compares Diffusion-TS and Chronos with the assembly median, as in RQ2; the two generators are fixed in advance, and other long-series generators are compared in RQ2 but do not enter the family;

  • C3 is the main effect of window selection: the mean of every ranked assembler against the random path;

  • C4 is the main effect of seam processing: every processing against no processing, on the paths that have every processing (in profile min, the path of zseries). In this profile the contrast is therefore the effect of seam processing on the zseries path of the VAE pool.

The ten two-sided pp-values (2 + 4 + 2 + 2) are corrected together by Holm. A test that cannot be computed enters with p=1p = 1, so it cannot weaken the correction. C1 gives the verdict differ or inconclusive. C2–C4 give better or worse if pHolm<αp_{\mathrm{Holm}} < \alpha and the median effect exceeds one noise unit, and same if TOST is significant after Holm. This joint rule of statistical and practical significance is referred to below as the pre-specified rule.

Several pool generators

The contrasts within assembly (RQ3, C3, C4) and the assembly median of RQ2 and C2 are computed on one pool, the VAE pool, whose random path gives the noise level. The methods of the zEDGE pool enter the rankings and all other questions in the same way as any other method. As an exploratory check that was not pre-specified, the same analysis is repeated with the zEDGE pool in place of the VAE pool (§ 9.5).

Protocol and amendments

The analysis protocol (docs/HYPOTHESES.md and the code of the analysis) was frozen at commit 7fea7530 on 21 October 2026, before the main run was carried out (24 to 26 October 2026); the replication run was completed before the freeze, on the same day. The analysis is therefore pre-specified with respect to the main run. The protocol fixes the core metrics, the contrasts of the research questions, the families of multiple comparisons, the exclusion rules and the algorithms for noise, ties and tiers; Table 12 lists them with their place in the protocol. Changes made after the freeze are listed in docs/PROTOCOL_AMENDMENTS.md and are exploratory: the reading of the leaderboard by tier rather than by place, which was adopted after the two runs were compared; a correction to the rank bootstrap, in which tied methods now share a place; the wording of RQ3; and the supplementary analyses of § 9, which are collected in § 9.6 or marked supplementary where they are reported.

ComponentWhat is fixedProtocol
Core metricsone or two canonical fidelity metrics per criterion for the core rankings; the primary metric of every utility taskRQ1, core rankings
Confirmatory familycells U2.1 and U2.6 at h=24h = 24, linear model, OOT, primary metric; tests C1–C4; the long-series generators of C2 (Diffusion-TS, Chronos)3.9
ContrastsRQ2: each long-series generator against the random path and the assembly median of one pool; RQ3: each assembler against the random path, each joiner against the same path without seam processing, and the main effectsRQ2, RQ3
MultiplicityC1–C4: Holm over the ten tests; secondary questions (RQ0, RQ2–RQ6, RQ8, RQ9): Benjamini–Hochberg within a question; RQ7: exploratory, no conclusions2, multiple comparisons
Noise and verdictsnoise: the median difference between seeds of the random path, or the larger between-seed spread for independently trained methods; “better” or “worse”: significant after correction and a median effect of more than one noise unit; “same”: TOST with the margin of one noise unit2, noise; 2, verdict
Exclusion rulespairs of a pretrained model and a dataset in its training data; seam-processing variants not ranked; calendar-continuity metrics F4.4–F4.5 left out; a missing utility cell enters at the chance level or the worst value of its block2; RQ1, ranked methods
Ties and tiersties within the noise on the overall leaderboard; the top tier: the best method and every method not worse than it by a one-sided Wilcoxon test with Holm correctionRQ1

Table 12. Components fixed by the analysis protocol (frozen on 21 October 2026, before the main run).

Section 7 describes the software that implements the benchmark and this analysis.

Software and reproducibility

MarathonTS is released as a Python package with the code of every open method, the configuration of the runs and the published tables. This section describes how a run is organised and how the benchmark is extended; the architecture is described in more detail in section N.

Stages. A run consists of a sequence of stages, shown in Figure 11:

Figure 11. The benchmark pipeline. Stages run in order. Each one writes its outputs (right) and can run in its own software environment (left). Task thresholds are fixed once on the real training split, committed to the repository, and only read at run time (dashed).

  • raw converts the source files to Parquet;

  • prepare builds the hourly series and the splits;

  • pools trains the pool generators and samples the pools;

  • generate produces the series of every method;

  • fidelity and utility evaluate them;

  • collect merges the results into the published tables, one row per (dataset, method, seed) for fidelity and cost and one row per task, model, regime and split for utility.

The analysis of section 6 is a separate package that only reads these tables. It runs on a CPU and writes the rankings, the answers to the research questions and the confirmatory family as Parquet tables and Markdown reports. Every stage writes one file per unit of work (Figure 15). This allows the work to be split across machines by dataset and seed, and an error in one unit to be logged without stopping the run.

Figure 15. Layout of the benchmark outputs. Every unit of work writes its own Parquet file. The collect stage merges them into the published tables in tables/.

Environments. The pool generator and the long-series generators have conflicting dependencies; therefore, each of them runs in its own environment with pinned requirements (Figure 13). The VAE runs in a TensorFlow environment without PyTorch; Diffusion-TS, Chronos, TimesFM and Timestral each run in a PyTorch environment; assembly, evaluation and the analysis run in a common core environment. The environments exchange data only through the Parquet files of the pools and the series. A Docker image and Colab notebooks reproduce the environments.

Figure 13. Isolation of conflicting dependencies. The VAE pool generator runs in a TensorFlow environment without PyTorch. Diffusion-TS, Chronos, TimesFM and Timestral each run in their own PyTorch environment, and assembly, evaluation and the statistics in a common core environment that never imports TensorFlow. The only link between the environments is the Parquet files they write.

Profiles. Three run profiles fix everything that trades accuracy for compute; they are shown in fig. 16.

  • Profile min, which is used for the main run, takes the last 10 000 hours of each dataset, trains one VAE and one Diffusion-TS model per dataset, uses light configurations of the downstream models and can run on a single GPU, for example in Colab.

  • Profile mid retrains the VAE on every seed and adds the second random path.

  • Profile max uses the full series, retrains the generators on every seed, applies every seam joiner to every assembler except the random path and image quilting, and tunes the downstream models on the tail of each regime’s own training data.

Each profile writes to its own folders, and its task thresholds are computed on its own training split.

Incremental recomputation. Every unit of work stores a fingerprint: a hash of its data settings, the settings of its components, the configuration files it depends on, and the fingerprints of the units it is made from (Figure 14). A unit is skipped only if its outputs exist and its fingerprint matches. Therefore, a changed setting recomputes exactly the units downstream of it. For example, changing the settings of one assembler recomputes its series and their evaluation and keeps the pools, the other methods, the labels and the TRTR models. Fidelity and utility are fingerprinted per metric group and per task and model, so an added metric group, task or model computes only its own cells. A long run can be stopped and resumed, and the cached results always correspond to the current configuration.

Figure 14. Dependency graph of unit fingerprints. Each unit of work stores a hash of its inputs, its own settings, the relevant configuration files, and the fingerprints of the units it depends on. When a setting changes, only the units downstream of it are recomputed.

Data integrity. The sources of the datasets are downloaded from their original locations and may change after a run; for example, Bitcoin is re-uploaded daily. The repository stores a hash of the content of every prepared series (for ETT, of the series of the replication run; § 9.7), and every stage that loads a prepared series compares it against this hash. A series that differs gives a warning that the results are not comparable with the published ones. The task thresholds are stored in the repository and are only read during a run (§ 4.3).

Extension. Every component is registered by name: datasets, tasks, pool generators, assemblers, seam joiners, long-series generators, downstream models, fidelity metric groups, utility metrics, statistical tests and hypotheses (Figure 17). A new component is a class or function in a separate plugin file plus one line of configuration, and no benchmark code changes. A new pool generator receives the whole set of assemblers and joiners, a new assembler runs on every pool, and a new method enters the rankings and the research questions automatically, since its family is read from its description. The results of methods whose code is not released, such as the proprietary ones of this paper, are kept in the published tables and take part in the analysis, so that any ranking can be recomputed with or without them.

Figure 17. Extension points. Every part of the benchmark is registered by name. A plugin file registers new components, and the configuration selects them. A new pool generator, assembler, seam joiner or long-series generator becomes part of the comparison and of the relevant hypotheses without changes to the benchmark code.

Release and licences. For the reproducibility of the study, the code is publicly available under the Apache License 2.0 at https://github.com/zypl-ai/MarathonTS (release tag v1.0.0), together with the published tables and analysis outputs of both runs, the configuration of the main run, the task thresholds, the data hashes and the commands that regenerate the tables and figures. TimesFM 3.0 and Timestral-209M are loaded at pinned revisions; Chronos is loaded from amazon/chronos-t5-small, whose revision (a971ba2) dates from November 2025, before both runs. The unit files of both runs (series, pools, the weights of the open generators, fidelity, utility metrics and predictions, labels) are on Hugging Face under the same tag. Not released are the code of xseries, zEDGE and zEDGE_TS and the weights of zEDGE and zEDGE_TS, the artifacts derived from ETTh1 and PJM (section A), and the weights of the downstream models. The rankings of both runs (§ 9.1) can be explored in an interactive leaderboard at https://huggingface.co/spaces/zyplai/MarathonTS, which reads the same published tables and opens on the main run. The datasets keep their own licences (section A), and third-party components keep theirs: the weights of TimesFM 3.0 are under a non-commercial licence, and the LKH solver used by the ATSP assembler is free for academic and non-commercial use.

The results of the main run are presented in the next section.

Results

Runs

The benchmark was run twice in profile min, with the same configuration, datasets, task thresholds and seeds, on different hardware and software stacks (Table 21). The settings of every generation unit are the same in both runs: the fingerprints of all 1 440 common series match. The code commits differ in naming and in the handling of results published without code. Eight datasets were identical, whereas the prepared ETT series differed at the value level (§ 9.7). The main run was carried out on one NVIDIA H2002 and covers all 30 methods of § 5. Every result, ranking and cost reported in this paper comes from the main run, unless the replication run is named. The replication run was carried out on one NVIDIA L4 and covers the methods of the VAE pool and the four open long-series generators; it serves only to check how well the results reproduce on another hardware and software stack (§ 9.7). The two runs are never mixed: every ranking is computed within one run, and times measured on different hardware are not compared.

Main runReplication run
GPU1× NVIDIA H200 (143 GB)1× NVIDIA L4 (24 GB)
CPU cores19212
SystemUbuntu 22.04.5, driver 580.173.02Google Colab (Linux 6.6)
SoftwarePython 3.12.13, PyTorch 2.9.1, TensorFlow 2.20.0 (environment manifests published)pinned requirements of the repository
Codeinternal commit 365232d6 (23 October 2026)completed before the freeze; results in commit 7fea7530 (21 October 2026)
Methods30 (24 ranked)16 of the VAE pool and the open generators (13 ranked)
Roleevery result of this paper unless statedreproducibility check (§ 9.7)

Table 21. The two runs of profile min.

Overall leaderboard

Table 4 gives the overall leaderboard of the main run in full: all 24 ranked methods. The six seam-processing variants are not ranked, because not every assembler has them (section 4.2).

PlaceMethodFamilyMean rank95 % intervalP(first)Top tierDatasets
1zedge_quilting*assembly8.27[7.35, 9.20]0.85yes9
2zedge_zseries*assembly8.79[8.28, 9.47]0.08yes9
3zedge_greedy*assembly8.95[7.97, 10.00]0.01yes9
4zedge_pctsp*assembly9.18[8.13, 10.25]0.00yes9
5zedge_knn*assembly9.33[8.32, 10.48]0.00yes9
6zedge_atsp*assembly9.39[7.83, 11.10]0.03yes9
7zedge_ts_ar*long-series9.40[7.99, 10.91]0.02yes9
8zedge_ts_long*long-series9.50[8.25, 10.89]0.00yes9
9zedge_herding*assembly10.48[9.15, 11.88]0.00yes9
10zedge_mbb*assembly10.58[9.10, 12.25]0.00yes9
11zedge_rand*assembly (random)10.75[9.12, 12.49]0.00yes9
12vae_herdingassembly13.53[12.79, 14.30]0.00no9
13vae_zseries*assembly13.68[12.15, 15.24]0.00yes9
14diffusion_tslong-series13.78[11.98, 15.39]0.00no9
15timesfmlong-series14.43[12.56, 16.23]0.00yes6
16vae_quiltingassembly14.44[12.64, 16.20]0.00yes9
17vae_pctspassembly15.27[13.80, 16.67]0.00no9
18vae_atspassembly15.28[13.26, 17.10]0.00no9
19vae_greedyassembly15.57[13.90, 17.19]0.00no9
20chronoslong-series15.78[14.04, 17.45]0.00yes7
21vae_knnassembly16.01[14.16, 17.70]0.00no9
22vae_mbbassembly16.30[14.24, 18.20]0.00no9
23vae_randassembly (random)16.47[14.43, 18.39]0.00no9
24timestrallong-series16.77[15.53, 18.08]0.00no9

Table 4. Overall leaderboard of the main run (profile min, NVIDIA H200; 9 datasets, 10 seeds): all 24 ranked methods. Mean rank over the datasets (lower is better), its 95 % two-level bootstrap interval (§ 7.1), the probability of being first, membership of the top tier and the number of datasets on which the method is ranked. * proprietary (zypl.ai), evaluated as a black box.

The table is read by tier. The place of a method is its position by mean rank; the top tier (§ 7.2) consists of the best method and every method that a paired test over the datasets does not find significantly worse than it. The tests do not separate the places within the tier, and the order within it differs between the two runs (§ 9.7). Membership is not monotone in the place: a method that loses to the best on almost every dataset leaves the tier even with a close mean rank, whereas a method whose ranks vary between datasets stays in it. TimesFM and Chronos, left out on three and two datasets for leakage, are compared with the leader on 6 and 7 datasets instead of 9, with less power.

The top tier holds 15 of the 24 methods. The eleven ranked methods of the zEDGE pool and of zEDGE_TS take places 1 to 11 and are all in the tier; zedge_quilting is first, with a probability of being first of 0.85. The best open method is vae_herding (place 12), which is outside the tier, as are Diffusion-TS and Timestral; vae_zseries, vae_quilting, TimesFM and Chronos are inside it. The two domains agree: the fidelity and utility ranks of the methods have Kendall’s τ=0.88\tau= 0.88.

The first place does not depend on the choices made in the leaderboard (§ 7). It is the same under every variant fixed in advance, with Kendall’s τ\tau between 0.92 and 0.98 against the leaderboard, including fidelity against val and OOT only (τ=0.94\tau= 0.94), which removes the training split as a reference. With ordinary average ranks instead of noise-aware ones, the first place is the same, three places within the top tier change, and zedge_rand, the random path of the zEDGE pool, leaves it, so that the tier holds 14 instead of 15 methods (§ 25). The first place is also the same when any one of the nine datasets is left out, and under 2 000 random weights of the criteria it is first in 99.7 % of the draws.

Rankings by slice

The four rankings (RQ1). zedge_quilting is first in all four rankings and in their versions on the core metrics; the Friedman test separates the methods in each of them (p<10−7p < 10^{-7}). The task type changes the order more than the reference: rankings with the same task type and same references agree with τ=0.95\tau= 0.95 (classification) and 0.93 (regression), whereas rankings with the same reference and different task types agree with τ=0.72\tau= 0.72 and 0.75. The pretrained forecasters gain most in regression: TimesFM has a mean rank of 11.4 in the regression ranking against val and OOT and 15.7 in the classification one. The leader of a single criterion is not always the overall leader: for example, zedge_ts_long leads the marginal distribution against OOT, the random path of the zEDGE pool the long structure against OOT, and zedge_ts_ar the utility against val.

Memorisation and cost. No synthetic window of any method is an exact copy of a training window, whereas every window of a control copy of train is one (an exact copy has a DCR below 0.05 in z units; the training windows are cut with step T/4T/4). The median signed log ratio of the DCR lies between −0.31 and 0.28 for every method (the real OOT series: −0.13), against −2.7 to −1.1 for control copies of train with noise of 0.05 to 0.25 standard deviations (section K); TimesFM and Timestral are slightly closer to train than the val windows (intervals below 0). Copies with noise of 0.5 standard deviations and copies of up to half of the series are not distinguished from the methods and the real series. The near-copy flag of the protocol is raised for no method (ratio to the real OOT series at most 1.0, median over datasets; Friedman p=0.92p = 0.92). The share of near copies of the real OOT series is itself 1 on three datasets and 0 on three others, so the flag is raised for none of the controls either, and the reading here uses the signed ratio and the exact copies. These are distance-based diagnostics, not a privacy guarantee; membership and attribute inference attacks are outside the scope (section 9). The cost and the Pareto fronts are described with RQ6 below.

Confirmatory family

Table 5 gives the ten tests of section 6.4. C1 covers every ranked method except Chronos and TimesFM; C2–C4 are computed on the VAE pool (section 6.5). C1 finds that the methods differ in both cells. The verdict holds with a permutation calibration and with the incomplete datasets kept. In the classification cell the difference comes from the methods of the zEDGE pool and zEDGE_TS: among the 11 methods of the replication run the test is not significant in that cell in either run (section H). The other comparisons are inconclusive; the rule requires both a significant test after Holm and a median effect of more than one noise unit:

Table 5: Confirmatory family of the main run: TSTR with the linear model on OOT, task U2.1 (ROC AUC) and task U2.6 at h=24h = 24 (MASE). Effect: the median over datasets of the effect in noise units (positive: the first method is better); intervals: the 95 % bootstrap intervals of the mean effect over datasets (pre-specified) and of the median effect (supplementary), not adjusted for multiplicity; the number of datasets on which it is better or worse, and the Holm-corrected pp-value over the ten tests.

TestComparisonEffectmeanmedianBetterWorsepHolmp_{\mathrm{Holm}}Verdict
Classification, U2.1
C1all methods (Friedman)0.012differ
C2Diffusion-TS vs assembly median1.89[−0.63, 4.85][−0.62, 4.68]520.49inconclusive
C2Chronos vs assembly median0.08[−4.96, 0.95][−2.34, 1.18]221.00inconclusive
C3assemblers vs random path0.06[−0.31, 1.01][−0.60, 1.41]201.00inconclusive
C4seam processing vs none−0.92[−3.47, −0.47][−4.44, −0.17]040.14inconclusive
Regression, U2.6, h=24h = 24
C1all methods (Friedman)1.4⋅10−71.4 \cdot10^{-7}differ
C2Diffusion-TS vs assembly median3.03[0.73, 8.59][−0.29, 13.40]610.27inconclusive
C2Chronos vs assembly median2.47[1.58, 5.53][1.28, 5.77]500.19inconclusive
C3assemblers vs random path0.61[0.02, 1.41][−0.27, 1.55]310.39inconclusive
C4seam processing vs none−0.77[−1.32, −0.27][−1.13, −0.11]020.03inconclusive

Table 5.

  • C2: Diffusion-TS is better than the assembly median of the VAE pool on 5 (classification) and 6 (regression) of nine datasets, with median effects of 1.9 and 3.0 noise units, but the tests are not significant; Chronos has no clear direction in classification and is better on 5 of 7 datasets in regression.

  • C3: the mean of the assemblers is not separated from the random path in either cell.

  • C4: seam processing is worse than no processing on every dataset with a verdict (4 in classification, 2 in regression). In regression the test is significant after Holm (p=0.03p = 0.03), but the median effect, −0.77 noise units, is below one noise unit, so the verdict is inconclusive.

The intervals of Table 5 are not adjusted for multiplicity. The interval of the median effect includes zero for Diffusion-TS and for window selection in the regression cell and excludes it for Chronos in the regression cell and for seam processing in both cells. C3 averages over every assembler, zseries included, and C4 is computed on the path of zseries. In a supplementary analysis that was not pre-specified, the mean of the open assemblers gives the same verdicts of C3 in both runs (in the main run the median effect changes from 0.06 to −0.02 noise units in the classification cell and from 0.61 to 0.48 in the regression cell), and no other verdict of the family changes.

Research questions

With nine datasets, the smallest one-sided pp-value of the Wilcoxon test over datasets is 0.002. After the Benjamini–Hochberg correction over the hundreds or thousands of comparisons of one question, no global comparison of RQ2, RQ8 and RQ9 reaches significance, and RQ3 has only verdicts of equivalence. These questions are therefore read from the direction and size of the effects and from the per-dataset verdicts, which the published tables give for every comparison. A run with more datasets would be needed for global verdicts at this level of correction.

RQ0: validity. Of the 297 units of check (c) of § 7.3 (237 combinations of a fidelity metric and a reference, the two memorisation metrics and 58 utility units), 231 are discriminative: the methods differ beyond the noise. Among the fidelity metrics the share is highest against train (72 of 79), then OOT (59 of 79), and lowest against val (44 of 79); one of the two memorisation metrics and 55 of the 58 utility units are discriminative. On twenty combinations the real OOT series does not score better than most methods, so they are flagged for inspection; they include the discriminator score (F6.1) against val and train and the share of near copies (F10.2), a diagnostic rather than a ranking metric. The reading of a flag is given in § 7.3. Thirteen pairs of metrics duplicate each other (∣ρ∣≥0.9|\rho| \ge0.9), for example the Wasserstein distance and the error of the mean. The order of the methods by a metric depends on the reference: the median rank correlation of a metric between val and OOT is 0.63, and between val and train 0.51.

RQ2: assembly versus long-series generation. Descriptively, the long-series generators split into two groups. zEDGE_TS is better than the assembly median of the VAE pool in 98 to 100 % of the utility units and in 84 to 85 % of the fidelity units, with median effects of 1.6 to 3.3 noise units. Diffusion-TS is better in 71 % of the utility units (median 0.4 noise units). TimesFM is better in utility (median 2.4 noise units) but not in fidelity, Chronos is worse in both, and so is Timestral (median −3.1 noise units in utility). In the exploratory repetition against the assembly median of the zEDGE pool, whose methods lead the leaderboard, the open long-series generators fall behind assembly: Diffusion-TS, TimesFM, Chronos and Timestral are better than this median in only 17 to 33 % of the utility units and 17 to 36 % of the fidelity units, whereas zEDGE_TS is better in 53 to 76 % of the utility units. Diffusion-TS and TimesFM are therefore competitive with assembly from the open VAE pool, but not with assembly from the zEDGE pool.

RQ3: within assembly. On the VAE pool, every assembler except the matched-block bootstrap moves the series towards the real data compared with the random path: the median effect is positive in 59 to 83 % of the fidelity units and 72 to 90 % of the utility units. The matched-block bootstrap is equivalent to the random path on 112 of 158 fidelity units, within the noise of the benchmark (96 to 131 under other orderings of the seeds; § 9.6). Seam processing goes the other way: on the path of zseries, the only path with joiners in profile min, each of the three joiners makes the series worse than the same path without processing in 83 to 93 % of the utility units. The exploratory repetition on the zEDGE pool gives the same direction: the assemblers are better than the random path in 59 to 90 % of the utility units, and the joiners are worse than no processing in 72 to 88 %. On this pool, the mean of the assemblers also passes the rule in the regression cell of C3 (Holm p=0.03p = 0.03, median effect 1.3 noise units); this test is exploratory and outside the confirmatory family.

RQ4: does fidelity predict utility? 388 of 956 pairs of a fidelity metric and a utility criterion (the memorisation metrics included) are predictive in the sense of the protocol: the metric’s rank correlation with the transfer ratio across the methods of this run is positive over the datasets after correction. Fidelity against train predicts best (173 of 316 pairs, against 116 for OOT and 99 for val), and classification on OOT is the easiest criterion to predict. The strongest predictors are the target metrics (F8), which measure the events that the tasks are built on, and the realism of window features (F6.6), with median correlations above 0.8. In the replication run, with 16 methods from five generator families, no pair is predictive after correction: with nine datasets the smallest attainable pp is 0.004, which 68 pairs reach where the correction over 956 pairs requires 75. The main run restricted to the same methods has 85 predictive pairs and a median correlation of 0.21 against 0.20, so the runs differ at the threshold of the correction rather than in the strength of the association. The number of predictive pairs depends on the generator families compared: in a supplementary leave-one-family-out analysis, the main run has 85 predictive pairs without the zEDGE family and 235 without the VAE family, and 44 of the 388 pairs stay predictive whichever family is left out. A metric chosen on the other families orders the methods of a held-out family against the rest in 75 to 88 % of the pairs (median over datasets), with the 95 % bootstrap interval above chance in 19 of 24 combinations of a criterion and a held-out family; in the replication run it does so for Chronos, TimesFM and Timestral but not for Diffusion-TS.

RQ5: gap to real data and stability. No method reaches the real series. For each of the 30 methods and each of the four criteria (classification and regression, on val and OOT), the median transfer ratio over datasets is below 1, and the one-sided Wilcoxon test that it is below 1 is significant after Benjamini–Hochberg correction (120 of 120; pBH≤0.031p_{\mathrm{BH}} \le0.031; the upper end of every 95 % bootstrap interval is below 1, at most 0.996). The test is against the value 1, without a margin; for the best methods the gap is small: the largest median ratio of a method on a single criterion is 0.98. For ROC AUC the ratio is undefined in 160 of the 2700 cells (5.9 %), in which the real series itself is within 0.02 of chance, mostly on Bitcoin and Beijing PM2.5; these cells are left out, and each test uses 6 to 9 datasets. Absolute scores are given in Table 15. Over the four criteria, the median ratio is 0.95 for the best method (zedge_ts_ar), 0.74 for Diffusion-TS, 0.67 for the best method of the VAE pool and 0.39 for Timestral. The order of the methods survives the shift from val to OOT (τ\tau from 0.51 for the windows to 0.97 for the seams, 0.80 to 0.83 in utility) and the change of the downstream model (τ\tau from 0.75 to 0.86 between the linear model and gradient boosting).

RQ6: cost. Training dominates the cost of every trained method. The median time of a series is 1.2 s for Timestral, 2.4 s for TimesFM and 33 s for Chronos. The VAE pool costs about 130 s per dataset; the solvers add 31 s (ATSP) and 46 s (PCTSP), and the other assemblers and joiners at most a few seconds. The zEDGE pool costs about 800 s, Diffusion-TS 2 350 s and zEDGE_TS 2 930 to 2 950 s. The Pareto front of rank against time therefore runs from the pretrained forecasters, which are fastest but low in the ranking, to zedge_quilting, which is best and costs about as much as its pool; every method of the zEDGE pool costs the same order of time. The slope of the cost in KK is not available in profile min.

RQ7: moderators. None of the 176 correlations between the dataset descriptors and the dataset effects of RQ2 and RQ3 is significant after correction. With nine datasets the analysis has little power, and the question remains open.

RQ8 and RQ9: class balance, outliers and extremes. In both slices, the order of the families is the same as in RQ2 and RQ3. zEDGE_TS is better than the assembly median in 95 to 97 % of the class-balance units and 83 % of the extreme-value units; most assemblers of the VAE pool are better than the random path in most units, and Timestral and Chronos are worse than the assembly median. The fidelity metrics of class balance predict the utility of the classification tasks (for example, the error of the class shares against OOT, ρ=0.80\rho= 0.80), whereas no fidelity metric of the tails predicts the error on extreme targets after correction.

Supplementary checks

The checks of this subsection were added after the protocol was frozen and are exploratory (§ 7.6).

Calendar. The calendar decoded from the VAE pool advances by exactly one hour at 68 to 78 % of the steps (median over datasets and seeds), against 99 % for the zEDGE pool and 100 % for the long-series generators; inside a window, its median error is one hour, and 3 to 20 % of the steps are three or more hours off (table 10). Most assemblers have fewer such steps than the random path (3 to 17 % against 18 %; the matched-block bootstrap 20 %); the jump at the seam, which several assemblers minimise, is measured over the calendar channels too, and kernel herding, which compares only the value with train, also tends to pick windows with a more accurate calendar. Part of the effect of window selection in RQ3 may therefore act through the calendar. Without the fidelity metrics that use the calendar of the series, the overall order changes only by a few swaps of neighbouring methods (Kendall’s τ=0.96\tau= 0.96), one borderline method leaves the top tier, and the fidelity shares of RQ2 and RQ3 change by at most four percentage points. The utility tasks cannot be compared in this way, because each uses the hour of the week as a feature.

Seam metrics. The chance level of the share of change points at seams (F5.6) depends on the density of the seams: 0.6 % of the positions lie within three points of a seam for Timestral, 4 % for the assemblers, 8 % for Diffusion-TS and zEDGE_TS and 11 % for Chronos. Against its chance level, obtained by shifting the seams, the share is 2.5 to 3.9 times the chance level for the random paths, the matched-block bootstrap and herding, and below it for Chronos (0.83), Diffusion-TS (0.45) and zEDGE_TS (0.38 and 0.69); the raw share therefore penalises the methods with dense seams, although the order of the methods by the two statistics agrees (τ=0.81\tau= 0.81). Ranked on the seam metrics defined for every method, the overall leaderboard keeps its order (τ=0.94\tau= 0.94; 0.92 in the replication run), one borderline method joins the top tier in each run (vae_atsp in the main run, vae_greedy in the replication run), and no verdict of C1–C4 changes (section J).

Noise margin. The margins are estimated from ten seeds, that is, from nine differences of neighbouring seeds. A zero margin, on which the dataset leaves the global test, occurs for 2.7 % of the dataset and unit pairs of RQ2 and 3.9 % of RQ3; about 0.1 % have a margin near zero, and no verdict of equivalence rests on them. The verdicts of superiority do not depend on the order of the seeds: with every pair of seeds and with ten random orderings, no verdict of RQ2 or of C1–C4 changes. The verdicts of equivalence do: of the 435 global verdicts of equivalence in RQ3, 144 hold under every ordering, and their number ranges from 385 to 521 (for the matched-block bootstrap, from 96 to 131 of 158 fidelity units, against 112). They also depend on the size of the margin: with half the margin, no verdict of equivalence remains and seam processing becomes worse than no processing in the regression cell of C4; with twice the margin, window selection becomes equivalent to the random path in both cells of C3, and seam processing in the regression cell of C4. Verdicts of equivalence are therefore stated relative to the estimated noise margin (§ 7.1).

Missing utility cells. In the main run, 3 462 of 156 398 TSTR cells (2.2 %) could not be trained, all of them classification cells whose synthetic labels have one class; no cell failed for a technical reason. They concentrate in the pretrained forecasters, whose median forecast rarely crosses a threshold: 31 % of the cells of Timestral, 22 % of TimesFM and 11 % of Chronos, against 2 % of Diffusion-TS and at most six cells of an assembler. With a fixed chance-level penalty instead of the pre-specified imputation (§ 7.1), the overall order changes by one swap of the last two methods; with the cells left out, Timestral rises from 24th to 21st place. In neither case does the top tier, a verdict of RQ2 or RQ3, or a test of C1–C4 change. In the replication run, with 4.4 % of the cells missing, the top tier and the tests of C1–C4 are unchanged as well.

Reproduction on another hardware and software stack

Thirteen ranked methods are common to both runs: the nine assemblers on the VAE pool and the four open long-series generators. Their overall leaderboards are compared in full in Table 22; for this comparison the main run’s own copies of these methods are ranked among themselves, as in the replication run.

MethodReplication run (L4)Main run (H200)
PlaceMean rank95 % intervalTierPlaceMean rank95 % intervalTier
vae_herding15.54[4.86, 6.18]yes15.57[4.62, 6.49]yes
diffusion_ts26.27[4.76, 7.59]yes36.24[4.91, 7.43]yes
vae_quilting36.36[5.82, 6.92]yes46.51[5.98, 7.10]yes
vae_zseries*46.41[5.67, 7.20]yes25.76[5.16, 6.35]yes
timesfm56.72[5.29, 8.27]yes56.77[5.43, 8.26]yes
vae_atsp67.11[6.55, 7.74]yes76.99[6.45, 7.51]yes
vae_rand77.20[6.45, 7.86]yes127.84[7.17, 8.52]yes
vae_mbb87.22[6.57, 7.83]yes107.71[7.02, 8.40]yes
vae_pctsp97.23[6.79, 7.70]yes66.97[6.63, 7.32]yes
chronos107.43[6.11, 8.59]yes117.75[6.78, 8.58]yes
vae_knn117.56[7.10, 8.08]yes97.53[7.06, 8.03]yes
vae_greedy127.60[7.16, 8.08]no87.18[6.76, 7.64]yes
timestral138.56[7.46, 9.50]no138.42[7.48, 9.27]no

Table 22. Overall leaderboard of the 13 methods common to both runs: the replication run (NVIDIA L4) and the main run’s own copies of these methods ranked among themselves (NVIDIA H200). Place, mean rank, its 95 % bootstrap interval and top-tier membership. * proprietary (zypl.ai).

  • The orders agree with Kendall’s τ=0.64\tau= 0.64. The first method is the same.

  • Every mean rank of the main run lies inside the 95 % interval of the replication run, and 12 of 13 mean ranks of the replication run lie inside the intervals of the main run.

  • The top tier is the same except for one method at its boundary: vae_greedy, outside the tier in the replication run (p=0.05p = 0.05 after Holm) and inside it in the main run.

  • The places within the tier do not reproduce: places 2 to 4 hold the same methods in a different order; the place shifts most for vae_rand (7th in the replication run, 12th in the main run) and vae_greedy (12th and 8th), and the mean rank for vae_zseries (4th and 2nd place) and vae_rand.

The differences have four sources, identified by comparing the per-unit tables of the two runs.

  1. Trained generators. The VAE (TensorFlow) and Diffusion-TS were retrained on a different GPU and software stack. Early stopping chose different epochs (by up to 148 epochs for the VAE), so the weights, the pools and every method built on them differ: the median relative difference of the Wasserstein distance (F1.2) between the runs is 9 to 28 % for the methods of the VAE pool and 13 % for Diffusion-TS. This is the main source.

  2. Sampling. Chronos is not trained, but with the same seeds its sampled trajectories differ between the GPUs (median relative difference of F1.2, 48 %).

  3. Downstream models. On identical series (the real series, TimesFM and Timestral) the downstream scores coincide in the median, with a maximum difference of 0.03 in AUC and 0.04 in MASE.

  4. Data. The prepared ETT series of the main run differs at the value level from that of the replication run, with the same length, split and dates. The cause is not established: the series that the source gives today is identical to that of the replication run and matches the published content hash. The effect on the metrics is small (at most 3⋅10−43 \cdot10^{-4} for the real series); the other eight datasets are identical in both runs.

The deterministic forecasters confirm this attribution. TimesFM and Timestral produce practically the same series on both GPUs: the median difference of their fidelity values is 0, and fewer than 0.3 % of the values differ by more than 1 % (at most 2.7 %). The two runs thus give similar rankings of the 13 shared methods: membership of the top tier reproduces up to one borderline method, whereas the places within the tier do not. The agreement is a reproducibility check; the variability across training, software, hardware and data is not estimated from two runs.

Limitations

This section describes the scope of the benchmark and the conditions that limit its conclusions.

Scope. The benchmark covers univariate hourly series with a window of one week. Multivariate series, other sampling frequencies, irregular sampling and exogenous covariates are not covered, and each dataset contributes a single target series. The nine datasets are taken from energy, environment, traffic, mobility and finance. Domains with other dynamics are not covered.

Length of the main run. The main run is carried out in profile min: the last 10 000 hours of each dataset are used, so the synthetic series consists of K=30K = 30 windows, which is about seven months. This length is sufficient for the seams and for structure of several weeks, but not for yearly seasonality and slow regimes. The scaling of the methods to the long series of profile max (KK up to 519) is not tested in this run. In profile min the VAE and Diffusion-TS are trained once per dataset; therefore, the seeds do not capture the variance of training, and a per-dataset conclusion is conditional on the trained model. The second random path is not run, so the noise level is taken from the spread between seeds of the random path, and check (a) of RQ0 (§ 7.3) is not available. This noise level, which is also the equivalence margin (§ 7.1), includes the sampling variance of the pool: it makes the verdicts of superiority conservative and the equivalence region wide, and the verdicts of equivalence vary with the order of the seeds and the size of the margin (§ 9.6). The independent estimate from the second random path belongs to profiles mid and max.

Long-range structure. Group F7 describes structure longer than a window through distributions of rolling means and standard deviations, weekly-profile summaries and the mean duration of the high-volatility regime. It does not isolate how variation changes with the aggregation scale, the chronological dependence between weeks, or the full distribution of regime durations, transition probabilities and recovery from drawdowns.

Statistical power. The unit of a global conclusion is the dataset, and there are nine datasets. After correction, a global verdict requires a large and consistent effect. An effect present on two or three datasets appears only in the per-dataset verdicts, and an inconclusive verdict leaves the question open. The top tier is defined by comparisons with the observed leader, without adjustment for its selection. With nine datasets the smallest one-sided pp is 1/512, against a corrected level near 10−410^{-4} for all pairs, so a Holm procedure over every pair of methods rejects no pair (section M). The moderator analysis (RQ7) has nine points and is exploratory.

Stability of places. The places within the top tier are not separated by the tests and differ between the two runs, mainly because the trained generators are retrained on another stack (§ 9.7); conclusions are therefore stated as tier membership. The replication run does not cover the zEDGE pool or zEDGE_TS, and the comparison of the two runs does not isolate the effect of the hardware.

Pool generators. Every assembly method draws on the pool of a generator, so the conclusions about assembly are conditional on the pool. The main run has two pool generators, the β\beta-VAE and zEDGE, with one pool size (N=300N = 300) and fixed settings; other window generators may favour other assemblers. The pre-specified contrasts within assembly and the comparison of long-series generators with assembly are computed on the VAE pool only (§ 7.5). In their exploratory repetition on the zEDGE pool, window selection and seam processing keep their direction, whereas the open long-series generators fall behind assembly; the open long-series generators are therefore competitive only with assembly from the VAE pool. The settings of the VAE that are not defaults of TSGM (β\beta, the latent size, the epoch limits and the prior scale) come from earlier work on other data, and the sensitivity of the results to them, in particular to the prior scale, is not tested. The decoder of the VAE ends with a sigmoid, so its windows never leave the range of train; in the main run, they do not reach its upper bound either (their maximum is 0.80 to 0.98 of the range on the nine datasets). The VAE pool therefore under-represents the extremes of train, which bears on the tail metrics of its assemblies.

Calendar. The method families obtain the calendar differently (section 4.3), and the calendar decoded from the VAE pool is less accurate than the real one (§ 9.6). The calendar-continuity metrics are left out of every ranking, and leaving out the other fidelity metrics that use the calendar changes the order little. The hour of the week is, however, a feature of every utility task, so the utility of the methods with a less accurate calendar, in particular those of the VAE pool, may be lowered by their calendar and not only by their values; for the same reason, part of the effect of window selection may act through the calendar. A comparison in which every method receives a common valid calendar requires the utility stage to be run again.

Pretrained forecasters. TimesFM and Timestral are evaluated in their median-output configurations: their series is a median forecast, which is deterministic and smoother than a sample. The conclusions concern these configurations, not the full probabilistic output of the models; a series built from their quantiles or from sampled paths could give other scores. Chronos is run with its small model (chronos-t5-small) in profile min. The leakage audit relies on the published descriptions of the data on which each model was trained, pretraining and, where described, post-training, and, where possible, on the data themselves. Timestral is pretrained on synthetic data only but post-trained on a mix of real and synthetic data whose real part is not listed [[8]]; its pairs are therefore marked unknown, and its results may include datasets that the model has seen.

Proprietary methods. zseries, zEDGE and zEDGE_TS are evaluated as black boxes: their code is not released, their design is not described apart from the sampling modes of zEDGE_TS, and third parties cannot rerun them. zseries uses the thresholds of task U2.1, which are fixed on train like every threshold of the benchmark; no open assembler uses them. U2.1 is the classification cell of the confirmatory family, and C3 averages over every assembler, zseries included; the verdicts of C3 are the same without it (§ 9.4). In profile min the joiners are applied to the path of zseries only (section 4.2), so C4 and the seam contrasts of RQ3 can be checked from the published results, while recomputing them requires the code of zseries. The joiners themselves are released and can be applied to the path of any open assembler; this extension is outside the pre-specified protocol. All authors are employed by the company that develops these methods. The protocol was frozen before the main run (§ 7.6), the methods receive the same inputs and evaluation as every other method, and every ranking can be recomputed without them (Competing interests).

Utility protocol. The downstream models are linear models, gradient-boosted trees and a small network with fixed hyperparameters and light configurations. Sequence models are not included, and in profile min the network is trained on one task only. All tasks are derived from the series itself, with the same 11 features. The benchmark measures the quality of the synthetic data (TSTR against TRTR), not its value as augmentation of real data. Group F8 describes the events of one task (U2.1).

Metrics and aggregation. Many fidelity metrics are correlated, and the equal weights of the hierarchy are a choice of the protocol. The robustness checks of § 9.2 show how much the order depends on this choice. The seam metrics are undefined for a series without seams, and the calendar-continuity metrics are left out of every ranking and question. Memorisation is measured by distances to the training windows; formal privacy guarantees such as differential privacy and membership or attribute inference attacks are outside the scope. The near-copy flag of the protocol is defined relative to one real OOT series per dataset, whose own share of near copies is 0 or 1 on six of the nine datasets, so the flag is raised for none of the controls, an exact copy of train included (section K); the signed ratio and the count of exact copies are reported instead. The validity check of the metrics against real data rests on one real block per dataset, the OOT series; controlled perturbations of the real series and repeated length-matched real blocks would separate the sensitivity of a metric from temporal drift. The fidelity–utility correlations (RQ4) are computed across methods, several of which share a pool or a generator, so the number of predictive pairs depends on the families compared (§ 9.5). The leave-one-family-out check covers six families in the main run and five in the replication run; families outside the benchmark are not tested.

Cost and data. Times and memory are comparable only on one type of hardware, and all costs reported here are obtained in one run. Some sources change over time (Bitcoin is updated daily); the data hashes detect such changes but cannot restore the original data. The ETT series of the main run differs slightly from the one that the source gives today (§ 9.7), so a new run reproduces the ETT results of the replication run rather than those of the main run. For ETTh1 and PJM the artifacts derived from the data are not published for licensing reasons (section A) and have to be regenerated.

Planned extensions. The next version adds an independent noise estimate from the second random path, a replication with identical input data and fully pinned environments, a common valid calendar for every method in the utility stage, the seam joiners on the paths of the open assemblers, a memorisation flag calibrated on controls, and controlled perturbations and repeated length-matched real blocks for the validity check of the metrics.

Conclusion

MarathonTS compares two ways of obtaining a long synthetic time series, the assembly of generated windows and long-series generation, under one protocol, with fidelity, utility, memorisation and cost measured on the whole series and a statistical analysis frozen before the main run was carried out. On nine hourly datasets, the main run leads to four conclusions. First, the methods differ, and nine datasets separate tiers rather than places: 15 of 24 methods are not shown to be worse than the observed leader, which is not equivalence. In a replication run of the 13 ranked methods of the VAE pool and the open long-series generators on a different hardware and software stack, in which the zEDGE methods were not rerun, the top tier of these methods reproduced up to one method and the order within it changed; results are therefore read by tier. Second, in assembly on the VAE pool, window selection often moves the series towards the real data compared with a random choice of windows, whereas the seam joiners, tested on the zseries assembly, often reduce its utility; neither confirmatory contrast is conclusive under the pre-specified rule. Third, long-series generators do not form one group: Diffusion-TS and TimesFM are competitive with assembly from the open VAE pool and Chronos and Timestral fall behind it, whereas assembly from the zEDGE pool is, descriptively, ahead of all four open generators. Fourth, no method trains the downstream models as well as the real series: the transfer ratio is significantly below 1 for every method, by a few per cent for the best ones, so the gap to real data remains open.

Nine datasets set the resolution of this run: the global comparisons of methods and of assembly choices (RQ2, RQ3, RQ8, RQ9) do not reach superiority after the correction for multiple comparisons, and the moderators of the effects are not identified. A larger set of datasets, the second random path of profiles mid and max and longer series are the natural next steps; the software is designed so that new datasets, generators and assemblers enter the comparison through a plugin, and the published tables allow every ranking to be recomputed with any set of methods.

Use of generative AI

The authors used Claude (Anthropic), a generative AI assistant, to help write and review code, edit the text and prepare figures. The authors checked all such output and take full responsibility for the content of the paper and the software.

Competing interests

All authors are employed by zypl.ai. Three of the evaluated components are proprietary methods of zypl.ai that are under development: the assembler zseries (vae_zseries, zedge_zseries), the pool generator zEDGE and the long-series generator zEDGE_TS (zedge_ts_long, zedge_ts_ar). Their code is not released, and of their design only the two sampling modes of zEDGE_TS are described (section 4.3). Their results are reported so that they are measured under the same protocol and on the same leaderboard as the open methods. The following steps are taken to keep this interest from shaping the comparison, and the limits of these steps are stated.

  1. The protocol, the metrics, the rankings and the pre-specified confirmatory family were frozen before the main run was carried out (§ 7.6); the changes made after the freeze are listed and are not part of the pre-specified analysis. They are identical for every method.

  2. No open method uses any component of the proprietary ones: each alternative assembler uses the seam criterion of its own literature, and the solvers without one use a generic jump at the seam.

  3. The proprietary methods use the training split only, like every other method, are evaluated by the same code, and their training time is included in their cost. One input differs: zseries uses the thresholds of task U2.1, fixed on train like every threshold of the benchmark, which no open assembler uses; U2.1 is also the classification cell of the confirmatory family (§ 7.4 and § 10).

  4. The results of the proprietary methods cannot be reproduced by third parties. All other results can be reproduced from the released code, and the released analysis recomputes every ranking with any set of methods excluded.

Datasets: sources, licences and leakage audit

Sources and licences. Table 6 lists the source, the licence and the reference of every dataset. The data themselves are not redistributed: they are downloaded from the sources at the first stage of a run. Results computed from the data (metrics, tables, rankings) are published for all nine datasets. Artifacts that contain values derived from the data (synthetic series, pools, generator weights, labels, predictions) are published under the licence of each dataset, with two exceptions. ETTh1 is licensed under CC BY-ND 4.0, which does not permit sharing adapted material. Whether synthetic series and generator weights are adapted material in this sense is not settled, so the conservative reading is taken. The primary source of PJM restricts the redistribution of data derived from it. For these two datasets the derived artifacts are withheld and are reproduced locally by running the pipeline.

DatasetSourceLicenceReference
ETTh1GitHub, ETDatasetCC BY-ND 4.0Zhou et al. [2021]
ElectricityUCI #321CC BY 4.0Trindade [2015]
HouseholdUCI #235CC BY 4.0Hébrail and Bérard [2012]
PJMEKaggle, Hourly Energy ConsumptionCC0 1.0Mulla [2018]
PM2.5UCI #381CC BY 4.0Liang et al. [2015]
JenaMPI-BGC weather station (Keras mirror)CC BY 4.0Max Planck Institute for Biogeochemistry [2016]
MetroUCI #492CC BY 4.0Hogue [2019]
BikeUCI #275CC BY 4.0Fanaee-T and Gama [2014]
BitcoinKaggle, Bitcoin Historical DataCC BY-SA 4.0Zielinski [2026]

Table 6. Dataset sources and licences.

Preprocessing per dataset. For each source, one target column is taken and brought to an hourly step. ETTh1 (oil temperature of a transformer, July 2016 – June 2018), PJME (load of the PJM East region, 2002–2018), PM2.5 (concentration at the US Embassy in Beijing, 2010–2014) and Metro (westbound traffic on I-94, Minneapolis – St Paul) are hourly at the source. Electricity is the total load of 370 Portuguese clients (2011–2014, 15-minute step), Household the active power of one household in France (2006–2010, 1-minute step) and Jena the air temperature of a weather station (2009–2016, 10-minute step); all three are averaged per hour. Bitcoin is the logarithm of the last BTC/USD price of each hour (Bitstamp, 1-minute step), taken from 1 January 2017, before which many minutes have no trades, to 1 October 2026. Duplicate hours of Metro are averaged, and the series starts after its last gap longer than 30 days (June 2015). A missing hour of Bike means that no bicycle was rented and is set to zero. PJME has isolated duplicates and gaps at the changes of daylight saving time. In every dataset, gaps of up to 6 hours are interpolated and longer gaps are left missing.

Leakage audit. In October 2026, every pair of a pretrained model and a dataset was checked against the primary description of the pretraining data: for Chronos, Table 3 of Ansari et al. [ [6], whose Benchmark II is used for zero-shot evaluation only; for TimesFM 3.0, the model card and the full list of sub-datasets of its pretraining corpus, with the candidate sub-datasets downloaded and compared with the benchmark’s raw files; for Timestral, the technical report Petralia et al., [8], which describes pretraining on synthetic data only and post-training on a mix of real and synthetic data without a list of the real data, the model card and the release note [66]. The results of the audit are shown in Table 7. Exact and related pairs are excluded (§ 4.2); weak pairs, which share only the phenomenon, are kept. Earlier versions of TimesFM were also trained on Electricity and Jena; the benchmark uses version 3.0 only.

DatasetChronosTimesFM 3.0Timestral
ETTh1none (evaluation only)none (a test set, not in pretraining)unknown
Electricityexact: Electricity at 15 min, hourly and weeklynone (a test set)unknown
Householdnoneexact: the same series under another name, 2007–2010unknown
PJMEnonenone foundunknown
PM2.5related: KDD Cup 2018, Beijing stations, 2017–2018related: Beijing Air Quality, 12 stations, 2013–2017unknown
Jenaweak: gridded reanalysis onlyweak: gridded reanalysis onlyunknown
Metrononenoneunknown
Bikeweak: bike rentals of another citynoneunknown
Bitcoinnonerelated: daily Bitcoin price to 2021unknown

Table 7.

Table 7: Leakage audit of the pretrained models. Exact: the same data in the pretraining corpus; related: the same object at other stations, years or resolution. Both lead to exclusion. Unknown: the training data of the model are not listed, so the pair cannot be checked; it is kept.

Compared methods and their settings

Table 8 lists the methods of the main run (profile min) and their settings. Every assembler and seam joiner is run on both pools, under the prefixes vae_ and zedge_. For the assemblers that minimise the jump at the seam, the jump is the Euclidean distance between the last point of a window and the first point of the next over all three channels.

MethodIdeaSettings
A. Window selection and order (KK of NN windows, without repetition)
<pool>_zseriesproprietary (zyp1.ai), evaluated as a black boxnot released
<pool>_randKK random windows in random order; control—
<pool>_greedynearest neighbour by the jump at the seam, from a random window—
<pool>_atspasymmetric TSP over the pool, Lin–Kernighan–Helsgaun [46]; cheapest run of KK consecutive windows of the tourLKH via elkai
MethodIdeaSettings
<pool>_pctspprize-collecting TSP [47]: which KK windows and their order at onceOR-Tools; at most 2 000 solutions or 60 s per unit
<pool>_mbbmatched-block bootstrap [43]: P(i→j)∝exp⁡(−12(dij/h)2)P(i \to j) \propto\exp\left(-\frac{1}{2}(d_{ij}/h)^2\right)h=median jumph = \text{median jump}
<pool>_knnkk-NN resampling [44]: next window among the kk nearest by first point, weight ∝1/rank\propto1/\text{rank}k=round⁡(N)k = \operatorname{round}(\sqrt{N})
<pool>_quiltingimage quilting [5]: random among near-best overlaps, cut at minimum erroroverlap 12; tolerance 1.1×1.1 \times best
<pool>_herdingkernel herding [48]: KK windows minimising the MMD to the training windows, random orderevery 4th point of a window as features
B. Seam processing (on the path of <pool>_zseries)
<pool>_zseries_xfadecrossfade of the overlap with a Hann window [30]overlap 12
<pool>_zseries_wsolaWSOLA [29]: shift up to 12 points to the best correlation, then crossfadeoverlap 12
<pool>_zseries_quiltswitch to the next window at the point of smallest differenceoverlap 12
C. Long-series generators
diffusion_tsDiffusion-TS [2], autoregressive infilling with context T/2T/2; calendar givenofficial recipe, 18 000 steps, early stopping on holdout; 50 DDIM steps; one model per dataset
chronosChronos [6], one sampled trajectory in chunks of 64chronos-t5-small, context 512, default sampling
timesfmTimesFM 3.0 [7], median forecast in one passgoogle/timesfm-3.0-pytorch, context up to 15 360
timestralTimestral-209M [8], median forecast in blocks of 1 024mistralai/Timestral-209M, context up to 32 768
zedge_ts_longzEDGE_TS (zypl.ai, proprietary): inpainting in blocks of 12 latent vectors, conditioned on the last 3 of the previous blocknot released; one model per dataset, shared with zedge_ts_ar
zedge_ts_arzEDGE_TS (zypl.ai, proprietary): autoregressive, one latent vector at a time, conditioned on the preceding 11not released; the same model as zedge_ts_long

Table 8.

Table 8: Methods of the main run and their settings. <pool> is vae or zedge.

The pool generators are the β\beta-VAE of section 3.4 (convolutional encoder and decoder of TSGM, latent dimension 16, β=2\beta= 2, Adam with learning rate 10−310^{-3}, batch 64, up to 400 epochs, early stopping on the holdout loss with patience 50 after a minimum of 150 epochs, sampling from N(0,0.52I)\mathcal{N}(0, 0.5^2 I), one model per dataset) and zEDGE, which is proprietary and not described. Both generators give N=300N = 300 windows per dataset and seed. Diffusion-TS is trained with the official recipe with the window length TT: batch 128 with gradient accumulation 2, Adam with a warm-up from 10−510^{-5} to 8⋅10−48 \cdot10^{-4} over 500 steps followed by reduction on plateau, and an exponential moving average of the weights. Two deviations from the official recipe are made: early stopping on the holdout windows is used, as for every trained generator of the benchmark, and infilling is carried out with 50 instead of 200 sampling steps, to bound the generation time. The pretrained forecasters are used zero-shot with the default settings of their libraries and pinned model revisions.

Full list of fidelity metrics

Table 9 lists the 66 fidelity metric IDs; 11 of them have several outputs (lags, quantiles, window lengths), 83 outputs in all. Unless stated otherwise, every metric is a deviation from the reference; that is, 0 is ideal and lower is better. Δk\Delta_k is the change over kk hours; zz means that both series are standardised with the mean and standard deviation of the reference. Windows are taken with length TT and step T/4T/4, at most 3 000 per series. The 16 window features are the mean, standard deviation, skewness, kurtosis, minimum, maximum, 10th and 90th percentiles, mean ∣Δ1∣|\Delta_1|, standard deviation of Δ1\Delta_1, lag-1 and lag-24 autocorrelation, rate of sign changes of the standardised window, relative positions of the maximum and the minimum, and the last minus the first value.

IDnnHow it is computed
F1. Marginal distribution of values
F1.11Kolmogorov–Smirnov statistic
F1.21Wasserstein-1 distance divided by the standard deviation of the reference
F1.31Anderson–Darling kk-sample statistic
F1.41correlation of the quantiles 0.01–0.99 (higher is better)
F1.53quantile errors in reference standard deviations: mean over p∈{0.01,0.05,0.25,0.5,0.75,0.95,0.99}p \in\{0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99\}, and at 0.01 and 0.99
F1.64errors of the mean, standard deviation (∣log⁡ ratio∣|\log\text{ ratio}|), skewness and excess kurtosis
F1.71share of points outside the range of the reference
F1.81share of the 50 quantile bins of the reference hit by the synthetic series (higher is better)
F1.91Jensen–Shannon distance of the histograms on the same bins
F1.101error of the Hill tail index on the upper 5%
F2. Increments
F2.15KS of Δk\Delta_k, k∈{1,6,24,168}k \in\{1, 6, 24, 168\}; Wasserstein-1 of Δ24\Delta_{24}
F2.22∣log⁡ ratio∣|\log\text{ ratio}| of the standard deviations of Δ1\Delta_1 and Δ24\Delta_{24}
F2.31error of the share of positive Δ1\Delta_1
F2.41error of the share of ∣Δ1∣|\Delta_1| above the reference 99th percentile
F3. Temporal dependence
F3.11mean absolute error of the autocorrelation, lags 1…T−11 \ldots T - 1
F3.21the same, lags T…3TT \ldots3T, beyond one window
F3.31mean absolute error of the partial autocorrelation, lags 1–48
F3.42mean absolute error of the autocorrelation of ∣Δ1∣|\Delta_1| and of Δ12\Delta_1^2, lags 1–48 (volatility clustering)
F3.51mean ∣log⁡10 ratio∣|\log_{10} \text{ ratio}| of the normalised Welch power spectra
F3.62error of the share of spectral power at the daily and the weekly frequency
F3.71error of the exponent of detrended fluctuation analysis, scales 16–256
F3.81error of the correlation of the value with the sine and cosine of the hour of the week
F4. Seasonality and calendar
F4.11mean absolute error of the mean profile over the 168 hours of the week (zz)
F4.21the same for the standard deviation profile
F4.31the same for the mean profile over the 24 hours of the day
F4.41share of steps at which the hour of the week advances by exactly one (higher is better)
F4.51share of seams at which it does not
F5. Seams (undefined for a series without seams, except F5.5)
F5.11KS between Δ1\Delta_1 at the seams and Δ1\Delta_1 of the reference
F5.21KS between Δ1\Delta_1 at the seams and Δ1\Delta_1 elsewhere in the same series
F5.32∣Δ1∣|\Delta_1| at a seam in standard deviations of the reference Δ1\Delta_1: mean and 95th percentile
F5.41difference of the lag-1 autocorrelation within ±24\pm24 points of the seams and of random positions
F5.51error of the rate of change points (PELT, on 2-hour means)
F5.61share of the detected change points within ±3\pm3 points of a seam
F5.71∣AUC−0.5∣|\mathrm{AUC} - 0.5| of a logistic regression telling a snippet around a seam from one elsewhere
F6. Distribution of windows
F6.11discriminative score: ∣accuracy−0.5∣|\mathrm{accuracy} - 0.5| of a GRU classifier, real against synthetic windows, split in time [1]
F6.21∣AUC−0.5∣|\mathrm{AUC} - 0.5| of a gradient-boosting classifier on window features (C2ST, Lopez-Paz and Oquab, [17])
F6.31predictive score: error of a one-step ridge forecast trained on the synthetic series against one trained on real data
F6.42unbiased MMD2\mathrm{MMD}^2 with an RBF kernel [16] between windows of length TT and 2T2T
F6.61Fréchet distance between the window features [18]
F6.81log⁡\log ratio of the median nearest-neighbour distance synthetic →\to real to real →\to real
F7. Long-range structure
IDnnHow it is computed
F7.11KS of the rolling mean (zz), windows of 168 and 720 hours
F7.21KS of the rolling standard deviation (zz), same windows
F7.31∣log⁡ ratio∣|\log\text{ ratio}| of the rates of crossing the reference 0.1, 0.5 and 0.9 quantiles
F7.41KS of the lengths of excursions above the 0.9 and below the 0.1 quantile
F7.51KS of the drawdown from the rolling weekly maximum (zz)
F7.62error of the linear trend per week; ∣log⁡ ratio∣|\log\text{ ratio}| of the standard deviation of the weekly means
F7.71∣log⁡ ratio∣|\log\text{ ratio}| of the mean duration of the high-volatility regime of a two-state Gaussian HMM on Δ1\Delta_1
F8. Tail events of task U2.1 (24-hour increase above the 0.90 quantile of train)
F8.11error of the share of events
F8.21KS of the durations of events
F8.31KS of the times between the starts of events
F8.41mean absolute error of the event rate by hour of the week
F8.51error of the event rate given the quintile of the past changes Δ1\Delta_1, Δ24\Delta_{24}, Δ168\Delta_{168}
F8.61KS of the next 24-hour change given the quintile of the current value
F9. Diversity and coverage (on window features, k=5k = 5; [20])
F9.12precision and recall (higher is better)
F9.22∣log⁡ density∣|\log\text{ density}|; coverage (higher is better)
F10. Memorisation (against train only; ranked separately)
F10.11∣log⁡ ratio∣|\log\text{ ratio}| of the median distance to the closest training window, synthetic against val windows
F10.21share of synthetic windows closer to train than the 5th percentile of the val windows’ distances
F11. Class balance (over every classification task)
F11.11mean ∣log⁡ ratio∣|\log\text{ ratio}| of the positive shares over binary tasks
F11.21the same over rare-event tasks (q≥0.95q \ge0.95)
F11.31total variation distance of the classes of the three-class task
F11.41share of binary tasks in which the synthetic series has no positive example
F12. Outliers and extremes
F12.11∣log⁡ ratio∣|\log\text{ ratio}| of the shares of points beyond the reference 0.5% and 99.5% quantiles
F12.21error of the mean exceedance beyond these quantiles
F12.31∣log⁡ ratio∣|\log\text{ ratio}| of the rates of local spikes (distance from the rolling median of 25 points)
F12.41KS of the weekly maxima
F12.51KS of the weekly minima

Table 9. Fidelity metrics. nn: number of metrics of the ID.

Calendar of the series

The calendar of a series is its hour of the week wt∈{0,…,167}w_t \in\{0,\ldots,167\}. For an assembly method it is decoded from the two calendar channels of each window, wt=round⁡(168⋅atan2⁡(2st−1,2ct−1)/2π) mod 168w_t = \operatorname{round}(168 \cdot\operatorname{atan2}(2s_t - 1, 2c_t - 1)/2\pi) \bmod168, where sts_t and ctc_t are the sine and cosine channels; the other methods return it directly. Two measures are computed for every series. The first is the share of steps at which the hour of the week advances by exactly one, as in Table 10. The second is the error inside a piece, the stretch between two seams: for every piece, the valid calendar (c+i) mod 168(c+i) \bmod168, i=0,1,…i = 0,1,\ldots, with the offset cc that is closest to the series is found, and the error of a step is the circular distance, in hours, between its hour and that calendar. A jump at a seam is therefore not an error, since a new piece may start at any hour, whereas an hour that jitters inside a window is. Table 10 gives the medians over datasets and seeds for both runs. Its code is marathonts/hypotheses/calendar_check.py.

Utility tasks and models

Tasks. Table 11 lists the 14 tasks. The quantile thresholds are computed on the changes of real train and are fixed in the repository before the run.

GroupTypeTargetPrimary metric
U2.1binaryΔ24\Delta_{24} above the 0.90 quantileROC AUC
U2.2binaryΔh\Delta_h above the 0.90 quantile, h∈{1,6,168}h \in\{1, 6, 168\}ROC AUC
U2.3binaryΔ24\Delta_{24} above the 0.95 and the 0.99 quantile (rare events)ROC AUC
U2.4binaryΔ24\Delta_{24} below the 0.10 quantileROC AUC
U2.53 classesΔ24\Delta_{24} below the 0.10, between, or above the 0.90 quantilelog-loss
U2.6regressionvalue xt+hx_{t+h}, h∈{1,24,168}h \in\{1, 24, 168\}MASE
U2.7quantilesquantiles 0.1, 0.5 and 0.9 of xt+24x_{t+24}pinball loss
U2.8multi-stepthe vector xt+1,…,xt+24x_{t+1}, \ldots, x_{t+24}MASE
U2.9regressionstandard deviation of the changes over the next 24 hoursMASE

Table 11. Downstream tasks. Δh\Delta_h is the change over the next hh hours.

Models. In the linear model and the network, the 11 features are standardised on their own training data, and for regression also the target; the trees use the raw features. The hyperparameters of profile min are fixed in advance and are as follows:

MethodMain runReplication run
Step +1Error90 %≥3\geq3 hStep +1≥3\geq3 h
vae_rand0.681418 %0.6820 %
vae_greedy0.721315 %0.6816 %
vae_atsp0.751210 %0.7512 %
vae_pctsp0.7412.510 %0.759 %
vae_mbb0.681520 %0.6919 %
vae_knn0.701417 %0.7017 %
vae_quilting0.741313 %0.7016 %
vae_herding0.76127 %0.6916 %
vae_zseries0.78123 %0.787 %
vae_zseries + seam joiners0.78–0.79123–7 %0.77–0.786–10 %
zEDGE pool, every assembler0.99000 %––
zedge_zseries + crossfade, WSOLA0.99004 %––
long-series generators1.00000 %1.000 %

Table 10 (PDF p. 40). Calendar of the series, medians over datasets and seeds. Step +1: share of steps at which the hour of the week advances by exactly one (F4.4). Error: distance of the hour of the week from the closest valid calendar inside a piece, in hours (median and 90th percentile). ≥3\geq3 h: share of steps with an error of three hours or more. The replication run has neither the zEDGE pool nor zEDGE_TS. Supplementary.

  • linear: logistic or softmax regression with balanced class weights and L2 penalty C=1C = 1 (L-BFGS, up to 300 iterations); ridge regression with α=1\alpha= 1 for regression, volatility and the multi-step task (exact solution); linear quantile regression with a smoothed pinball loss for the three quantiles at once.

  • gbdt: XGBoost with 100 trees, depth 4, learning rate 0.2 and 64 histogram bins; the positive class reweighted by (1−p)/p(1-p)/p for binary tasks and balanced weights for the three-class task; one model with the quantile loss for the three quantiles; a single ensemble with vector leaves for the multi-step task.

  • mlp: a fully connected network 11→64→32→k11 \to64 \to32 \to k with ReLU, Adam with learning rate 10−310^{-3}, weight decay 10−410^{-4}, batch 512, 30 epochs without early stopping; trained on task U2.1 only in profiles min and mid.

The probabilities of every classifier are mapped back from the balanced shares to the class shares of its own training data, P(c∣x)∝Pbal(c∣x)⋅share⁡(c)P(c \mid x) \propto P_{\mathrm{bal}}(c \mid x) \cdot\operatorname{share}(c), so that the Brier score, the log-loss and the calibration error measure calibration and not the reweighting.

Metrics. The following metrics are computed. Binary tasks: ROC AUC, PR AUC, Brier score, log-loss, expected calibration error (10 equal-width bins), F1 at the threshold that predicts the positive share of real train, and the lift in the top decile. Three-class task: macro one-vs-rest AUC, log-loss and macro F1. Regression and volatility: MAE, RMSE, MASE, sMAPE and the MAE on the test rows with the 5% lowest and highest targets. Quantiles: pinball loss, coverage of the 10–90% interval and the share of rows with crossing quantiles. Multi-step: MAE and MASE over the horizon and at its last step. MASE, the pinball loss and the error on extreme targets are divided by the MAE of the seasonal naive forecast (lag 168) on real train; MAE and RMSE stay in the units of the series.

Analysis protocol

Table 12 lists what the analysis protocol fixes and where it is stated in docs/HYPOTHESES.md; the date of the freeze and the changes made after it are given in § 7.6 and docs/PROTOCOL_AMENDMENTS.md. Two ablations of the zseries assembler listed in the protocol are not part of the release, and the published analysis is run without them.

Chance levels of the utility metrics

Table 13 lists the value that a utility cell which a synthetic series could not train receives under the pre-specified imputation (§ 7.1): the worse of the chance level of the metric and the worst value of any method on the same dataset, seed, task, model and split.

Eligibility and calibration of C1

C1 is a Friedman test with the datasets as blocks (§ 7.4). Its design is complete: Chronos (left out on Beijing PM2.5 and Electricity) and TimesFM (Beijing PM2.5, Bitcoin and Household) are the only ranked methods with a missing cell in either confirmatory cell, and both drop out of the test. C1 therefore compares 22 methods in the main run (the nine assemblers of the VAE pool, the nine of the zEDGE pool, the two variants of zEDGE_TS, Diffusion-TS and Timestral) and 11 in the replication run (the nine assemblers of the VAE pool, Diffusion-TS and Timestral), on all nine datasets. Table 14 gives the pre-specified test with a permutation pp (the values permuted within each dataset, 100 000 permutations) and three variants: every ranked method with the incomplete datasets kept (Skillings–Mack test; Skillings and Mack, [67] every ranked method on the five datasets where all of them have a value, and, in the main run, the 11 methods of the replication run. With the permutation pp, the Holm-corrected pp of C1 in the main run is 0.005 (classification) and 10−410^{-4} (regression), and no verdict of C1–C4 changes in either run.

RunVariantMethods × datasetsClassification ppRegression pp
mainpre-specified22 × 90.0014 / 0.00061.4⋅10−81.4 \cdot10^{-8} / ≤10−5\le10^{-5}
every method, incomplete (Skillings–Mack)24 × 90.0025 / 0.00123.6⋅10−93.6 \cdot10^{-9} / ≤2⋅10−4\le2 \cdot10^{-4}
every method, complete datasets24 × 50.0015 / 0.00020.0018 / 0.0003
methods of the replication run11 × 90.79 / 0.810.0002 / 3⋅10−53 \cdot10^{-5}
replicationpre-specified11 × 90.29 / 0.291.2⋅10−41.2 \cdot10^{-4} / 4⋅10−54 \cdot10^{-5}
every method, incomplete (Skillings–Mack)13 × 90.45 / 0.473.0⋅10−53.0 \cdot10^{-5} / ≤2⋅10−4\le2 \cdot10^{-4}
every method, complete datasets13 × 50.88 / 0.900.0065 / 0.0015

Table 14. C1 and its sensitivity. pp: chi-square approximation / permutation; ≤\le: the smallest attainable permutation pp (100 000 permutations; 5 000 for Skillings–Mack). Supplementary.

Absolute utility scores

Table 15 gives the absolute scores of the two confirmatory tasks in the main run: ROC AUC of U2.1U_{2.1} (higher is better) and MASE of U2.6U_{2.6} at h=24h = 24 (lower is better), for each downstream model and test split. TRTR is the real series (mean over seeds, median over the nine datasets); TSTR is given as the median over the 30 methods and for the best method, each the mean over seeds and the median over the datasets on which the method is evaluated (TimesFM and Chronos on fewer). In single cells the best method reaches the real series (a method of the zEDGE pool in classification, TimesFM and Timestral in regression); the transfer ratios of RQ5 are medians over the tasks and models of a criterion. The scores of every task in both runs are in the supplementary outputs (rq5_check).

Seam metrics

F5.6F_{5.6} is the share of the change points of a series (PELT on two-hour means) that lie within three points of a seam. If the change points had nothing to do with the seams, this share would be about the share of positions within three points of a seam, which grows with the density of the seams. For every series, the supplementary check (seam_check) recomputes F5.6F_{5.6} (equal to the published value in every series) and its chance level: the mean of the same share over 200 circular shifts of the seam positions by a random offset, which keep the spacing of the seams but not their placement relative to the change points. The ratio of the two is 1 when the change points fall on the seams as often as by chance and above 1 when more often. Table 16 gives the medians per method in the main run; the replication run gives the same picture. Across methods, the median F5.6F_{5.6} hardly follows the share of exposed positions (Spearman 0.08; 0.23 in the replication run), since most methods have a seam every 168 points; the methods with denser seams, Chronos, Diffusion-TS and zEDGE_TS, rank better when F5.6F_{5.6} is compared with its chance level, and Kendall’s τ\tau between the two orders is 0.81.

settingmin (default)midmax
seam joiners onzseries pathzseries pathevery assembler but random
pool size Nmax(300, 2K) = 300as minmax(300, 2K)
series length, splitlast 10 000 pts; 50/25/25as minfull series; 60/20/20
VAE trainings1 per dataset (9)1 per seed (90)1 per seed (90)
Diffusion-TS trainings1 per dataset (9)1 per dataset (9)1 per seed (90)
hyper-parameter searchnonogrid on tail of own train (TSTR: synthetic)
Chronost5-smallt5-smallt5-base
MLP tasksU2.1 onlyU2.1 onlyall tasks
MLP epochs30, no early stopas min≤ 200, early stop
GBDT100 trees, depth 4, 64 binsas min200 trees, depth 6, 256 bins
decision grid (RQ5 c)offoffon
seeds × datasets10 × 910 × 910 × 9
Each profile writes to data/processed/<profile>/ and artifacts/<profile>/; mid reuses min data.

Table 16. The three run profiles. min (default, intended for Colab) shares one trained generator per dataset across seeds. mid adds generator replicas. max uses the full series, retrains generators on every seed, and turns on hyperparameter search and the decision-transfer grid.

The seam criterion of the core rankings is formed by F5.7F_{5.7} (Table 3) and is undefined for TimesFM, which has no seams, and for Timestral, which has fewer than the ten classifier needs; both are then ranked on the remaining criteria of

fidelity (§ 7.2). In a second supplementary variant, the analysis is repeated without the seam metrics that are undefined for some method, so that only the rate of change points (F5.5F_{5.5}) remains and the seam criterion leaves the core rankings for every method. The overall order hardly changes (Kendall’s τ=0.94\tau= 0.94 with the published leaderboard in the main run, 0.920.92 in the replication run); in each run one borderline method joins the top tier (vae_atsp in the main run, vae_greedy in the replication run). The largest moves are zedge_ts_ar* from place 7 to 3 and Chronos from 20 to 22 in the main run and Chronos from 10 to 12 in the replication run; TimesFM and Timestral keep their places, and no verdict of C1–C4 changes. The outputs of both variants are in the supplementary outputs (seam_check, common_metrics).

Memorisation diagnostics

F10F_{10} cuts windows of length TT from the synthetic series, the real val series and train, z-scored with val, and takes for every synthetic and every val window the distance to the closest training window (DCR). F10.1F_{10.1} is the absolute logarithm of the ratio of the median DCR of the synthetic windows to that of the val windows; F10.2F_{10.2} is the share of synthetic windows whose DCR is below the 5th percentile of the val windows’ DCR (near copies). The supplementary check (memorisation_check) recomputes the same windows for every series and for the real OOT series, keeps the sign of the logarithm (below 0: closer to train than the val windows), counts the exact copies (DCR below 0.05 in z units; the training windows are cut with step T/4T/4, so a copy that is not aligned with this grid is not counted) and gives medians over datasets of the mean over seeds with two-level bootstrap intervals (datasets, then seeds). The recomputed values equal the published ones (largest difference 3×10−43 \times10^{-4} for F10.1F_{10.1}; F10.2F_{10.2} differs in one series, by 0.009). The same windows are scored for controls on every dataset and seed: the first points of train (every window an exact copy), the same with Gaussian noise of 0.05 to 0.5 standard deviations of train, and the real OOT series with its first tenth, quarter or half replaced by train.

The near-copy flag of the analysis compares the share of near copies of a method with that of the real OOT series: it is raised when the median ratio over datasets is at least 2 and the excess is significant (one-sided Wilcoxon over datasets, Holm). The share of the real series is 1 on three datasets (bike, electricity, metro: its windows are closer to train than almost all val windows), 0 on three (Beijing PM2.5, bitcoin, household) and 0.13 to 0.57 on the others. For an exact copy of train the ratio is then 1 or undefined on six of the nine datasets, and its median is 1.38 (Holm p=0.125p = 0.125); no control is flagged. The signed ratio and the exact copies separate the controls (see Table 17). Copies with noise up to 0.25 standard deviations lie far below every method, whose windows are never exact copies; noise of 0.5 standard deviations and partial copies are within the spread of the methods and of the real series. The replication run gives the same picture.

MethodLog ratio95 % intervalNear copies95 % intervalExact copies
timestral−0.31-0.31[−0.74-0.74, −0.08-0.08]0.610.61[0.000.00, 1.001.00]0 / 10 530
timesfm−0.19-0.19[−0.50-0.50, −0.05-0.05]0.500.50[0.090.09, 0.860.86]0 / 10 530
diffusion_ts−0.18-0.18[−0.47-0.47, 0.170.17]0.270.27[0.050.05, 0.690.69]0 / 10 530
vae_pctsp−0.17-0.17[−0.55-0.55, 0.680.68]0.220.22[0.000.00, 0.750.75]0 / 10 530
vae_atsp−0.15-0.15[−0.58-0.58, 0.670.67]0.240.24[0.000.00, 0.770.77]0 / 10 530
vae_zseries_xfade*−0.12-0.12[−0.23-0.23, 0.780.78]0.120.12[0.000.00, 0.400.40]0 / 10 530
vae_zseries_wsola*−0.12-0.12[−0.34-0.34, 0.590.59]0.350.35[0.000.00, 0.760.76]0 / 10 530
vae_quilting−0.12-0.12[−0.57-0.57, 0.650.65]0.210.21[0.000.00, 0.730.73]0 / 10 530
vae_zseries_quilt*−0.11-0.11[−0.23-0.23, 0.780.78]0.110.11[0.000.00, 0.380.38]0 / 10 530
vae_zseries*−0.11-0.11[−0.59-0.59, 0.580.58]0.320.32[0.000.00, 0.790.79]0 / 10 530
vae_knn−0.08-0.08[−0.48-0.48, 0.720.72]0.190.19[0.000.00, 0.730.73]0 / 10 530
vae_greedy−0.07-0.07[−0.55-0.55, 0.630.63]0.210.21[0.000.00, 0.780.78]0 / 10 530
vae_rand−0.05-0.05[−0.35-0.35, 0.760.76]0.150.15[0.000.00, 0.550.55]0 / 10 530
vae_mbb−0.04-0.04[−0.37-0.37, 0.770.77]0.150.15[0.000.00, 0.500.50]0 / 10 530
vae_herding−0.02-0.02[−0.30-0.30, 0.520.52]0.090.09[0.010.01, 0.600.60]0 / 10 530
zedge_quilting*−0.02-0.02[−0.80-0.80, 0.220.22]0.250.25[0.020.02, 0.760.76]0 / 10 530
zedge_atsp*0.000.00[−0.94-0.94, 0.200.20]0.360.36[0.020.02, 0.860.86]0 / 10 530
zedge_pctsp*0.000.00[−0.95-0.95, 0.210.21]0.340.34[0.020.02, 0.840.84]0 / 10 530
zedge_zseries*0.010.01[−0.89-0.89, 0.250.25]0.260.26[0.030.03, 0.900.90]0 / 10 530
zedge_zseries_wsola*0.010.01[−0.82-0.82, 0.260.26]0.320.32[0.010.01, 0.890.89]0 / 10 530
zedge_zseries_quilt*0.020.02[−0.42-0.42, 0.360.36]0.050.05[0.020.02, 0.440.44]0 / 10 530
zedge_zseries_xfade*0.020.02[−0.43-0.43, 0.350.35]0.050.05[0.020.02, 0.440.44]0 / 10 530
zedge_knn*0.020.02[−0.72-0.72, 0.220.22]0.330.33[0.010.01, 0.890.89]0 / 10 530
zedge_greedy*0.030.03[−0.89-0.89, 0.220.22]0.340.34[0.020.02, 0.880.88]0 / 10 530
zedge_ts_ar*0.070.07[−0.43-0.43, 0.250.25]0.160.16[0.040.04, 0.770.77]0 / 10 530
zedge_ts_long*0.080.08[−0.37-0.37, 0.230.23]0.170.17[0.050.05, 0.770.77]0 / 10 530
zedge_herding*0.110.11[−0.52-0.52, 0.360.36]0.210.21[0.010.01, 0.660.66]0 / 10 530
zedge_mbb*0.120.12[−0.51-0.51, 0.370.37]0.210.21[0.010.01, 0.610.61]0 / 10 530
zedge_rand*0.140.14[−0.47-0.47, 0.380.38]0.190.19[0.020.02, 0.600.60]0 / 10 530
chronos0.280.28[−0.24-0.24, 1.261.26]0.230.23[0.010.01, 0.460.46]0 / 10 530
real OOT series−0.13-0.13–0.130.13–0 / 442
copy−∞-\infty–1.001.00[1.001.00, 1.001.00]10 300 / 10 300
copy + noise 0.05−2.67-2.67[−3.15-3.15, −2.12-2.12]1.001.00[1.001.00, 1.001.00]0 / 10 300
copy + noise 0.1−1.98-1.98[−2.46-2.46, −1.43-1.43]1.001.00[1.001.00, 1.001.00]0 / 10 300
copy + noise 0.25−1.06-1.06[−1.20-1.20, −0.51-0.51]1.001.00[1.001.00, 1.001.00]0 / 10 300
copy + noise 0.5−0.37-0.37[−0.85-0.85, 0.180.18]0.500.50[0.000.00, 1.001.00]0 / 10 300
partial copy 0.1−0.13-0.13[−0.63-0.63, 0.050.05]0.120.12[0.000.00, 0.950.95]0 / 4 419
partial copy 0.25−0.15-0.15[−0.62-0.62, 0.080.08]0.170.17[0.020.02, 0.840.84]0 / 4 403
partial copy 0.5−0.30-0.30[−0.55-0.55, 0.060.06]0.380.38[0.140.14, 0.810.81]212 / 4 509

Table 17. Memorisation diagnostics, main run: signed log ratio of the median DCR (synthetic against val windows; below 0: closer to train), share of near copies, each the median over datasets with a 95 % bootstrap interval, and exact copies among all windows. The real OOT series is scored once per dataset; the controls are described in the text (copy: DCR = 0). Supplementary. * proprietary (zypl.ai).

Noise-aware ranks

Within a block (dataset, seed and leaf) with kk methods and the noise ϵ\epsilon of the leaf on that dataset, the rank of method ii is 1+#{j:sj1 + \#\{j : s_j beats sis_i by more than ϵ}+12#{j≠i:∣sj−si∣≤ϵ}\epsilon\} + \frac{1}{2}\#\{j \ne i : |s_j - s_i| \le\epsilon\} (see § 7.2). Table 18 works through three methods whose ties are not transitive. The ranks do not depend on the order in which the methods are listed; this is checked by a test of the released code, and on the published tables, where renaming the methods so that they come in another order leaves every rank unchanged. In the main run, 28.5 % of the pairs of methods compared within a block tie, and a further 11.7 % do not tie but are linked by a method that ties with both (26.9 % and 10.3 % in the replication run). Table 19 lists the methods whose place or top-tier membership differs with ordinary average ranks. The contrasts of the research questions and of C1–C4 use paired differences of the values, not ranks, and do not depend on this choice.

MethodScoreNoise-aware rankOrdinary rank
A1.21+0+12⋅1=1.51 + 0 + \frac{1}{2} \cdot1 = 1.51
B0.61+0+12⋅2=21 + 0 + \frac{1}{2} \cdot2 = 22
C0.01+1+12⋅1=2.51 + 1 + \frac{1}{2} \cdot1 = 2.53

Table 18. Noise-aware and ordinary ranks of three methods in one block, noise e=1e = 1, higher is better: A ties B and B ties C, but A beats C. The ranks sum to k(k+1)/2=6k(k + 1)/2 = 6 in both cases.

RunMethodNoise-aware PlaceNoise-aware Top tierOrdinary PlaceOrdinary Top tier
Mainzedge_knn5yes8yes
zedge_ts_ar7yes5yes
zedge_ts_long8yes7yes
zedge_rand11yes11no
Replicationvae_mbb8yes9yes
vae_pctsp9yes8yes
vae_knn11yes12yes
vae_greedy12no11no

Table 19. Methods whose place or top-tier membership differs between the noise-aware ranks of the leaderboard and ordinary average ranks; every other method keeps its place and tier. Kendall’s τ\tau between the mean ranks: 0.97 (main run) and 0.95 (replication run).

Table 18: Noise-aware and ordinary ranks of three methods in one block, noise e=1e = 1, higher is better: A ties B and B ties C, but A beats C. The ranks sum to k(k+1)/2=6k(k + 1)/2 = 6 in both cases.

Table 19: Methods whose place or top-tier membership differs between the noise-aware ranks of the leaderboard and ordinary average ranks; every other method keeps its place and tier. Kendall’s τ\tau between the mean ranks: 0.97 (main run) and 0.95 (replication run).

Top tier and the selection of the leader

Table 20 gives, for both runs, the number of datasets on which each method is paired with the observed leader, the Holm-corrected pp-value of the one-sided Wilcoxon test that it is worse, and its membership of the top tier. Every method also belongs to the set of methods that no other method is shown to beat when every ordered pair is tested with Holm correction over all pairs (552 pairs in the main run, 156 in the replication run): with nine datasets, the smallest one-sided pp is 1/5121/512, and no pair is rejected. Both sets are non-rejection sets (§ 7.2).

Main runReplication run
MethodDatasetspHolmp_{\mathrm{Holm}}TierMethodDatasetspHolmp_{\mathrm{Holm}}Tier
zedge_quilting*9–yesvae_herding9–yes
zedge_zseries*90.410yesdiffusion_ts90.500yes
zedge_greedy*90.410yesvae_quilting90.500yes
zedge_pctsp*90.293yesvae_zseries*90.500yes
zedge_knn*90.410yestimesfm60.500yes
zedge_atsp*90.410yesvae_atsp90.088yes
zedge_ts_ar*90.410yesvae_rand90.088yes
zedge_ts_long*90.191yesvae_mbb90.088yes
zedge_herding*90.107yesvae_pctsp90.088yes
zedge_mbb*90.123yeschronos70.391yes
zedge_rand*90.107yesvae_knn90.059yes
vae_herding90.045novae_greedy90.047no
vae_zseries*90.055yestimestral90.047no
diffusion_ts90.045no
timesfm60.125yes
vae_quilting90.055yes
vae_pctsp90.045no
vae_atsp90.045no
vae_greedy90.045no
chronos70.094yes
vae_knn90.045no
vae_mbb90.045no
vae_rand90.045no
timestral90.045no

Table 20. Comparisons with the observed leader of each run: paired datasets, Holm-corrected pp that the method is worse, and top-tier membership. Leader: zedge_quilting (main run) and vae_herding (replication run).

Software

The figures of this appendix give the details of § 8: the architecture of the code (Figure 12), the software environments (Figure 13), the fingerprints of incremental recomputation (Figure 14), the layout of the outputs (Figure 15), the run profiles (fig. 16) and the extension points (fig. 17`).

Figure 12. Layered architecture of the benchmark. The bench package runs the pipeline stages and finds every component by name in a registry. The official Diffusion-TS code sits underneath it. All stages exchange data only through Parquet files in data/ and artifacts/. The hypotheses package reads the collected tables and tests the hypotheses.

Runs and reproduction

Table 21 describes the two runs of § 9.1. Both used profile min, the environments pinned in the repository and the same task thresholds and seeds; every generation record stores the GPU on which it ran, and each run used a single GPU type. The software versions of the replication run were not recorded beyond the pinned requirements.

artifacts/<profile>/ ├── pools/<ds>/<gen>/seed<k>.parquet pool windows (N, T, 3) + record ├── generators/<ds>/<gen>/…parquet generator weights (VAE, Diffusion-TS, SDV) ├── series/<ds>/<method>/seed<k>.parquet t, value, window, piece, seam ├── records/<ds>/<method>/seed<k>.parquet time, memory, hardware, diagnostics ├── fidelity/<ds>/<method>/seed<k>.parquet all F-metrics ├── labels/<ds>/<task>.parquet real test labels: split, row, y ├── utility/<ds>/<method>/seed<k>/ │ ├── metrics.parquet U-metrics per task × model × regime × split │ ├── predictions.parquet every prediction (scalar or vector) │ └── models.parquet weights of every fitted model ├── tables/ timing, fidelity, utility, pools, cross_seed, datasets, inventory, hardware, config.json ├── reports/ Markdown reports of the analysis: │ ├── REPORT.md RQ0–RQ9, C1–C4 │ ├── OVERALL.md overall leaderboard, robustness checks │ ├── LEADERBOARD.md ranks of every method per slice │ └── METRIC_VALUES.md metric values by reference, dataset, task, model, split ├── overall/ overall leaderboard tables, robustness checks ├── leaderboard/ rankings.parquet, ranks per slice, tests, stability ├── metric_values/ fidelity, utility: median and IQR over seeds ├── hypotheses/ RQ* global / per dataset, confirmatory └── errors.log failed units (run continues)

Parquet, zstd; float32 except series values (float64). Every unit file carries its unit_fp.

References

  1. [1]Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
  2. [2]Xinyu Yuan and Yan Qiao. Diffusion-TS: Interpretable diffusion for general time series generation. In International Conference on Learning Representations, 2024. arXiv:2403.01742.arxiv.org/abs/2403.01742
  3. [3]Andrew J. Hunt and Alan W. Black. Unit selection in a concatenative speech synthesis system using a large speech database. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1996. doi:10.1109/ICASSP.1996.541110.
  4. [4]Arno Schödl, Richard Szeliski, David H. Salesin, and Irfan Essa. Video textures. In Proceedings of SIGGRAPH, 2000. doi:10.1145/344779.345012.
  5. [5]Alexei A. Efros and William T. Freeman. Image quilting for texture synthesis and transfer. In Proceedings of SIGGRAPH, 2001. doi:10.1145/383259.383296.DOI
  6. [6]Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024. arXiv:2403.07815.arxiv.org/abs/2403.07815
  7. [7]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In International Conference on Machine Learning, 2024. arXiv:2310.10688.arxiv.org/abs/2310.10688
  8. [8]Adrien Petralia et al. Timestral, 2026. arXiv:2610.37585.arxiv.org/abs/2610.37585
  9. [9]Guoxiong Long, Huizhen Huang, Qikun Cai, Tao Huang, and Chen Hou. Evaluating sequence assembly strategies for differentially private synthetic time-series forecasting, 2026. arXiv:2610.10222.arxiv.org/abs/2610.10222
  10. [10]Yihao Ang, Qiang Huang, Yifan Bao, Anthony K. H. Tung, and Zhiyong Huang. TSGBench: Time series generation benchmark. Proceedings of the VLDB Endowment, 17(3), 2023. arXiv:2309.03755.arxiv.org/abs/2309.03755
  11. [11]Alexander Nikitin, Letizia Iannucci, and Samuel Kaski. TSGM: A flexible framework for generative modeling of synthetic time series. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024. arXiv:2305.11567.arxiv.org/abs/2305.11567
  12. [12]Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
  13. [13]Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems, volume 34, 2021. arXiv:2107.03502.arxiv.org/abs/2107.03502
  14. [14]Paul Jeha, Michael Bohlke-Schneider, Pedro Mercado, Shubham Kapoor, Rajbir Singh Nirwan, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. PSA-GAN: Progressive self attention GANs for synthetic time series. In International Conference on Learning Representations, 2022. arXiv:2108.00981.arxiv.org/abs/2108.00981
  15. [15]Zhe Li, Xiangfei Qiu, Peng Chen, Yihang Wang, Hanyin Cheng, Yang Shu, Jilin Hu, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, and Bin Yang. TSFM-Bench: A comprehensive and unified benchmark of foundation models for time series forecasting. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2025. doi:10.1145/3711896.3737442. arXiv:2410.11802.DOI
  16. [16]Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723–773, 2012.DOI
  17. [17]David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Conference on Learning Representations, 2017. arXiv:1610.06545.
  18. [18]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two-time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017. arXiv:1706.08500.
  19. [19]Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards universal representation of time series. In AAAI Conference on Artificial Intelligence, 2022. arXiv:2106.10466.
  20. [20]Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, 2020. arXiv:2002.09797.
  21. [21]Carl H. Lubba, Sarab S. Sethi, Philip Knaute, Simon R. Schultz, Ben D. Fulcher, and Nick S. Jones. catch22: Canonical time-series Characteristics. Data Mining and Knowledge Discovery, 33:1821–1852, 2019. doi:10.1007/s10618-019-00647-x.DOI
  22. [22]Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional GANs, 2017. arXiv:1706.02633.
  23. [23]Shaocheng Lan, Shuqi Gu, Zhangzhi Xiong, and Kan Ren. ConTSG-Bench: A unified benchmark for conditional time series generation, 2026. arXiv:2603.04767.
  24. [24]Yihao Ang, Qiang Wang, Qiang Huang, Yifan Bao, Xinyu Xi, Anthony K. H. Tung, Chen Jin, and Zhiyong Huang. CTBench: Cryptocurrency time series generation benchmark, 2025. arXiv:2508.02758.arxiv.org/abs/2508.02758
  25. [26]Kiwan Kwon, Kangmin Kim, Hojin Lee, Yeseong Jung, Hyeongwoo Kong, Vamsi K. Potluru, Saerom Park, and Yongjae Lee. Seq2Synth: Benchmarking temporal fidelity in synthetic sequential tabular data. In Proceedings of the ACM International Conference on Information and Knowledge Management, 2026. doi:10.1145/3799682.3841108. arXiv:2607.15606.
  26. [27]Lucas Kovar, Michael Gleicher, and Frédéric Pighin. Motion graphs. In Proceedings of SIGGRAPH, 2002. doi:10.1145/566654.566605.DOI
  27. [28]Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. Graphcut textures: Image and video synthesis using graph cuts. In Proceedings of SIGGRAPH, 2003. doi:10.1145/882262.882264.
  28. [29]Werner Verhelst and Marc Roelands. An overlap-add technique based on waveform similarity (WSOLA) for high quality time-scale modification of speech. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1993.
  29. [30]Jont B. Allen and Lawrence R. Rabiner. A unified approach to short-time Fourier analysis and synthesis. Proceedings of the IEEE, 65(11):1558–1564, 1977. doi:10.1109/PROC.1977.10770.DOI
  30. [31]Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting using denoising diffusion probabilistic models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. arXiv:2201.09865.
  31. [32]Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusion paths for controlled image generation. In International Conference on Machine Learning, 2023. arXiv:2302.08113.
  32. [33]Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming-Yu Liu. DiffCollage: Parallel generation of large content with diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. arXiv:2303.17076.arxiv.org/abs/2303.17076
  33. [34]Hans R. Künsch. The jackknife and the bootstrap for general stationary observations. The Annals of Statistics, 17(3), 1989. doi:10.1214/aos/1176347265.DOI
  34. [35]Dimitris N. Politis and Joseph P. Romano. A circular block-resampling procedure for stationary data. In Raoul LePage and Lynne Billard, editors, Exploring the Limits of Bootstrap. Wiley, 1992.
  35. [36]Dimitris N. Politis and Joseph P. Romano. The stationary bootstrap. Journal of the American Statistical Association, 89 (428), 1994. doi:10.1080/01621459.1994.10476870.DOI
  36. [37]Dimitris N. Politis and Halbert White. Automatic block-length selection for the dependent bootstrap. Econometric Reviews, 23(1), 2004. doi:10.1081/ETC-120028836.DOI
  37. [38]Anna E. Dudek, Jacek Leskow, Efstathios Paparoditis, and Dimitris N. Politis. A generalized block bootstrap for seasonal time series. Journal of Time Series Analysis, 35(2), 2014.DOI
  38. [39]Peter Bühlmann. Sieve bootstrap for time series. Bernoulli, 3(2), 1997.DOI
  39. [40]M. B. Rajarshi. Bootstrap in Markov-sequences based on estimates of transition density. Annals of the Institute of Statistical Mathematics, 42, 1990. doi:10.1007/BF00050835.
  40. [41]James Theiler, Stephen Eubank, André Longtin, Bryan Galdrikian, and J. Doyne Farmer. Testing for nonlinearity in time series: The method of surrogate data. Physica D, 58, 1992. doi:10.1016/0167-2789(92)90102-S.
  41. [42]Thomas Schreiber and Andreas Schmitz. Improved surrogate data for nonlinearity tests. Physical Review Letters, 77, 1996. doi:10.1103/PhysRevLett.77.635.DOI
  42. [43]Edward Carlstein, Kim-Anh Do, Peter Hall, Tim Hesterberg, and Hans R. Künsch. Matched-block bootstrap for dependent data. Bernoulli, 4(3):305–328, 1998.
  43. [44]Upmanu Lall and Ashish Sharma. A nearest neighbor bootstrap for resampling hydrologic time series. Water Resources Research, 32(3):679–693, 1996. doi:10.1029/95WR02966.
  44. [45]Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems, volume 33, pages 19667–19679, 2020.arxiv.org/abs/2007.03898
  45. [46]Keld Helsgaun. An effective implementation of the Lin–Kernighan traveling salesman heuristic. European Journal of Operational Research, 126(1):106–130, 2000. doi:10.1016/S0377-2217(99)00284-2.DOI
  46. [47]Egon Balas. The prize collecting traveling salesman problem. Networks, 19(6):621–636, 1989. doi:10.1002/net.3230190602.DOI
  47. [48]Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Conference on Uncertainty in Artificial Intelligence (UAI), 2010.
  48. [49]Daniël Lakens. Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4):355–362, 2017. doi:10.1177/1948550617697177.
  49. [50]Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–83, 1945. doi:10.2307/3001968.DOI
  50. [51]Donald J. Schuirmann. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15(6):657–680, 1987. doi:10.1007/BF01068419.
  51. [52]Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979.DOI
  52. [53]Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995. doi:10.1111/j.2517-6161.1995.tb02031.x.DOI
  53. [54]Milton Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701, 1937. doi:10.1080/01621459.1937.10503522.
  54. [55]Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should we really use post-hoc tests based on mean-ranks? Journal of Machine Learning Research, 17(5):1–10, 2016.
  55. [56]Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7: 1–30, 2006.DOI
  56. [66]Mistral AI. Introducing Timestral. https://mistral.ai/news/timestral, October 2026. Blog post, 15 October 2026. Code and weights: https://github.com/mistralai/timestral. Accessed 26 October 2026.
  57. [67]John H. Skillings and Gregory A. Mack. On the use of a Friedman-type statistic in balanced and unbalanced block designs. Technometrics, 23(2):171–177, 1981.DOI

Paper details

vlnc:W_p89pknsc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude

View statement
Looking for related papers…
Contents