betaFeedback

WorksComputer Science

Beyond Static Leaderboards: Capability-Focused Evaluation with Structural Diversity for Time Series Foundation Models

Abstract

Static public benchmarks for time series foundation models (TSFMs) typically have fixed and narrow test distributions. Their repeated reuse encourages benchmark-specific tuning, making reported performance reflect adaptation to the fixed test set rather than genuine forecasting generalization. In addition, a single aggregate score cannot fully characterize model capabilities across distinct temporal structures and input relationships. We propose CaFE (Capability-Focused Evaluation), a refreshable evaluation extension grounded in original benchmark data that enriches static benchmarks with structural diversity and enriches single-score black-box evaluation with multi-dimensional, capability-oriented assessment. CaFE first profiles the original benchmark and defines eight Structural Features (SFs) that characterize its temporal structure. For each SF, it resamples the original data according to its structural support, removes edge-of-distribution samples, and augments the retained samples by varying the SF's salience, producing samples with different structural profiles. After filtering out out-of-distribution samples while preserving diversity, CaFE yields a refreshed evaluation set. TSFMs are then evaluated on this set using both conventional forecast-error metrics and intervention-response metrics, giving each model a per-feature capability profile alongside endpoint accuracy. Experiments on GIFT-Eval and FEV-Mini20 show that CaFE yields stable evaluation results and reveals per-feature strengths and weaknesses that a single aggregate score hides. Moreover, on genuine data filtered by structural condition, models selected by CaFE outperform those chosen by the aggregate leaderboard. CaFE therefore makes model selection conditional on the structural response of interest.

Introduction

Time series foundation models (TSFMs), pretrained on large and diverse data corpora, are increasingly used for forecasting across domains such as energy, transportation, finance, and cloud operations [3, 5] (Woo et al. 2024). As more TSFMs become publicly available, selecting the right model for a specific forecasting task has become a critical practical challenge. Existing benchmarks evaluate TSFMs on widely reused datasets such as ETT, Electricity, and Traffic [16, 1, 15] (Zhou et al. 2021; Trindade 2015), typically summarizing performance with a single aggregate score. These leaderboards rank models by overall performance and have become the de facto standard for TSFM comparison.

Fixed public test sets create two intertwined problems. First, their repeated reuse conflates forecasting ability with familiarity with the specific test content, and their narrow structural coverage leaves structural diversity underrepresented. Second, a single aggregate score collapses many distinct observed cases into one ranking. As a result, the model that attains the best endpoint accuracy may differ from the model that responds faithfully to a controlled structural change, and existing benchmarks offer no direct way to inspect that difference (Figure 1).

A static test set provides one finite snapshot of structural conditions, while a single aggregate score obscures how forecasts respond across those conditions.

Figure 1. A static test set provides one finite snapshot of structural conditions, while a single aggregate score obscures how forecasts respond across those conditions.

To address these two problems, we present CaFE (Capability-Focused Evaluation), a refreshable evaluation extension grounded in original benchmark data. CaFE enriches static benchmarks with structural diversity and enriches single-score evaluation with multi-dimensional, capability-oriented assessment. Concretely, it first profiles the original benchmark and defines eight Structural Features (SFs) spanning intrinsic and relational structure. Table 2 summarizes the eight SFs and their controlled changes. Guided by these SFs, it resamples the original data along specific structural dimensions and removes edge-of-distribution samples to establish controlled structural conditions. It then augments the resampled data by varying the salience of each SF, producing samples with different structural profiles, and filters out out-of-distribution augmented samples while preserving diversity, yielding a refreshed evaluation set. By spanning a wider range of structural conditions than the fixed test set, this refreshed set lowers the risk that reported performance reflects adaptation to a narrow distribution rather than genuine forecasting generalization.

Structural featureControlled changeControl parameterApplicability
SF1 TrendPersistent directional continuationNormalized trend magnitudeUni./multi.
SF2 Multi-SeasonalityConcurrent stable periodsNumber of active periodsUni./multi.
SF3 Time-Varying SeasonalityEvolving amplitude or phaseNormalized evolution magnitudeUni./multi.
SF4 Regime SwitchingBounded transition to a persistent stateTransition locationUni./multi.
SF5 IntermittencySparse recurring eventsMean event gapUni./multi.
SF6 Common FactorShared trajectory across targetsNormalized shared-profile magnitudeMulti., ≥3\ge3 targets
SF7 Cross-Series DependenceDirected driver–responder relationNormalized transfer magnitudeMulti., ≥2\ge2 targets
SF8 Covariate ResponseCovariate impulse with matched target responseJoint normalized response magnitudeNative dynamic covariate

Table 2. Structural features, controlled changes, control parameters, and input requirements. Full constructions appear in this appendix.

On the evaluation side, CaFE measures each model along two axes. The first retains conventional forecast-error metrics, capturing endpoint accuracy on the refreshed set, as in existing benchmarks. The second introduces intervention-response metrics: for each Structural Feature, the evaluator varies the feature’s salience and measures how the model’s forecast changes in response, yielding a per-feature capability profile. The profile reveals structural sensitivity that endpoint accuracy alone misses: a model can forecast accurately while ignoring cross-series relationships or covariate conditions. CaFE therefore reports endpoint accuracy alongside a per-feature capability profile, exposing condition-specific strengths and weaknesses that a single aggregate score cannot express. In summary, we make the following main contributions:

  • We propose CaFE, a refreshable evaluation extension grounded in original benchmark data that enriches static benchmarks with structural diversity and enriches single-score evaluation with multi-dimensional, capability-oriented assessment.

  • We introduce Structural Features (SFs) and, guided by them, resample and augment the original data under controlled structural conditions, filtering out out-of-distribution samples to yield a refreshed evaluation set. We then expose per-feature response profiles through intervention-response metrics alongside conventional forecast-error metrics, adding a multi-dimensional, capability-oriented axis to evaluation.

  • Across GIFT-Eval and FEV-Mini20, we show that CaFE’s per-feature response profiles expose condition-specific strengths and weaknesses a single score hides, that repeated regenerations preserve the overall ranking while refreshing local winners, that training recipes can trade endpoint accuracy against response fidelity, and that models selected by CaFE outperform the aggregate leaderboard on genuine data filtered by structural condition.

Related Work

Time series foundation model evaluation spans four broad lines. Fixed retrospective and TSFM suites, including M4/M5, TFB, BasicTS+, ProbTS, TSFM-Bench, GIFT-Eval, and fev-bench, provide data, pipelines, and standardized comparisons ([19, 20, 16, 1] (Qiu et al. 2024; Shao et al. 2025; Zhang et al. 2024; Schür et al. 2025)). Structureaware suites such as It’s TIME and QuitoBench organize observed patterns or balanced regimes, and live platforms such as TS-Arena and Impermanent use preregistration or rolling streams [21, 8] (Qiao et al. 2026; Xue et al. 2026). Synthetic-generation benchmarks such as GRATIS, TSGM, TimeSynth, and SynTSBench provide programmable mechanisms and behavior or fidelity diagnostics [10, 7, 13, 6, 2, 11] (Sun et al. 2026; Yoon et al. 2019; Yuan & Qiao 2024; Nikitin et al. 2024; Tan et al. 2025). These lines vary fixed collections, observed regimes, incoming streams, or generators; the fixed suites emphasize endpoint comparisons, leaving source-task anchoring and controlled paired changes as separate design choices.

For CaFE, four properties distinguish an evaluation design: whether it anchors to original benchmark instances, makes structure explicit, applies a controlled paired change, and is refreshable. Retrospective, TSFM, and covariate-aware suites anchor to original instances but keep structure implicit and apply no paired changes; observed-structure suites make structure explicit but do not refresh; live suites refresh streams but apply no paired changes; synthetic suites expose mechanisms but abandon the source task context. CaFE combines all four, applying one Structural Feature change to an original instance under screening and then reporting endpoint error on the References and generated samples together with intervention-response fidelity. Table 1 summarizes this positioning.

FamilyOriginal instancesExplicit structurePaired changeRefresh
Retrospective suites✓×××
TSFM suites✓×××
Covariate-aware suites✓×××
Observed-structure suites✓✓××
Live evaluation✓××✓
Synthetic mechanisms×✓××
CaFE (Ours)✓✓✓✓

Table 1. Benchmark families compared on four design properties: anchored to original instances, explicit structure, paired change, and refresh. CaFE is the only family that combines all four.

The CaFE Framework

CaFE addresses two failures of static benchmarks, a narrow structural distribution and a single aggregate score, by enriching a benchmark’s own data with structural diversity and its evaluation with capability-oriented assessment. Concretely, it proceeds in three generation stages followed by an evaluation stage (Figure 2). Profiling defines eight Structural Features (SFs) and maps each eligible source forecast window to an unchanged Reference. Augmentation resamples the References by structural support, removes edge-of-distribution windows, and, for each retained Reference, draws random salience coefficients to generate a family of samples spanning diverse structural conditions. Inspection and Sifting admits only complete families whose samples realize the intended change without drifting out of distribution. Finally, Evaluation scores each model along two axes.

The CaFE workflow

Figure 2. The CaFE workflow. Profiling maps source forecast windows to unchanged References; Augmentation generates, for each Reference, a family of samples at varying salience; Inspection and Sifting admit only complete families that realize the intended change; Evaluation scores each model on forecast-error and intervention-response metrics.

The first axis retains forecast-error metrics, capturing endpoint accuracy on the refreshed set. The second adds intervention-response metrics that measure how faithfully a forecast tracks each induced change. Because every generated sample differs from its Reference only through a known change in one SF, the same benchmark yields both an endpoint-accuracy profile and a per-SF response profile, so model selection can follow capability for each specific SF rather than a single aggregate score.

Profiling

Profiling anchors every intervention to the original forecasting task. It defines eight Structural Features (SFs) and maps each eligible source forecast window to an unchanged Reference RiR_i that preserves the targets, forecast origin and horizon, input visibility, and scoreable positions. Because the Reference is a real, unmodified forecasting example, every generated sample stays grounded in real benchmark data rather than synthetic data. A Structural Feature is a temporal pattern or relationship whose salience can be varied in a controlled way to probe a specific forecasting capability. CaFE groups the eight features into two classes: intrinsic structure, which operates within a single target series, and relational structure, which operates across target series or between covariates and targets. Figure 3 summarizes the eight features, and Appendix § 10 provides details.

The Structural Features of CaFE

Figure 3. The Structural Features of CaFE: intrinsic structure for within-series patterns and relational structure for cross-series and covariate dependencies.

Intrinsic Structure. Intrinsic structure comprises five features (SF1–SF5) spanning three forms of within-series variation. Trend and Seasonal Patterns (SF1–SF2) vary in persistent direction and concurrent periodicity, testing whether forecasts extrapolate long-term structure and combine multiple temporal scales. Nonstationary Dynamics (SF3–SF4) vary how seasonal behavior and temporal regimes evolve, testing whether forecasts track changes over time. Intermittent Event Dynamics (SF5) varies the regularity of sparse recurring events, testing whether forecasts recognize predictable structure when observations are mostly inactive. Moving from persistent patterns to evolving dynamics and sparse events, these interventions distinguish different forms of temporal generalization while retaining the same task anchor.

Relational Structure. Relational structure comprises three features (SF6–SF8) spanning shared and directed dependencies. Shared Cross-Series Structure (SF6) varies a temporal component common to multiple target series, testing whether forecasts recognize movement shared across targets. Directed Predictive Relations (SF7–SF8) vary a lagged relation from a driver to a responder or a target’s response to a dynamic covariate, testing whether forecasts use signals from other series or covariates. Together, these features reveal whether a forecast uses information shared across targets or directed information from a specific series or covariate, while the series involved and the forecast task remain fixed. Because SF6–SF8 expose auxiliary inputs, CaFE additionally reports input utility by removing each input and measuring the resulting change in forecast error.

Augmentation

Augmentation turns each Reference into a family of generated samples that differ from it only in a single Structural Feature. It first resamples the References according to their structural support for each SF, removing edge-of-distribution windows so that the refreshed set stays within the benchmark’s typical structural range. For each retained Reference and selected SF, it then draws random salience coefficients, each controlling how strongly the feature is expressed, and generates one sample per coefficient. Because the coefficients are drawn over declared ranges, the resulting families span a wider and more diverse set of structural conditions than the fixed test set.

For each Reference RR and Structural Feature cc, Augmentation draws a random salience coefficient λ\lambda and generates a sample V=Tc(R,λ)V = T_c(R, \lambda). All samples generated from the same Reference and feature differ only in λ\lambda, so the family spans a range of salience levels while the forecast origin, horizon, target order, input visibility, and scoreable positions remain fixed. Each Structural Feature specifies its own transformation TcT_c, which injects the designated pattern or relation into the Reference while leaving all other structure untouched. The concrete construction of each feature and the range from which λ\lambda is drawn are given in Appendix § 10; for the present purpose, every generated sample is a controlled perturbation of its Reference with a known salience coefficient.

Together, the eight features and their randomly drawn salience coefficients give the refreshed set a structural coverage far wider than the fixed test set, so that reported accuracy reflects generalization across structure rather than familiarity with a narrow distribution. The generated families are then screened by Inspection and Sifting before they enter the evaluation set.

Inspection and Sifting

Augmentation produces candidate families, and Inspection and Sifting determine whether they can serve as evaluation data. Inspection works on each generated sample, verifying that the intended structural change is present, that non-target structure stays within tolerance, and that the sample remains valid and scoreable. Sifting works on the family linked to one Reference, requiring the family to be complete, checking that all members share the same Reference and configuration, and confirming that the family stays within the source distribution while preserving the structural diversity introduced by augmentation.

A family is admitted into the Refreshed Benchmark only when every member passes Inspection, and the family as a whole passes Sifting. Rejected or incomplete families receive an explicit reason and do not contribute to the evaluation, while their References remain available for the original task. The checks cover basic validity, realization of the intended structural change, non-target tolerance, source distance, scoreable support, distribution difference, and diversity. See Appendix § 11 for details.

Evaluation

CaFE reports two metric classes. Forecast-Error Metrics measure endpoint accuracy on the References and the generated samples, using Reference MASE and Probe MASE. Intervention-Response Metrics measure whether forecasts respond to the structural change introduced by augmentation along each SF, using Forecast-Change NRMSE, the normalized root-mean-square error between the induced target change and the model’s forecast change, with the Auxiliary-Input Gap (ΔMASE) reported additionally for SF6–SF8. Together, these metrics characterize endpoint accuracy, structural response, and auxiliary-input utility.

For a given model and Structural Feature, Forecast-Change NRMSE measures how closely the model’s forecast change tracks the target change induced by varying the feature’s salience. Let Δy\Delta y and Δy^\Delta\hat{y} denote the target change and the forecast change between a generated sample and its Reference on the affected positions. Forecast-Change NRMSE is

NRMSE⁡=∑r=1∣Δy∣(Δy^r−Δyr)2∑r=1∣Δy∣(Δyr)2(1)\operatorname{NRMSE} = \sqrt{\frac{\sum_{r=1}^{\lvert\Delta y\rvert}(\Delta\hat{y}_r-\Delta y_r)^2}{\sum_{r=1}^{\lvert\Delta y\rvert}(\Delta y_r)^2}} \tag*{(1)}

where rr indexes the affected positions. A value of 0 indicates an exact response, 1 indicates no forecast response, and lower values mean closer agreement with the induced change. Eligibility and aggregation details are given in Appendix § 11.

The Auxiliary-Input Gap measures the change in MASE after removing the designated auxiliary input, with positive values indicating that the full input helps. The two metric classes keep their own rankings: Forecast-Error Metrics summarize accuracy over References and generated samples, while Intervention-Response Metrics remain within each SF, whose controls, scales, and scoreable populations differ. Any cross-feature summary is a secondary equal-weight macro over direction-corrected feature-local ranks and does not pool raw values or define a cross-feature capability scale.

Experiments

The experiments establish CaFE as an evaluation framework for time series foundation models. We first apply CaFE to GIFT-Eval (Short, Medium, and Long horizons) and fev-bench Mini-20, reporting endpoint accuracy and response fidelity for each model. On genuine data filtered by intrinsic structure, models selected by CaFE outperform those chosen by the aggregate leaderboard. We then validate the relational features SF6–SF8, verify that the augmentation is faithful, and check that the rankings are stable under regeneration. Finally, we show that the two metrics trade off under a changed training objective.

Experimental Setup

We evaluate CaFE on three horizon subsets of GIFT-Eval (Aksu et al. [1]) (Short, Medium, and Long) and the Mini-20 subset of fev-bench (Shchur et al. (2025)), using one fixed augmentation realization per suite. The comparison roster contains seven foundation models, Chronos-2 (Ansari et al. [4]), Timer-3.5 (Liu et al. [18]), TimesFM 2.5 (Google Research [9]), TimesFM 3.0 (Das et al. [5]), TiREx-2 (Podest et al. (2026)), Moirai 2 (Liu et al. [17]), and Toto 2.0 (Khwaja et al. [14]), plus four simple statistical baselines (Drift, Naive, Seasonal Naive, and Linear Trend). TiREx-2 is reported on GIFT-Eval Short and fev-bench Mini-20 because its output-length limit leaves others unsupported. Each Structural Feature is controlled by a salience coefficient λ\lambda, instantiated as five ordered settings L1–L5 (Appendix § 10); we report Reference MASE and Probe MASE for endpoint accuracy, Forecast-Change NRMSE for response fidelity, and Auxiliary-Input Gap for input utility, with model support and aggregation details in Appendices § 11 and § 12.

Accuracy and Response Fidelity

CaFE reports, for each model and Structural Feature, endpoint accuracy (Probe MASE on the Refreshed Benchmark) and response fidelity (Forecast-Change NRMSE); Figure 4 and Figure 5 show the two views across the four suites, and Figure 6 condenses the fidelity view into a per-feature profile. Accuracy ranks models overall, while response fidelity reveals where each model’s response is strong or weak: a model can be accurate overall yet respond poorly to a structural change, such as a cross-series or covariate signal. The leaders differ by what is measured, with TimesFM 3.0 winning 5/8, 7/8, 5/8, and 7/8 feature tracks by Probe MASE but only 4/8, 6/8, 6/8, and 3/8 by Forecast-Change NRMSE, while Linear Trend has the lowest SF1 NRMSE despite weak endpoint accuracy.

Figure 4. Probe MASE rank by suite, model, and Structural Feature for the four suites in one horizontal strip. Within each feature, rank 1 is the most accurate model (lowest Probe MASE), and the underlying values are reported in Appendix C.

Figure 5. Forecast-Change NRMSE rank by suite, model, and Structural Feature for the four suites in one horizontal strip. Within each feature, rank 1 is the most faithful response to the induced change; NRMSE = 1 is the zero-response reference, and the underlying values are reported in Appendix C.

Figure 6. Feature-level Forecast-Change NRMSE values (lower is better) across four suites. Each SF uses an independent range over the six shared models, with larger values toward the center and smaller values on extended outer spokes; endpoint labels show both bounds.

At the aggregate level the two rankings are largely consistent: the Spearman correlation between Probe-MASE rank and NRMSE rank is 0.029, 0.800, 0.900, and 0.771 on the four suites, and the refreshed set does not distort accuracy rankings (Reference MASE and Probe MASE are rank-correlated at 0.943, 0.700, 0.900, and 0.657; Appendix § 12). The complementarity therefore lies in per-feature granularity rather than a reordering of the leaderboard: response fidelity flags the specific structural conditions under which a model’s endpoint accuracy does not transfer. The Linear Trend baseline exposes a further limit of the response metric: it attains the lowest SF1 NRMSE because a linear model trivially reproduces an injected linear trend, so the metric is most informative on the relational features (SF6–SF8), where matching the induced change requires modeling cross-series or covariate structure rather than reproducing a functional form.

Selecting Models by Intrinsic Structure

An aggregate leaderboard selects a single model for every scenario, yet the best model can depend on the target series’ structural condition. For each intrinsic Structural Feature (SF1–SF5), we gather genuine tasks with strong corresponding structure, hold out one source family at a time, and compare the three models ranked highest by aggregate Reference MASE with those ranked highest by feature-specific Probe MASE. Figure 7 shows that CaFE raises Top-3 coverage of the Reference-MASE-best model from 41.4% (12/29) to 72.4% (21/29). The gain concentrates on SF1 (37.5% → 62.5%), SF3

Figure 7. Top-3 coverage of the Reference-MASE-best model on genuine strong-structure tasks from held-out source families (higher is better). SF1–SF5 use 8, 4, 7, 7, and 3 eligible families, respectively.

(57.1% →\to 85.7%), and SF4 (42.9% →\to 85.7%), while SF2 (multi-seasonality) and SF5 (intermittency) stay flat, consistent with their few eligible families (4 and 3) and weaker feature-local signal.

Validation of Relational Features

The relational features SF6–SF8 test whether a model exploits auxiliary information: a shared latent factor across targets (SF6), a driver series (SF7), or a dynamic covariate (SF8). To validate these features, we remove the auxiliary input and measure how much the assessed target’s error changes, where a positive ΔMASE=MASEablated−MASEfull\Delta\mathrm{MASE} = \mathrm{MASE}_{\mathrm{ablated}} - \mathrm{MASE}_{\mathrm{full}} is the gain from the input. Figure 8 shows this gain against response fidelity: TimesFM 3.0 has the largest SF6 gap, Toto 2.0 the largest SF7 gap, and TimesFM 2.5 the largest SF8 gap, while several models lie near or below zero, marking models for which the auxiliary input acts as a distractor. The two axes are deliberately different: the Gap measures whether a model uses an input in the original task, whereas NRMSE measures whether a forecast responds to an injected change, so their lack of a clean monotone relationship is expected rather than a failure of either metric. The gains are small and model-specific, and the strongest input gain is not consistently held by the model with the lowest NRMSE, so the value of these inputs cannot be read from a single aggregate score. As a calibration, the simple baselines that cannot exploit cross-series or covariate structure sit near the zero-response reference on SF6–SF8, whereas the foundation mod-

Figure 8. Model-level response fidelity and input utility for SF6–SF8. Each point is an equal-weight model mean across datasets; lower NRMSE and positive Gap are preferred.

els that use these inputs score lower; this confirms that the response metric separates models that track the induced change from those that ignore it.

Figure 8 makes the two roles visible in one plot: the horizontal axis is response fidelity and the vertical axis is input utility. In the displayed means, TimesFM 3.0 has the largest SF6 gap, Toto 2.0 the largest SF7 gap, and TimesFM 2.5 the largest SF8 gap, while several other models lie near zero or below zero. The model with the strongest input gain is therefore not consistently the model with the lowest NRMSE, and the three SFs should be read on their own supports rather than as one difficulty scale. The task counts, descriptive intervals and details are reported in Appendix C.

Faithfulness of the Augmentation

A useful augmentation extends a benchmark without distorting model ordering; otherwise the structural signals CaFE adds would be noise rather than faithful evaluation. We verify this by ranking five small models on the References and the augmented Reference Families. Figure 9 shows the paired Reference and Augmented-Reference-Family MASE across 2,436 common source pairs and 21,793 valid augmented samples: agreement is Spearman ρ=0.90\rho= 0.90 and Kendall τb=0.80\tau_b = 0.80, with nine of ten pairwise orderings preserved, though heterogeneous across groups (ρ=1.00\rho= 1.00 multivariate, 0.30 univariate, −0.10-0.10 Trend; Appendix C.7). We therefore treat this as conditional evidence for paired rank stability on this suite, not universal rank preservation. The single exception is Trend (ρ=−0.10\rho= -0.10), the same phenomenon seen in Section 4.2: a linear intervention is nearly trivial to reproduce, so the Trend feature is the least discriminative and the least order-preserving; faithfulness is strongest where the paired change carries genuine relational information.

Figure 9. MASE on the References and on the Augmented Reference Families for five small models, ordered by Reference MASE. Points are three-seed means with ±1 sample SD; near-identical pairs within a model indicate that augmentation preserves that model’s accuracy ordering.

Stability under Benchmark Refresh

Because CaFE is refreshable, its conclusions should survive regeneration. Regenerating the GIFT-Eval Short benchmark five times, changing only the augmentation seed, leaves the overall ranking unchanged, with TimesFM 3.0 first on both metrics (Table 10); the scores vary by 0.26%–0.51% (Probe MASE) and 0.59%–1.22% (Forecast-Change NRMSE). Feature-level winners are less stable, the lowest-NRMSE model changing across seeds on SF3, SF6, and SF8. Overall ordering is thus robust to regeneration, while feature-level winners reveal where a comparison depends on the generated conditions. This feature-level sensitivity means the per-feature winners reported in § 4.2 are themselves seed-dependent; the robust finding is the stable overall ordering, while the specific per-feature leaders should be read as indicative rather than fixed.

Configuration–modelBC
M4 Hourly–Transformer1.83489.3486
M4 Hourly–DLinear3.03421.8950
M4 Hourly–PatchTST1.44682.1678
SZ Taxi–PatchTST0.65681.4263
SZ Taxi–TSMixer0.99360.6860

Table 10. Trend-treatment sensitivity retained in the small-model package. B is the paired original MASE and C is the valid treatment MASE; lower is better.

Tradeoff between Two Metrics

The two metrics also move in opposite directions when the training objective changes. Fine-tuning Chronos-2 on the same 11,269 CaFE-generated samples, official quantile-loss training lowers MASE by 13.46% but improves NRMSE by only 3.96%, whereas the Forecast-Change-NRMSE objective lowers NRMSE by 26.12% while raising MASE by 13.04% (Figure 11). The training objective therefore trades endpoint accuracy against response fidelity, so a practitioner can steer the model toward the capability their scenario needs. This is a single training realization. Optimizing Forecast-Change NRMSE teaches the model to track the injected changes, which trades away the endpoint fit on the refreshed distribution; the two objectives therefore move a single accuracy–fidelity operating point rather than demonstrating that one objective dominates the other.

Figure 11. Chronos-2 fine-tuned on 11,269 CaFE-generated samples under two objectives. The two panels track MASE and Forecast-Change NRMSE over 0–10k steps; official quantile-loss training improves MASE, while the Forecast-Change-NRMSE objective improves response fidelity at the cost of endpoint accuracy.

Five-seed GIFT-Eval Short stability table

Figure 10. Five-seed GIFT-Eval Short stability on the generated cohorts. Values are mean ± seed SD of Probe MASE and Forecast-Change NRMSE; ordering is unchanged across five seeds.

ModelProbe MASEForecast-Change NRMSE
TimesFM 3.01.21 ± 0.0030.580 ± 0.005
TiREx-21.20 ± 0.0030.637 ± 0.007
Chronos-21.22 ± 0.0040.646 ± 0.007
Toto 2.01.24 ± 0.0040.671 ± 0.004
TimesFM 2.51.25 ± 0.0040.684 ± 0.008
Timer-3.51.28 ± 0.0050.788 ± 0.009
Moirai 21.30 ± 0.0050.805 ± 0.010

Discussion

The novelty of CaFE is that a single, controlled, salience-varying perturbation of an unchanged Reference yields two measurements at once: endpoint accuracy and response fidelity. This paired Reference–sample design is what separates CaFE from fixed leaderboards, which report only endpoint accuracy, and from synthetic benchmarks, which replace the source task. Because every intervention stays anchored to a real instance, one refreshed benchmark produces a per-feature capability profile alongside the leaderboard, so the contribution is a reproducible axis that is grounded in, rather than divorced from, the original evaluation.

Across GIFT-Eval and fev-bench Mini-20, the two metrics reveal condition-specific strengths and weaknesses that a single aggregate score hides, the augmentation preserves model ordering, and the rankings are stable under regeneration. Most importantly, on genuine data filtered by intrinsic structure, models selected by CaFE’s per-feature profiles outperform the aggregate leaderboard pick, so structural response can guide model selection where a single score cannot. Several limits remain. Stability rests on five seeds on one suite, fine-tuning on a single realization, and the relational validation has few significant cells; selection is shown only for intrinsic structures. Future work includes extending to relational and covariate conditions, more sources, and multi-seed recipes, along with controlled calibration of the response metric.

Conclusion

CaFE extends static time series foundation model benchmarks with refreshable structural diversity and reports endpoint accuracy, response fidelity, and auxiliary-input utility as separate views. On genuine data filtered by intrinsic structure, models selected by CaFE outperform the aggregate leaderboard pick, so model selection can be made conditional on the structural response of interest. The augmentation preserves model ordering, the rankings are stable under regeneration, and endpoint accuracy and response fidelity trade off under a changed training objective.

AI Use Disclosure

Generative AI tools were used to assist with limited aspects of this work, including code implementation support, experiment analysis, and manuscript refinement. These tools were used for tasks such as language editing, text organization, formatting conversion, compilation and layout checks, and consistency verification of terminology and figures. We did not use generative AI tools to generate datasets, experimental results, or mathematical claims. All AI-assisted outputs were reviewed and validated by the authors, who take full responsibility for the final manuscript, including all text, claims, analyses, figures, code, and research artifacts.

Reproducibility Statement

The anonymous artifact, including code and compact numerical inputs for the reported analyses, is available at https://anonymous.4open.science/r/CaFE-ICLR2027-5210. Appendix § 11 specifies data construction, scoring, and aggregation; Appendix § 12.12 records the evaluation panels, model support, and evidence scope; and Appendix § 12.13 documents the computing environments and procedures for reconstructing the tables and figures and rerunning the evaluation. The reported results use the declared panels and one training realization.

References

Profiling, Augmentation, and Structural Feature Constructions

The contracts below define task-qualified References and the eight Structural Feature constructions. The admitted GIFT-Eval tasks form a narrower population than the complete official benchmark; Appendix C reports the support of each analysis.

Stage 1: Profiling

Task contract for each benchmark

Let tt denote one benchmark task. Its adapter contract identifies the source data and fixes the forecast targets and their order, forecast-origin enumeration and horizon, the input history in the benchmark’s native representation, auxiliary inputs allowed by the task and their visibility roles, observation policy, and validity conditions. A univariate setting has one prediction target, whereas a multivariate setting preserves multiple ordered targets as one forecasting object. Either target setting is covariate-informed when the task additionally provides non-target covariates allowed by the task. The adapter maps the native representation to CaFE’s common interface while preserving these semantics; any model-required channel decomposition is recorded as an inference adaptation rather than a change to the task’s target setting.

For the iith valid forecast origin of task tt, the adapter returns a Reference, an adapter-qualified forecast instance from the original benchmark task in CaFE’s common representation, written as

Ri=(Hi,Yi+,Zi,Mi,oi,ιi).(2)R_i = (H_i, Y_i^{+}, Z_i, M_i, o_i, \iota_i). \tag*{(2)}

Here, HiH_i is the target history, Yi+Y_i^{+} is the forecast truth, ZiZ_i contains the auxiliary inputs permitted by tt, MiM_i records the applicable observation masks, including the future-target observation mask that identifies scoreable cells, oio_i is the forecast origin, and ιi\iota_i identifies the benchmark task and native window used for the forecast. The task definition provides the information needed to reconstruct the corresponding Reference. The References of task tt form

At={Ri:Ri satisfies the adapter contract for t}.(3)A_t = \{R_i : R_i \text{ satisfies the adapter contract for } t\}. \tag*{(3)}

Reference materialization and information boundary. The task adapter owns forecast-origin enumeration and future-label eligibility; it may require a fully observed horizon or retain a future-target observation mask, with the resulting scoreability state carried by the Reference. History repair, when required, follows a deterministic adapter-resolved rule derived from history, and the corresponding scale policy declares whether repaired values or originally observed pairs enter history-based statistics. The source history, task contract, and recorded augmentation seed determine Structural Feature selection and every structural, control-setting, and random choice; future target values enter the workflow as scoreable truth after these choices. The future-target observation mask may constrain where a constructed effect must remain visible and determines which future cells are scoreable. Horizon visibility follows the declaration of task tt: known-future dynamic covariates extend across the horizon, and past-only fields end at the forecast origin. Profiling records the adapter, task, source, and native-window identities together with the bound task contract for each RiR_i, supporting later contract checks and source reconstruction under the verification policy selected for the evaluation round.

Stage 2: Augmentation

Contract for an ordered set of samples

For Structural Feature s∈{1,…,8}s \in\{1,\ldots,8\}, Augmentation first applies an SF-specific source filter FspreF_s^{\mathrm{pre}} to References from the original benchmark. Its inputs are source history, Structural Feature evidence, and the task contract. It retains RiR_i only when the selected SF has sufficient source support, every declared setting admits a feasible construction that can satisfy the Structural Feature Expectation, and the inputs and construction domain are physically valid. The retained source Reference is written Ri′R_i' to mark the selection step; the prime changes no task field or provenance. For each selected Reference Ri′R_i' and Structural Feature ss, Augmentation selects a shared structural configuration ξis\xi_{is}. It fixes the affected targets and feature-specific temporal form or predictive relation across the ordered index set Ls\mathcal{L}_s. The evaluation round declares this set; the experimental setting counts and parameter ranges appear in Appendix § 12 and Appendix § 10. The index ℓ∈Ls\ell\in\mathcal{L}_s marks a position within ss, and λisℓ\lambda_{is\ell} is the setting of the feature-specific Control Parameter. The corresponding generated sample and sample set are

Visℓ=Ts(Ri′;ξis,λisℓ),V~is={Visℓ:ℓ∈Ls},(4)\begin{aligned} V_{is\ell} &= T_s(R'_i; \xi_{is}, \lambda_{is\ell}),\\ \widetilde{V}_{is} &= \{V_{is\ell}:\ell\in\mathcal{L}_s\}, \tag*{(4)} \end{aligned}

where TsT_s changes the feature-declared fields while preserving the task, source identifier, origin, horizon, target order, visibility roles, and shared configuration ξis\xi_{is}. The Control Parameter may represent amplitude, change location, period count, or event spacing, so its setting order and scale are local to each Structural Feature. Applying a setting may determine compatible details such as the phase used by SF5; all settings retain Ri′R'_i as their common baseline.

The protocol permits source selection to resample existing References or structural regions according to a declared Structural Feature Distribution. Resampling changes the structural conditions supplied to Augmentation and records the selected view Ri′R'_i. The unchanged Reference population remains the estimand for original-task results, and each run records how its selection policy is instantiated.

Field-level transformation contract. The feature-specific transformation can be written componentwise as

Visℓ=(Hi+ΔHisℓ,  Yi++ΔYisℓ,  Zi+ΔZisℓ,  Mi,  oi,  ιi)(5)V_{is\ell} = \left(H_i + \Delta H_{is\ell},\; Y_i^{+} + \Delta Y_{is\ell},\; Z_i + \Delta Z_{is\ell},\; M_i,\; o_i,\; \iota_i\right) \tag*{(5)}

The deltas identify the fields changed by Structural Feature ss; zero deltas retain the corresponding selected-Reference fields. SF1–SF7 act on target histories and their corresponding generated future targets, so ΔZisℓ=0\Delta Z_{is\ell}=0. SF8 adds a dynamic-covariate impulse allowed by the task and its corresponding target response, with its future covariate component restricted by the task’s visibility declaration. Thus, every transformation changes only the feature-declared fields of Ri′R'_i; Evaluation measures the resulting forecast changes.

Structure selection and setting feasibility. Each mechanism determines applicability and constructs a feasible shared structure. For amplitude-controlled features, the generator normalizes the selected component by its source distance and selects settings only from the feasible interval; SF2, SF4, and SF5 vary period count, transition location, and event spacing. The selected structure is fixed across the ordered settings.

Source-distance evidence. Qualification uses a common distance to keep each generated change visible and bounded relative to its source. Let Tis\mathcal{T}_{is} be the affected target set, CC a protocol-declared history view, and σid>0\sigma_{id}>0 the source-history standard deviation for target dd. A nonfinite or degenerate standard deviation falls back first to the standard deviation of first differences and then to the declared positive value 1. This quantity normalizes construction distance, while the MASE scale σidMASE\sigma_{id}^{\mathrm{MASE}} normalizes forecast error. Stage 2 records

disℓC=1∣Tis∣∑d∈Tis1∣C∣∑u∈C(Hisℓ,u,dV−Hi,u,dRσid)2.(6)d_{is\ell C} = \frac{1}{|\mathcal{T}_{is}|}\sum_{d\in\mathcal{T}_{is}}\sqrt{\frac{1}{|C|}\sum_{u\in C}\left(\frac{H_{is\ell,u,d}^{V}-H_{i,u,d}^{R}}{\sigma_{id}}\right)^2}. \tag*{(6)}

This distance uses the selected Reference Ri′R'_i as the comparator for every declared setting. For every required model-visible context, the macro value must lie in [0.10,2.0][0.10, 2.0]; each affected channel has an upper bound of 3.0. Structural variables such as period count or event spacing serve as feature-specific Control Parameters and express structural progression.

Structural Feature Constructions

Table 2 summarizes the eight constructions. The reported GIFT-Eval and fev-bench Mini-20 experiments use Ls={1,…,5}\mathcal{L}_s=\{1,\ldots,5\}, displayed as L1–L5. The formulas specify the changed fields, shared structural configuration, and feature-specific control settings; the numerical ranges below belong to this experimental configuration.

Shared Notation and Control Contract

For Reference RiR_i, let LiL_i and HihorH_i^{\mathrm{hor}} denote its history length and forecast horizon. We use u∈{0,…,Li+Hihor−1}u \in\{0,\ldots,L_i+H_i^{\mathrm{hor}}-1\} on the concatenated target path

Xi,d(u)={Hi[u,d],0≤u<Li,Yi+[u−Li,d],Li≤u<Li+Hihor,(7)X_{i,d}(u)= \begin{cases} H_i[u,d], & 0 \le u < L_i,\\ Y_i^{+}[u-L_i,d], & L_i \le u < L_i+H_i^{\mathrm{hor}}, \end{cases} \tag*{(7)}

where dd indexes prediction targets. Let Ui,q(u)U_{i,q}(u) denote task-permitted dynamic covariate qq on its declared visible range. A generated sample changes these fields by

XisℓV=Xi+ΔXisℓ,UisℓV=Ui+ΔUisℓ.(8)X_{is\ell}^{V}=X_i+\Delta X_{is\ell}, \qquad U_{is\ell}^{V}=U_i+\Delta U_{is\ell}. \tag*{(8)}

The future values in Equation (7) provide scoreable truth after construction. Structural configuration ξis\xi_{is} is selected from source history and task information before those values enter the workflow.

For the amplitude-controlled Structural Features SF1, SF3, SF6, SF7, and SF8, let GisX(u,d)G_{is}^{X}(u,d) and GisU(u,q)G_{is}^{U}(u,q) be the raw target and covariate components fixed by ξis\xi_{is}. With affected-target set Jis\mathcal{J}_{is} and source-history standard-deviation scale σid\sigma_{id}, define the full-history macro normalized root-mean-square

Di(GisX;Jis)=1∣Jis∣∑d∈Jis1Li∑u=0Li−1(GisX(u,d)σid)2.(9)D_i(G_{is}^{X};\mathcal{J}_{is})=\frac{1}{|\mathcal{J}_{is}|}\sum_{d\in\mathcal{J}_{is}}\sqrt{\frac{1}{L_i}\sum_{u=0}^{L_i-1}\left(\frac{G_{is}^{X}(u,d)}{\sigma_{id}}\right)^2}. \tag*{(9)}

Writing qis=Di(GisX;Jis)q_{is}=D_i(G_{is}^{X};\mathcal{J}_{is}), Salience Tuning at control setting ℓ∈Ls\ell\in\mathcal{L}_s gives

(ΔXisℓ,ΔUisℓ)=λisℓqis(GisX,GisU).(10)(\Delta X_{is\ell},\Delta U_{is\ell})=\frac{\lambda_{is\ell}}{q_{is}}\left(G_{is}^{X},G_{is}^{U}\right). \tag*{(10)}

In this experimental instantiation, the five nominal intervals for λisℓ\lambda_{is\ell} are [0.10,0.15][0.10,0.15], [0.17,0.22][0.17,0.22], [0.25,0.32][0.25,0.32], [0.36,0.46][0.36,0.46], and [0.50,0.65][0.50,0.65], intersected with the feasible interval of the selected structure. SF2, SF4, and SF5 instead use period count, transition location, and event gap as their feature-specific Control Parameters. Each SF therefore interprets L1–L5 through its own Control Parameter and Control Settings, and Evaluation measures the forecast changes under those settings.

Figure 12 illustrates the constructed components defined below.

Illustrative interventions for SF1–SF8

Figure 12. Illustrative interventions for SF1–SF8. Each panel joins the modified history to its analytic continuation over the forecast horizon and preserves the task's target and auxiliary-input roles.

SF1 Trend

SF1 generates a trend by selecting a nonempty target subset Ji1\mathcal{J}_{i1} with history-stable trend signs sidtr∈{−1,1}s_{id}^{\mathrm{tr}}\in\{-1,1\}. For each selected target, the raw component is

Gi1X(u,d)=1{d∈Ji1}σidsidtrpi1(u),ΔXi1ℓ=λi1ℓqi1Gi1X.(11)G_{i1}^{X}(u,d)=\mathbf{1}\{d\in\mathcal{J}_{i1}\}\sigma_{id}s_{id}^{\mathrm{tr}}p_{i1}(u), \qquad\Delta X_{i1\ell}=\frac{\lambda_{i1\ell}}{q_{i1}}G_{i1}^{X}. \tag*{(11)}

where pi1p_{i1} is one structure shared across all settings:

plin(u)=uLi−1,pdel(u)=(u−u0)+Li−1,pcurv(u)=(uLi−1)r,ppw(u)=1Li−1∑v=0u−1b⌊(v+ϕ)/K⌋ mod B,pi1∈{plin,pdel,pcurv,ppw}.(12)\begin{aligned} p_{\mathrm{lin}}(u) &= \frac{u}{L_i-1},\\ p_{\mathrm{del}}(u) &= \frac{(u-u_0)_+}{L_i-1},\\ p_{\mathrm{curv}}(u) &= \left(\frac{u}{L_i-1}\right)^r,\\ p_{\mathrm{pw}}(u) &= \frac{1}{L_i-1}\sum_{v=0}^{u-1} b_{\lfloor(v+\phi)/K\rfloor\bmod B},\\ p_{i1} &\in\{p_{\mathrm{lin}},p_{\mathrm{del}},p_{\mathrm{curv}},p_{\mathrm{pw}}\}. \tag*{(12)} \end{aligned}

The four alternatives are whole-history linear, delayed linear, slow-curvature, and recurring piecewise-linear continuation. The onset u0u_0, curvature rr, knot interval KK, phase ϕ\phi, slope cycle (b0,…,bB−1)(b_0,\ldots,b_{B-1}), affected targets, and signs belong to ξi1\xi_{i1}; only normalized magnitude λi1ℓ\lambda_{i1\ell} changes across settings. Construction requires a stable history-derived direction and continues the same profile law through the horizon.

SF2 Multi-Seasonality

For each target, SF2 constructs multi-seasonality by selecting a harmonic anchor from up to three qualified history-spectrum candidates, with a protocol-generated anchor when none qualifies. For target dd, let the selected anchor and five additional independent periods be Pid0,…,Pid5P_{id0},\ldots,P_{id5}. Component rr is either a sinusoid or a two-harmonic waveform,

hidr(u)=∑m=1Midraidrmsin⁡(2πmuPidr)+∑m=1Midrbidrmcos⁡(2πmuPidr),Midr∈{1,2}.(13)\begin{aligned} h_{idr}(u) &= \sum_{m=1}^{M_{idr}} a_{idrm}\sin\left(\frac{2\pi m u}{P_{idr}}\right)\\ &\quad+ \sum_{m=1}^{M_{idr}} b_{idrm}\cos\left(\frac{2\pi m u}{P_{idr}}\right),\\ M_{idr} &\in\{1,2\}. \tag*{(13)} \end{aligned}

Each component is history-normalized to the target scale before aggregation. At Control Setting ℓ∈{1,…,5}\ell\in\{1,\ldots,5\}, the generated sample contains the anchor and the first ℓ\ell additional periods:

Si2ℓ(u,d)=∑r=0ℓhidr(u),ΔXi2ℓ(u,d)=ρi2Di(Si2ℓ;Ji2)Si2ℓ(u,d),(14)S_{i2\ell}(u,d)=\sum_{r=0}^{\ell}h_{idr}(u),\qquad\Delta X_{i2\ell}(u,d)=\frac{\rho_{i2}}{\mathcal{D}_i(S_{i2\ell};\mathcal{J}_{i2})}S_{i2\ell}(u,d), \tag*{(14)}

where Ji2\mathcal{J}_{i2} contains all targets and the shared full-history distance ρi2∈[0.22,0.28]\rho_{i2}\in[0.22,0.28] is fixed across the five settings. Hence the Control Parameter is the number of additional independent periods, producing two through six periods while holding aggregate history amplitude fixed. Periods, phases, waveform families, and their order are selected from history-qualified or protocol-generated candidates without future target values.

SF3 Time-Varying Seasonality

SF3 constructs time-varying seasonality. For an eligible target dd, the selected carrier, its quadrature component, and a slower envelope are

cid(u)=aidcsin⁡(ωidcu)+bidccos⁡(ωidcu),cid⊥(u)=bidcsin⁡(ωidcu)−aidccos⁡(ωidcu),eid(u)=aidesin⁡(ωideu)+bidecos⁡(ωideu),ωide<ωidc.(15)\begin{aligned} c_{id}(u)&=a^{c}_{id}\sin(\omega^{c}_{id}u)+b^{c}_{id}\cos(\omega^{c}_{id}u),\\ c^{\perp}_{id}(u)&=b^{c}_{id}\sin(\omega^{c}_{id}u)-a^{c}_{id}\cos(\omega^{c}_{id}u),\\ e_{id}(u)&=a^{e}_{id}\sin(\omega^{e}_{id}u)+b^{e}_{id}\cos(\omega^{e}_{id}u),\qquad\omega^{e}_{id}<\omega^{c}_{id}. \tag*{(15)} \end{aligned}

The raw component uses the carrier for amplitude modulation and the quadrature carrier for the periodic phase-drift component:

bidmod(u)={cid(u),amplitude modulation,cid⊥(u),periodic phase drift,Gi3X(u,d)=1{d∈Ji3}eid(u)bidmod(u),b^{\mathrm{mod}}_{id}(u)= \begin{cases} c_{id}(u), & \text{amplitude modulation},\\ c^{\perp}_{id}(u), & \text{periodic phase drift}, \end{cases} \qquad G^{X}_{i3}(u,d)=\mathbf{1}\{d\in\mathcal{J}_{i3}\}e_{id}(u)b^{\mathrm{mod}}_{id}(u),
ΔXi3ℓ=λi3ℓqi3Gi3X.(16)\Delta X_{i3\ell}=\frac{\lambda_{i3\ell}}{q_{i3}}G^{X}_{i3}. \tag*{(16)}

History fits determine the carrier and slower envelope; for each eligible target, the seed selects one qualified carrier and modulation from a leading pool of at most eight, then selects a nonempty target subset. Control Setting ℓ\ell changes only the normalized magnitude. Qualification requires the envelope to remain active on the declared fraction of observed horizon positions.

SF4 Regime Switching

SF4 constructs regime switching. Let τi4ℓ\tau_{i4\ell} be the setting-specific transition location, wi4w_{i4} the shared transition width, and

p(u)=clip⁡(u−τw,0,1),p(u)=\operatorname{clip}\left(\frac{u-\tau}{w},0,1\right),
rκ(u;τ,w)={1{u≥τ},κ=step,p(u),κ=ramp,3p(u)2−2p(u)3,κ=sigmoid.(17)r_{\kappa}(u;\tau,w)= \begin{cases} \mathbf{1}\{u\ge\tau\}, & \kappa=\text{step},\\ p(u), & \kappa=\text{ramp},\\ 3p(u)^2-2p(u)^3, & \kappa=\text{sigmoid}. \end{cases} \tag*{(17)}

With affected-target signs sidrgs^{\mathrm{rg}}_{id}, the five treatments are

ΔXi4ℓ(u,d)=αi41{d∈Ji4}σidsidrgrκi4(u;τi4ℓ,wi4).(18)\Delta X_{i4\ell}(u,d)=\alpha_{i4}\mathbf{1}\{d\in\mathcal{J}_{i4}\}\sigma_{id}s^{\mathrm{rg}}_{id}r_{\kappa_{i4}}(u;\tau_{i4\ell},w_{i4}). \tag*{(18)}

The seed fixes Ji4\mathcal{J}_{i4}, the signs, transition family κi4\kappa_{i4}, and width. The Control Parameter is the transition location, represented by the ordered settings τi4ℓ/Li\tau_{i4\ell}/L_i; larger settings place the same state change closer to the forecast origin and therefore expose less post-transition history. The shared gain αi4\alpha_{i4} is chosen once for the five settings to meet the history-distance contract.

SF5 Intermittency

SF5 constructs intermittency. Let wi5w_{i5} be the shared pulse width. For offset rr, the selected pulse kernel is

κi5(r)={1,0≤r<wi5,rectangular,1−∣r−(wi5−1)/2∣max⁡(1,(wi5+1)/2),0≤r<wi5,triangular,exp⁡[−(log⁡2)rmax⁡(1,wi5/2)],0≤r<wi5,exponential,0,otherwise.(19)\kappa_{i5}(r)= \begin{cases} 1, & 0 \le r < w_{i5},\quad\text{rectangular},\\ 1-\dfrac{\lvert r-(w_{i5}-1)/2\rvert}{\max(1,(w_{i5}+1)/2)}, & 0 \le r < w_{i5},\quad\text{triangular},\\ \exp\left[-\dfrac{(\log2)r}{\max(1,w_{i5}/2)}\right], & 0 \le r < w_{i5},\quad\text{exponential},\\ 0, & \text{otherwise}. \end{cases} \tag*{(19)}

For Control Setting ℓ\ell, event centers Ei5ℓE_{i5\ell} follow mean gap gi5ℓg_{i5\ell} with bounded alternating jitter ϵi5ℓ\epsilon_{i5\ell}, so consecutive gaps are max⁡(wi5+1,gi5ℓ±ϵi5ℓ)\max(w_{i5}+1,g_{i5\ell}\pm\epsilon_{i5\ell}). The target increment is

ΔXi5ℓ(u,d)=αi51{d∈Ji5}σidsidev∑t∈Ei5ℓκi5(u−t).(20)\Delta X_{i5\ell}(u,d)=\alpha_{i5}\mathbf{1}\{d\in\mathcal{J}_{i5}\}\sigma_{id}s_{id}^{\mathrm{ev}}\sum_{t\in E_{i5\ell}}\kappa_{i5}(u-t). \tag*{(20)}

The shared structure fixes the affected targets, event signs, pulse family, width, and jitter fraction. The Control Parameter is the mean-gap fraction of the maximum legal gap; larger settings are sparser. A deterministic compatible phase retains at least three history events and an event on an observed future position. The common gain αi5\alpha_{i5} is fixed across settings and satisfies the history-distance and minimum future-effect gates.

SF6 Common Factor

SF6 constructs common factor. Standardize the target history as Zi(u,d)=(Hi[u,d]−μid)/σidZ_i(u,d)=(H_i[u,d]-\mu_{id})/\sigma_{id} and write its singular value decomposition as Zi=UiΣiViTZ_i=U_i\Sigma_iV_i^{\mathsf{T}}. Candidate loading vectors are masked and renormalized members of

Bi={v1,v1±0.55v2,v1±0.45v3},\mathcal{B}_i=\{v_1,v_1\pm0.55v_2,v_1\pm0.45v_3\},
Wi={mask⁡η(w):w∈Bi, η∈{0.10,0.20,0.35}}.(21)\mathcal{W}_i=\{\operatorname{mask}_{\eta}(w):w\in\mathcal{B}_i,\ \eta\in\{0.10,0.20,0.35\}\}. \tag*{(21)}

where vjv_j is row jj of ViTV_i^{\mathsf{T}} and mask⁡η\operatorname{mask}_{\eta} removes loadings below fraction η\eta of the largest absolute loading while retaining at least three targets. For a selected loading widw_{id}, the history-fitted factor ZiwiZ_iw_i supplies a stable harmonic carrier fi(u)=aifsin⁡(ωifu)+bifcos⁡(ωifu)f_i(u)=a_i^{f}\sin(\omega_i^{f}u)+b_i^{f}\cos(\omega_i^{f}u). The controlled component is

Gi6X(u,d)=1{d∈Ji6}σidwidfi(u),ΔXi6ℓ=λi6ℓqi6Gi6X.(22)G_{i6}^{X}(u,d)=\mathbf{1}\{d\in\mathcal{J}_{i6}\}\sigma_{id}w_{id}f_i(u),\qquad\Delta X_{i6\ell}=\frac{\lambda_{i6\ell}}{q_{i6}}G_{i6}^{X}. \tag*{(22)}

After qualification and deduplication, the seed selects one of at most 12 leading loading–carrier candidates; the loading, participating targets, and carrier remain fixed across settings. The Control Parameter is normalized common-factor magnitude. This construction measures observable comovement under the selected loading and carrier.

SF7 Cross-Series Dependence

SF7 constructs lagged dependence by enumerating ordered driver–responder pairs and positive lags up to 24 or one eighth of the Reference-history length, whichever is smaller. For each ordered driver pp, responder dd, and positive lag τ\tau, SF7 fits the history-only predictive comparison

zi,d(u)=β0+β1zi,d(u−1)+β2zi,p(u−τ)+εu,z_{i,d}(u)=\beta_0+\beta_1z_{i,d}(u-1)+\beta_2z_{i,p}(u-\tau)+\varepsilon_u,
Gi7X(u,j)=1{j=d}σidβ2z~i,p(u−τ),G_{i7}^{X}(u,j)=\mathbf{1}\{j=d\}\sigma_{id}\beta_2\widetilde{z}_{i,p}(u-\tau),
ΔXi7ℓ=λi7ℓqi7Gi7X.(23)\Delta X_{i7\ell}=\frac{\lambda_{i7\ell}}{q_{i7}}G_{i7}^{X}. \tag*{(23)}

Here zi,dz_{i,d} is the standardized target history, and z~i,p\widetilde{z}_{i,p} equals the observed driver history before the forecast origin and its history-fitted linear continuation afterward. Candidate edges must improve predictive fit over the responder-only model. SF7 ranks them by incremental predictive fit, scans at most 48 edges, and retains at most 12 that satisfy source-distance feasibility; the seed selects one retained edge. The driver, responder, lag, and coefficient β2\beta_2 are fixed across settings, while λi7ℓ\lambda_{i7\ell} controls transfer magnitude. Under this history-only construction, the resulting edge is interpreted as a predictive diagnostic for SF7.

SF8 Covariate Response

SF8 constructs covariate response by selecting a legal dynamic covariate qq, target dd, event set Ei8E_{i8}, covariate sign siUs_i^U, response sign siXs_i^X, terminal pre-origin impulse t0t_0 with multiplier γ0\gamma_0, and a one-sided response kernel. With delay δ\delta and time-scale parameter hh (the half-life for the exponential family), the raw kernel is

kˉi8(r)={0,r<δ,exp⁡[−(log⁡2)(r−δ)/h],exponential,r−δhexp⁡[1−(r−δ)/h],delayed gamma,max⁡(0,1−r−δ2h),triangular.(24)\bar{k}_{i8}(r)= \begin{cases} 0, & r<\delta,\\ \exp[-(\log2)(r-\delta)/h], & \text{exponential},\\ \frac{r-\delta}{h}\exp[1-(r-\delta)/h], & \text{delayed gamma},\\ \max\left(0,1-\frac{r-\delta}{2h}\right), & \text{triangular}. \end{cases} \tag*{(24)}

The peak-normalized kernel is

ki8(r)=kˉi8(r)max⁡v≥0kˉi8(v),max⁡r≥0ki8(r)=1.(25)k_{i8}(r)=\frac{\bar{k}_{i8}(r)}{\max_{v\ge0}\bar{k}_{i8}(v)},\qquad\max_{r\ge0}k_{i8}(r)=1. \tag*{(25)}

Define the signed impulse path

Ii8(u)=siU[∑t∈Ei81{u=t}+γ01{u=t0}].(26)I_{i8}(u)=s_i^U\left[\sum_{t\in E_{i8}}1\{u=t\}+\gamma_0 1\{u=t_0\}\right]. \tag*{(26)}

Let σiqU\sigma_{iq}^U be the source covariate scale. The raw covariate and target components and their shared setting scaling are

Gi8U(u,j)=1{j=q} σiqUIi8(u),G_{i8}^U(u,j)=1\{j=q\}\,\sigma_{iq}^U I_{i8}(u),
Gi8X(u,j)=1{j=d} σidsiX∑v≤uIi8(v)ki8(u−v),G_{i8}^X(u,j)=1\{j=d\}\,\sigma_{id}s_i^X\sum_{v\le u}I_{i8}(v)k_{i8}(u-v),
(ΔXi8ℓ,ΔUi8ℓ)=λi8ℓqi8(Gi8X,Gi8U).(27)(\Delta X_{i8\ell},\Delta U_{i8\ell})=\frac{\lambda_{i8\ell}}{q_{i8}}(G_{i8}^X,G_{i8}^U). \tag*{(27)}

Repeated historical impulses and the terminal impulse use the same kernel. A future covariate impulse is included only when the task declares that covariate known over the horizon; past-only covariates stop at the forecast origin. SF8 constructs at most eight candidate structural configurations; after feasibility qualifications, the seed selects one qualified structural configuration, and λi8ℓ\lambda_{i8\ell} jointly scales the covariate impulse and target response without changing their signs, timing, target, covariate, or kernel.

Inspection, Sifting, and Evaluation Protocol

This appendix gives the admission rules, execution contracts, and metric definitions for the constructions in Appendix A. Empirical support is reported in Appendix C.

Inspection and Sifting

Check scopes and experiment binding. Structural Feature Expectation, Source Distance, and physical or task checks operate on individual samples; Distribution Difference and Diversity operate on the generated sample set relative to a declared source population. The contracts below specify both qualification scopes. A reported run binds each check to its descriptor, population, statistic, calibrated threshold or coverage requirement, and observed decision.

Sample Inspection

Sample Inspection evaluates each realized generated sample ViℓV_{i\ell} against four recorded requirements: consistency with the structure selected in Augmentation; the realized Structural Feature Expectation and non-target tolerance; Source Distance and physical or task constraints on every required model-visible history suffix; and any feature-specific horizon or future-effect support condition. The Source Distance evidence uses (6): every required history view must have macro distance in [0.10,2.0][0.10, 2.0], and every affected channel must have distance at most 3.03.0. The verification record stores pass or nonnumeric failure states, and numeric aggregation includes scored samples with their recorded support.

Admitting a complete sample set

A candidate sample set V~is\widetilde{\mathcal{V}}_{i}^{s} qualifies by satisfying all of the following conditions:

  1. Structural Feature ss is applicable to Ri′R_{i}^{\prime}, a generated sample is present for every ℓ∈Ls\ell\in\mathcal{L}_{s}, and every sample passes Sample Inspection;

  2. all generated samples share the selected Reference Ri′R_{i}^{\prime}, fields, and structural configuration required by Equations (4) and (5); and

  3. the ordered Control Settings and other sample-set construction invariants of Structural Feature ss hold across the complete set; and

  4. the set-level Structural Feature Descriptor distribution satisfies the declared Distribution Difference bound and its structural configurations and Control Settings satisfy the declared Diversity requirement.

Each (Ri′,s)(R_{i}^{\prime},s) pair contributes all ∣Ls∣|\mathcal{L}_{s}| generated samples or a recorded nonnumeric state, while its original Reference remains in At\mathcal{A}_{t}. Denote a passing set by Vis\mathcal{V}_{is}; the qualified sample sets of task tt form

Pt={(Ri′,Vis):V~is passes sample-set qualification}.(28)\mathcal{P}_{t}=\{(R_{i}^{\prime},\mathcal{V}_{is}):\widetilde{\mathcal{V}}_{is}\text{ passes sample-set qualification}\}. \tag*{(28)}

The record stores the reason and non-numeric state for each inapplicable or rejected construction. Structural Features that declare a minimum future change enforce it during sample-set construction; other features apply truth-change sufficiency at NRMSE scoring, while an eligible MASE value remains available.

Task-level contract verification. Verification checks treatment records for source distance, mechanism support, and applicable horizon support, recording accepted and rejected contracts. When the source data support exact replay, it also reconstructs the References and generated samples and compares their task records. Write VerOK⁡t=1\operatorname{VerOK}_{t}=1 when the declared policy accepts the task.

Verified evaluation bundle. For an accepted task, Stage 3 assembles

Dt=(At,Pt)if VerOK⁡t=1.(29)\mathcal{D}_{t}=(\mathcal{A}_{t},\mathcal{P}_{t})\quad\text{if }\operatorname{VerOK}_{t}=1. \tag*{(29)}

The bundle carries the links between each Reference and generated sample together with the accepted verification policy and report. Inference receives only bundles with an accepted report. Any SF6–SF8 ablated-input view is derived from an admitted sample after verification and recorded under a separate inference-ablation contract.

Evaluation

Model Configuration and Execution States

Each paired Reference–sample comparison uses one fixed model configuration, including model identity, preprocessing, target/covariate mapping, context policy, and permitted adaptation. Auxiliary-input pairs additionally fix the randomization policy and change only the declared input view.

Input projection and output restoration. The Evaluator checks horizon and input compatibility, projects each constructed object to the model context, restores target order after any permitted channel-wise adaptation, and records unsupported, failed, and nonfinite requests separately from valid forecasts. Auxiliary removal is evaluated only for configurations that receive the designated input.

Metric scoreability is recorded separately from execution: each metric uses only valid forecasts and required pairings, while unsupported and nonnumeric states retain their coverage counts.

MASE scale from the Reference history

Mean absolute scaled error (MASE) [12] measures forecast error across target scales. Let LiL_i be the history length, let ptp_t be the adapter-resolved seasonality, and set τi=min⁡(max⁡(1,pt),max⁡(1,Li−1))\tau_i = \min(\max(1,p_t),\max(1,L_i-1)). For target dd, let Qid(τi)Q_{id}(\tau_i) contain the history-index pairs admitted by the adapter-resolved scale policy. A complete repaired-history policy admits the repaired cells, whereas a history-observation-preserving scale policy admits pairs whose two source cells are observed. The Reference-history scale is

σidMASE=1∣Qid(τi)∣∑u∈Qid(τi)∣Hi,u,d−Hi,u−τi,d∣.(30)\sigma^{\mathrm{MASE}}_{id} = \frac{1}{\lvert Q_{id}(\tau_i)\rvert}\sum_{u\in Q_{id}(\tau_i)} \lvert H_{i,u,d} - H_{i,u-\tau_i,d}\rvert. \tag*{(30)}

The evaluator uses the declared fallback when the seasonal-pair scale is unavailable or degenerate. The resulting Reference-history scale is shared by the Reference, generated samples, and any eligible ablation pair.

For each Reference and SF, its generated samples and any eligible full/ablated inference pair share the resulting σidMASE\sigma^{\mathrm{MASE}}_{id}.

For object type X∈{R,V}X \in\{R,V\}, let ΩiX\Omega_i^X contain its observed future cells. Its instance-level score is

MASEiX=1∣ΩiX∣∑(h,d)∈ΩiX∣Y^i,h,dX−Yi,h,dX∣σidMASE.(31)\mathrm{MASE}_i^X = \frac{1}{\lvert\Omega_i^X\rvert}\sum_{(h,d)\in\Omega_i^X}\frac{\lvert\widehat{Y}_{i,h,d}^X-Y_{i,h,d}^X\rvert}{\sigma^{\mathrm{MASE}}_{id}}. \tag*{(31)}

Within a fixed task and result unit, the evaluator averages instance-level values equally. Lower MASE therefore denotes smaller forecast error relative to the Reference history’s variation scale. For X=RX=R, this result object is Reference MASE; for X=VX=V, it is Probe MASE. The input-ablation comparison uses the same formula restricted to the assessed target under the full and ablated input views.

NRMSE for response to controlled changes

Fix task tt, model configuration mm, Structural Feature ss, and control setting ℓ\ell. For an affected and observed future cell k=(h,d)k=(h,d) of instance ii, let superscript (a)(a) denote the generated-sample view and superscript (0)(0) denote its selected Reference. Define the truth and forecast changes as

ΔYik=Yik(a)−Yik(0),ΔY^ik=Y^ik(a)−Y^ik(0).(32)\Delta Y_{ik}=Y_{ik}^{(a)}-Y_{ik}^{(0)},\qquad\Delta\widehat{Y}_{ik}=\widehat{Y}_{ik}^{(a)}-\widehat{Y}_{ik}^{(0)}. \tag*{(32)}

Let ΩiΔ\Omega_i^\Delta contain the affected observed cells declared by TsT_s, and use the Reference-history scale σikMASE=σidMASE\sigma_{ik}^{\mathrm{MASE}}=\sigma_{id}^{\mathrm{MASE}}. The standardized truth-change root-mean-square is

qi=1∣ΩiΔ∣∑k∈ΩiΔ(ΔYikσikMASE)2.(33)q_i=\sqrt{\frac{1}{\lvert\Omega_i^\Delta\rvert}\sum_{k\in\Omega_i^\Delta}\left(\frac{\Delta Y_{ik}}{\sigma_{ik}^{\mathrm{MASE}}}\right)^2}. \tag*{(33)}

The eligible set It,m,s,ℓΔI_{t,m,s,\ell}^\Delta contains instances with two valid forecasts, nonempty ΩiΔ\Omega_i^\Delta, and qi≥0.05q_i\ge0.05. The task-level normalized root-mean-square error (NRMSE) is

NRMSEt,m,s,ℓ=∑i∈It,m,s,ℓΔ∑k∈ΩiΔ(ΔY^ik−ΔYikσikMASE)2∑i∈It,m,s,ℓΔ∑k∈ΩiΔ(ΔYikσikMASE)2.(34)\mathrm{NRMSE}_{t,m,s,\ell}= \sqrt{ \frac{ \sum_{i\in I_{t,m,s,\ell}^\Delta}\sum_{k\in\Omega_i^\Delta} \left(\frac{\Delta\widehat{Y}_{ik}-\Delta Y_{ik}}{\sigma_{ik}^{\mathrm{MASE}}}\right)^2 }{ \sum_{i\in I_{t,m,s,\ell}^\Delta}\sum_{k\in\Omega_i^\Delta} \left(\frac{\Delta Y_{ik}}{\sigma_{ik}^{\mathrm{MASE}}}\right)^2 } }. \tag*{(34)}

The ratio pools standardized squared sums over eligible instances and cells and requires a nonempty set with positive truth-change energy. Equivalently, its square is the mean of the squared pair-level ratios weighted by each pair’s standardized target-change energy, which includes its number of affected observed cells. Instances below the 0.050.05 truth-change threshold remain available to MASE and retain a non-numeric NRMSE state.

Auxiliary-Input Gap after removing an auxiliary input

The inference ablation applies to model/view pairs that expose the designated auxiliary input. For the SF6 intervention, the highest-absolute-loading target is assessed and the ablated view retains only that target, deleting every other target channel. For the SF7 intervention, the responder is assessed and the ablated view retains only the responder, again deleting every other target channel. For the SF8 intervention, the target panel is retained in both views while the ablated view deletes the designated dynamic-covariate carrier together with its corresponding name, availability, future-visibility, and type metadata. For both SF6 and SF7, task-permitted non-target covariates remain available in the full and ablated views. The SF6/SF7 comparison includes configurations receiving other target channels; the SF8 comparison includes configurations receiving the designated carrier.

Let Ωiabl\Omega_i^{\mathrm{abl}} contain the observed future cells of the feature-specific assessed target. Let Ai(a)A_i^{(a)} and AiablA_i^{\mathrm{abl}} denote (31) restricted to Ωiabl\Omega_i^{\mathrm{abl}} for the full generated-sample and ablated-input views, respectively. The two views share the assessed-target history and future, forecast origin, horizon, observation mask, and Reference-history scale. Let It,m,s,ℓabl\mathcal{I}_{t,m,s,\ell}^{\mathrm{abl}} contain the eligible pairs for which both forecasts are scoreable. Auxiliary-Input Gap is

ΔMASE⁡t,m,s,ℓ=1∣It,m,s,ℓabl∣∑i∈It,m,s,ℓabl(Aiabl−Ai(a)).(35)\Delta\operatorname{MASE}_{t,m,s,\ell} = \frac{1}{\lvert\mathcal{I}_{t,m,s,\ell}^{\mathrm{abl}}\rvert} \sum_{i \in\mathcal{I}_{t,m,s,\ell}^{\mathrm{abl}}} \left(A_i^{\mathrm{abl}} - A_i^{(a)}\right). \tag*{(35)}

Positive values mean that auxiliary-input removal increases assessed-target error, zero denotes no average signed difference, and negative values mean that removal decreases the error. The quantity records net predictive contribution under the declared deletion.

Recorded Analysis Aggregation and Uncertainty

The analyses in this paper use the following CAFE-native task-equal rule; source-benchmark evaluator and leaderboard parity are separate validation targets. For a result unit rr selected by the analysis contract for suite-level reporting, let Metric⁡t,r,m\operatorname{Metric}_{t,r,m} be the task-level value for model configuration mm, and let Tr,m\mathcal{T}_{r,m} be the tasks on which that value is computed. The unit identifies one metric and its evaluation object or comparison condition, plus a structural feature and control setting where applicable. Its recorded task-equal suite result is

SuiteMetric⁡r,m=1∣Tr,m∣∑t∈Tr,mMetric⁡t,r,m,∣Tr,m∣>0.(36)\operatorname{SuiteMetric}_{r,m} = \frac{1}{\lvert\mathcal{T}_{r,m}\rvert} \sum_{t \in\mathcal{T}_{r,m}} \operatorname{Metric}_{t,r,m}, \qquad\lvert\mathcal{T}_{r,m}\rvert> 0. \tag*{(36)}

Thus, each benchmark task contributes one value regardless of its number of References or qualified sample sets. The suite row records the task-set cardinality, and task-level values remain available for paired comparisons.

For paired model comparisons, the task set is Tr,m1∩Tr,m2\mathcal{T}_{r,m_1} \cap\mathcal{T}_{r,m_2}, and the analysis averages one difference per common task with equal weight. The bootstrap resamples tasks for a single-model result and paired task differences for a model comparison; the analysis contract records its seed, repetition count, interval rule, and task identities. Each metric retains its own scoreable support. The secondary structural-response macro uses equally weighted, direction-corrected SF-local ranks on common model support and excludes raw NRMSE pooling and Auxiliary-Input Gap.

Evaluation and Replay Scope

Evaluation records. Each result unit retains its benchmark task, Reference, structural feature, control setting, model configuration, and scoreable support. These fields keep paired comparisons aligned while allowing unsupported or nonnumeric cases to remain explicit:

St→Stages 1–2Gt→Stage 3(Dt,Vt)→mPt,m→MetricsAt,m.(37)\mathcal{S}_t \xrightarrow{\text{Stages 1--2}} \mathcal{G}_t \xrightarrow{\text{Stage 3}} (\mathcal{D}_t,\mathcal{V}_t) \xrightarrow{m} \mathcal{P}_{t,m} \xrightarrow{\text{Metrics}} \mathcal{A}_{t,m}. \tag*{(37)}

where St\mathcal{S}_t denotes the benchmark-task input, Gt\mathcal{G}_t the generation conditions, Vt\mathcal{V}_t the qualification result, Pt,m\mathcal{P}_{t,m} the predictions, and At,m\mathcal{A}_{t,m} the metric results. The analysis records each result unit’s task-set cardinality and keeps the task-level values available for paired comparisons.

The retained records support progressively broader checks, from treatment verification to construction and metric reconstruction and then end-to-end forecasting replay. Appendix § 12.13 specifies the required inputs and procedures for each reproduction level.

SuiteHorizonConfigurationsModels
GIFT-Eval Short30–605511
GIFT-Eval Medium480–6002110
GIFT-Eval Long720–9002110
fev-bench Mini-205–2882011

Table 3. Displayed model support of the four latest result packages. Short and FEV use 11 models; Medium and Long use 10 because TiREx-2 is outside the supported horizon.

Supporting Results and Reproducibility

The following sections supply experiment-specific support, aggregation details, and complete numerical results in the order of the main experiments. Earlier analysis blocks retain their original populations; reproduction requirements follow the results.

Rank-Correlation Definitions

For two quantities XX and YY observed on the same nn items, Spearman’s rank correlation is the Pearson correlation of their ranks, ρs=corr⁡(RX,RY)\rho_s = \operatorname{corr}(R_X, R_Y); when there are no ties, this is equivalently ρs=1−6∑i=1ndi2/[n(n2−1)]\rho_s = 1 - 6\sum_{i=1}^{n}d_i^2/[n(n^2-1)], where did_i is the difference between the two ranks (?). We use average ranks for tied values.

Kendall’s tie-adjusted rank correlation is reported as

τb=C−D(C+D+TX)(C+D+TY)(38)\tau_b = \frac{C-D}{\sqrt{(C+D+T_X)(C+D+T_Y)}} \tag*{(38)}

where CC and DD count concordant and discordant item pairs, and TXT_X and TYT_Y count pairs tied only in XX and only in YY, respectively (?). Thus ρs\rho_s summarizes agreement after rank transformation, whereas τb\tau_b measures pairwise ordering agreement while accounting for ties; both statistics are computed on the stated common model support.

Computational Environment

Experiments used a standard KVM-based compute-node configuration with 48 Intel Xeon Gold 6530 vCPUs, 188 GiB of system memory, and four NVIDIA GeForce RTX 5090 GPUs with 32,607 MiB of memory per GPU, running Ubuntu 24.04.4 LTS. The CAFE data-generation, validation, and aggregation pipeline used Python 3.12.3. Model inference services used Python 3.11.15, PyTorch 2.10.0, CUDA 12.8, cuDNN 9.10, and NVIDIA driver 575.57.08. This profile establishes the standard per-node hardware and software environment. Appendix § 12.12 summarizes the corresponding evaluation support and analysis scope.

Evaluation Support and Aggregation

Table 3 records the supported horizons, configuration counts, and model roster for each frozen suite. Timer-3.5 denotes the evaluated Timer-S1 checkpoint [18].

Suite aggregation follows Equation (36). Probe MASE and Forecast-Change NRMSE first average eligible task-level values within each SF–setting cell, then apply equal setting weights within the feature; NRMSE pools standardized squared terms within each task as in Equation (34). Feature-local rankings use common support and average ranks for ties. The secondary macro maps direction-corrected ranks to [0,1][0,1] before averaging SF1–SF8 equally; Auxiliary-Input Gap is excluded.

Accuracy and Response Fidelity

The winner table uses the complete evaluated roster, whereas the common ten-model heatmap displays only TiREx-2. The two views therefore compare different model sets on Short and fev-bench Mini-20.

Table 4: Secondary equal-weight structural-response ranking on common model support. Each SF score is converted to a direction-corrected feature-local rank score before averaging equally across SF1–SF8; raw NRMSE values are not pooled.

SuiteEqual-weight diagnostic ranking
GIFT-Eval Short1. TimesFM 3.0 (0.925); 2. Toto 2.0 (0.625); 3. Chronos-2 (0.600); 4. TimesFM 2.5 (0.550); 5. Timer-3.5 (0.175); 6. Moirai 2 (0.125)
GIFT-Eval Medium1. TimesFM 3.0 (0.925); 2. Timer-3.5 (0.650); 3. Chronos-2 (0.600); 4. TimesFM 2.5 (0.375); 5. Toto 2.0 (0.300); 6. Moirai 2 (0.150)
GIFT-Eval Long1. TimesFM 3.0 (0.950); 2. Chronos-2 (0.600); 3. Timer-3.5 (0.600); 4. TimesFM 2.5 (0.400); 5. Toto 2.0 (0.375); 6. Moirai 2 (0.075)
fev-bench Mini-201. Chronos-2 (0.750); 2. TimesFM 3.0 (0.700); 3. TimesFM 2.5 (0.500); 4. Toto 2.0 (0.375); 5. Timer-3.5 (0.350); 6. Moirai 2 (0.325)

Table 4. Secondary equal-weight structural-response ranking on common model support. Each SF score is converted to a direction-corrected feature-local rank score before averaging equally across SF1–SF8; raw NRMSE values are not pooled.

Latest four-panel inventory and winners

Table 5 records the panel prediction counts. These panels retain the source-window, unordered-variant, and configuration aggregation supplied by their numerical inputs.

Table 5: Inventory of the four latest result panels after restricting each view to the comparison roster.

SuiteConfigsModels shownPanel predictions
GIFT-Eval Short5511399,740
GIFT-Eval Medium2110127,380
GIFT-Eval Long2110121,810
fev-bench Mini-202011121,902

Table 5. Inventory of the four latest result panels after restricting each view to the comparison roster.

Figure 4 and Figure 5 display the cell values. The main-paper radar restricts the NRMSE view to six models shared by all suites and uses a separate scale for each feature.

Intrinsic Model Selection

The regime-switching coverage table below belongs to an earlier frozen score-cache analysis. It retains that analysis’s seven-family slice, separately from the current intrinsic-selection result in the main paper.

The model-shortlisting analysis uses a frozen score-cache population distinct from the four latest suite panels. The cache contains 55 configurations, 19 source families, 15,673 common-scored original windows, and 6,016 capability–window rows; a capability row can correspond to the same original window as another capability. Twelve model columns are retained, while the recommendation analysis uses eight frozen foundation-model configurations. The analysis record retains their names and complete input order.

For the leave-source-family-out analysis, the held-out family is excluded from all A-side rankings and threshold fitting. The generic strategy uses all A-side original windows, the qualified strategy uses capability-qualified A-side original windows, and the augmented strategy averages the matched augmentation variants within each window and ranks by augmented MASE. Within each family, windows are averaged by configuration before equal-family Borda ranks are averaged across families. The A-side Q75 strength cutoff is applied to B; B is scored only on original windows and the family-balanced champion is determined after per-family configuration aggregation. The effect-NRMSE ranking is reported as a separate response diagnostic; MASE determines the recommendations.

The pooled case uses the underlying collection plus original item as the source-group key. All windows, frequency variants, and augmentation variants for one group remain on the same side. Five seeds and both directions yield ten A/B directions; Q75 is fitted on A and applied unchanged to B. The reported Bitbrains subset is an exploratory post-hoc slice selected after inspection of the completed grid and is evaluated only on original MASE. The ten directions share a source pool and are summarized descriptively; no significance test is applied, so this slice provides exploratory descriptive evidence. The A-only restriction applies to threshold fitting and ranking; upstream qualification retains its offline future-quality and feasibility checks.

Table 6: Winners for each structural feature in the four latest result panels. Entries give the model and the number of feature tracks won; ties would be reported jointly.

SuiteLowest Probe MASELowest Forecast-Change NRMSE
GIFT-Eval ShortTimesFM 3.0 (5/8); Toto 2.0 (2/8); Chronos-2 (1/8)TimesFM 3.0 (4/8); Toto 2.0 (2/8); Linear Trend (1/8); TimesFM 2.5 (1/8)
GIFT-Eval MediumTimesFM 3.0 (7/8); Chronos-2 (1/8)TimesFM 3.0 (6/8); Chronos-2 (1/8); Linear Trend (1/8)
GIFT-Eval LongTimesFM 3.0 (5/8); Timer-3.5 (2/8); Chronos-2 (1/8)TimesFM 3.0 (6/8); Linear Trend (1/8); Timer-3.5 (1/8)
fev-bench Mini-20TimesFM 3.0 (7/8); TiREx-2 (1/8)TimesFM 3.0 (3/8); Chronos-2 (1/8); Linear Trend (1/8); Naive (1/8); TimesFM 2.5 (1/8); Toto 2.0 (1/8)

Table 6. Winners for each structural feature in the four latest result panels. Entries give the model and the number of feature tracks won; ties would be reported jointly.

Table 7: Regime-switching candidate coverage in the leave-source-family-out analysis. A hit means that the B-side family-balanced original-MASE champion is present in the frozen Top-KK list.

RecommendationTop-1Top-3Top-5
Generic original MASE1/73/77/7
Qualified original MASE1/74/77/7
Capability-augmented MASE1/76/77/7

Table 7. Regime-switching candidate coverage in the leave-source-family-out analysis. A hit means that the B-side family-balanced original-MASE champion is present in the frozen Top-KK list.

Relational Feature Validation

The historical numerical block retains the input-removal analysis’s task counts and descriptive intervals. Its population is separate from the latest suite panels; the deletion and assessed-target contracts are specified in Appendix B.

Augmentation Faithfulness: Paired-Ranking Study

The small-model paired-ranking study contains five architectures, eight native GIFT-Eval configurations, three training seeds, and three augmentation seeds; the seven analyzed capabilities are trend, multi-seasonal, time-varying seasonality, regime switching, intermittency, common factor, and cross-series dependence. A is the sampled original-source union (2,294 windows), B is the per-capability subset of original windows whose paired treatment enters C, and C is the valid treatment set. Across capabilities there are 2,712 source pairs; the three augmentation seeds retain 2,468, 2,466, and 2,460 B pairs, with 2,436 common valid pairs and 21,793 valid treatment samples across the three rounds. These counts define the analyzed support for the paired-ranking result.

For each source, unordered treatment variants are averaged first, followed by equal source averaging within each configuration–feature cell, equal averaging over valid configurations within each feature, and equal averaging over the seven feature summaries; absent features are not imputed. B uses the same source membership, target mask, weights, and original full-history seasonal MASE denominator as C, rather than a denominator recomputed from the augmented history or the 512-point model context. Training-seed scores are averaged after the three augmentation seeds; the reported ±\pm values are sample SD over the three training seeds.

Training uses only the original train/validation splits with context 512 and horizon 48. Minimum original-validation MASE selects the checkpoint across eligible epochs and at most one prescribed regularized retry; selected weights are frozen before testing. Eight model trajectories used a retry and eight used a bounded refinement, while four selected trajectories met the late-overfit diagnostic and therefore used earlier validation-best checkpoints. The analysis reports selected-checkpoint behavior for this suite.

The rank uncertainty analysis uses 1,000 paired bootstrap replicates with fixed selected-suite item clusters, or circular blocks of four consecutive forecast origins for single-item configurations. Training and augmentation seeds are averaged before resampling. The conditional intervals are ρ∈[0.70,0.90]\rho\in[0.70, 0.90] and τb∈[0.60,0.80]\tau_b \in[0.60, 0.80]; the near-first C difference between iTransformer and DLinear has interval [−0.0269,0.0114][-0.0269, 0.0114], which spans zero under this resampling. Support is heterogeneous: the multivariate group has three native items, while common-factor and cross-series dependence have 59 and 39 source windows concentrated in three and two items, respectively. These counts define the scope of the selected-suite result. Effect NRMSE remains a separate response diagnostic, while original MASE supplies the ranking metric.

Table 8: Bitbrains exploratory case under Q75 family-Borda selection. Values are direction-equal descriptive means over ten disjoint source-group directions; the B-side support has at least 16 A groups and 14 B groups per direction.

RecommendationMean gain vs genericWins–ties–losses
Qualified original MASE11.02%–
Capability-augmented MASE9.06%8–1–1

Table 8.

Table 9 preserves the input-group and feature-level results; six of the seven features have ρ≥0.90\rho\ge0.90. Table 10 localizes selected trend changes, while the causes of the rank reversals remain unresolved.

Table 9: Small-model B-to-C rank agreement by input group and structural feature. Correlations use five architectures; the multivariate group has only three native items.

Group or structural featureSpearman ρ\rho
Multivariate1.00
Univariate0.30
Common factor1.00
Cross-series dependence1.00
Intermittency1.00
Regime switching1.00
Multi-seasonal0.90
Time-varying seasonality0.90
Trend−0.10

Table 9.

Stability under Benchmark Refresh

The ten-seed local-winner table below belongs to an earlier fixed ten-task panel. The main paper’s five-seed analysis uses the current 55-configuration Short panel and seven-model roster; the two seed studies retain separate supports.

Trade-off between Accuracy and Response Fidelity

The full checkpoint trajectories are retained below. The matched training conditions and checkpoint schedule are specified under Reproduction Levels and Procedures.

Historical and Expanded Numerical Blocks

The expanded Short results and the earlier six-model cross-suite results use separate populations and aggregation rules. The earlier ten-seed stability and input-removal tables retain their original support; the fine-tuning trajectories describe one training realization.

The expanded Short block pairs Linear Trend’s SF1 response NRMSE of 3.24×10−63.24 \times10^{-6} with Probe MASE of 8.074. Its input-removal estimates include an SF6 gap of 0.157 for TimesFM 3.0 and −0.020 for Chronos-2, whose 95% interval includes zero.

Earlier Six-Model Cross-Suite Data by Structural Feature

The expanded GIFT-Eval Short package is reported separately from the earlier six-model cross-suite block. The compact cross-suite tables below report Probe MASE / Forecast-Change NRMSE pairs (P/N); lower is better for each component, and NRMSE = 1 is the zero-response reference. The expanded Short tables then report Official MASE, matched original / augmented MASE, and Forecast-Change NRMSE in separate blocks. Values are configuration-equal after the package’s source-window and unordered-variant reductions.

Evaluation Coverage and Scope

Evaluation settings

Table 24 distinguishes the main panels from the supplementary analyses and records their support.

PanelEvaluation settingMain supportProvenance
GIFT-Eval Short latestFrozen latest Short panel55 configurations; 3 unordered variants; 11 modelsAnalysis record
GIFT-Eval Medium latestFrozen latest Medium panel21 configurations; 10 modelsAnalysis record
GIFT-Eval Long latestFrozen latest Long panel21 configurations; 10 modelsAnalysis record
fev-bench Mini-20 latestFrozen latest Mini-20 panel20 configurations; 11 modelsAnalysis record
Input removalMatched full-input and target-only viewsSame treatment contracts; assessed target history and future fixedSeparate block
Stability (latest)Five augmentation seeds on Short55 configurations; 7 modelsAnalysis record
Stability (earlier)Fixed ten-task Short panelTen augmentation seedsHistorical block
Fine-tuningOne shared generated corpus and two objectives11,269 samples; 10,000 steps; checkpoints every 1,000 stepsSingle realization
Model shortlistingFrozen window-level score cache15,673 common-scored windows; 55 configurations; 19 source families; 8-model rosterCPU reconstruction

Table 24. Evaluation panels and their declared support. Detailed source provenance and run identities are retained with the analysis record.

SFNChronos-2Timer-3.5TimesFM 2.5TiREx-2Moirai 2Toto 2.0
SF1182.132 / 0.0821.966 / 0.2431.898 / 0.2161.843 / 0.2562.152 / 0.2151.876 / 0.243
SF2202.370 / 0.7792.758 / 0.9462.499 / 0.8022.532 / 0.8444.171 / 1.0101.934 / 0.637
SF3191.526 / 0.5291.738 / 0.6741.462 / 0.6791.484 / 0.5801.853 / 0.7401.572 / 0.571
SF4201.703 / 0.0961.717 / 0.1331.627 / 0.0911.597 / 0.0711.910 / 0.1341.574 / 0.144
SF5202.929 / 0.9052.872 / 0.9142.833 / 0.8372.745 / 0.9724.211 / 0.9652.699 / 0.955
SF640.618 / 0.2570.644 / 0.3010.648 / 0.3330.658 / 0.2590.695 / 0.3530.625 / 0.291
SF782.895 / 0.9892.982 / 1.0432.896 / 1.1872.865 / 1.0053.060 / 1.0442.746 / 1.460
SF863.638 / 0.6263.833 / 0.5543.491 / 0.4953.485 / 0.8505.907 / 0.6843.522 / 0.632

Table 11. Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Short. Lower is better for each component; NRMSE = 1 is the zero-response reference.

The model-shortlisting analysis uses a frozen window-level score cache and history-profile strengths. Its reconstruction record contains the source-group mapping, analysis procedure, results, reference checks, and redacted provenance; the raw time series, point forecasts, model weights, and private inference endpoints are outside this compact record.

The small-model paired-ranking analysis separates training, generation, evaluation, and analysis inputs. The compact handoff contains stage-level summaries; a full end-to-end replay additionally requires the model checkpoints, full predictions, and source time series.

Summary of evidence

Table 25 connects each claim family to the corresponding evaluation evidence and presentation in the paper.

Claim familyEvaluation evidenceResult presentation
SF1–SF8 interventionsStructural constructions and their ordered control settingsFeature-level response profiles and heatmaps
Generated-sample accuracy and responseFour evaluation panels with suite-specific supportMASE and Forecast-Change NRMSE tables
Structural-response profilesFeature-local comparisons on the eligible model supportMain and supplementary heatmaps
Refresh stabilityFive-seed and earlier fixed-panel comparisonsStability summary and figure
Auxiliary-input utilityMatched full-input and target-only comparisonsAuxiliary-Input Gap table and figure
Training-objective trade-offTwo objectives trained on the same sample setFine-tuning curves and objective comparison

Table 25. Summary of evaluation evidence used for the current results.

Evaluation inventory and analysis coverage

Table 26 gives the evaluation inventory for the four main panels. Candidates and References count evaluated windows, generated samples count admitted materializations, and predictions count model forecasts. failure counts refer to issued requests, and metric-specific inapplicable cases remain separate from scored cases.

SuiteConfigsCandidates/ReferencesGenerated samplesModelsPredictionsFailuresHorizon
GIFT-Eval Short5515,74517,77311399,740030–60
GIFT-Eval Medium214,2596,87510127,3800480–600
GIFT-Eval Long214,1796,55210121,8100720–900
fev-bench Mini-20202,8095,60911121,90205–288

Table 26. Evaluation inventory across the four main panels.

The four latest panel summaries contribute 88, 80, 80, and 88 model–feature cells for Short, Medium, Long, and FEV, respectively, with the filtered prediction counts shown above. The stability block contains 35 overall seed–model rows in the seven-model view, with feature-level rows retained in the compact summaries. The input-removal block preserves task-level and coverage summaries under the retained model roster. The fine-tuning block contains 22 curve rows and two objective summaries, all from one training realization and a shared 11,269-sample corpus. These counts index downstream records: panel configuration is the latest reduction unit, seed indexes the observed stability panel, and the fine-tuning curves share one training realization. The analysis encodes unavailable, unsupported, not-run, failed, invalid, and metric-undefined states as nonnumeric and preserves their denominators; NRMSE additionally distinguishes scoreable instances from low-truth-effect and no-observed-effect states as defined in Appendix B.

ModelOfficialSF1SF2SF3SF4SF5SF6SF7SF8
Chronos-21.5672.208/2.2160.716/0.7301.232/1.2411.575/1.8590.871/0.8960.828/1.1940.773/0.8020.734/0.765
TimesFM 2.51.6012.140/2.2200.713/0.7221.244/1.2461.911/2.3180.969/1.0520.828/1.2320.782/0.8130.745/0.782
TimesFM 3.01.4642.003/2.0500.667/0.6861.166/1.1651.628/1.9160.885/0.9230.807/1.1880.748/0.7720.697/0.732
TiREx-21.6102.239/2.2160.706/0.7321.241/1.2461.722/1.9140.876/0.9210.838/1.2330.788/0.8170.751/0.801
Moirai 21.7012.258/2.2600.766/0.7971.353/1.3721.688/1.9980.904/1.0300.922/1.3270.824/0.8630.809/0.877

Table 12. Expanded GIFT-Eval Short full55 package: Official MASE and matched original / augmented MASE for the first five model configurations.

ModelOfficialSF1SF2SF3SF4SF5SF6SF7SF8
Timer-3.51.7572.534/2.6440.711/0.7411.185/1.1821.644/1.9310.865/0.8990.855/1.2600.797/0.8280.723/0.772
Toto 2.01.4521.971/2.0650.673/0.7001.186/1.1771.580/1.8480.934/0.9630.837/1.2200.771/0.7930.690/0.726
Naive2.5793.531/4.3701.543/1.8912.041/2.0152.210/2.5951.064/1.2371.215/1.7791.171/1.2391.678/2.001
Seasonal Naive2.1932.945/3.9201.276/1.5491.722/1.7281.953/2.3131.045/1.2541.143/1.6591.086/1.1371.250/1.335
Drift2.5653.431/3.4311.542/1.8972.060/2.0362.631/3.2691.100/1.2831.248/1.8211.191/1.2581.686/2.012
Linear Trend4.8068.074/8.0742.211/2.4492.566/2.5855.219/7.1051.599/1.8672.155/3.0161.905/2.0281.841/2.168

Table 13. Expanded GIFT-Eval Short full55 panel: Official MASE and matched original / augmented MASE for the remaining six model configurations.

ModelSF1SF2SF3SF4SF5SF6SF7SF8
Chronos-21.3710.4370.6550.4010.9910.3920.4270.420
TimesFM 2.51.5060.4720.6770.3960.9010.4190.5370.475
TimesFM 3.01.1790.3400.6240.3870.9280.3920.4030.389
TiREx-21.1190.5340.6500.4030.9410.4180.4970.504
Moirai 22.1090.6090.6420.4191.0240.4490.6090.617

Table 14. Expanded GIFT-Eval Short full55 package: Forecast-Change NRMSE by structural feature for the first five model configurations.

ModelSF1SF2SF3SF4SF5SF6SF7SF8
Timer-3.52.2290.4980.6780.4230.9650.4420.5450.454
Toto 2.01.6110.4790.6240.3970.9510.4220.4330.386
Naive1.0001.1560.7500.5601.0910.5540.6500.789
Seasonal Naive2.2131.0200.7470.5321.0680.5630.6280.760
Drift3.42×10−63.42 \times10^{-6}1.1750.7530.5781.1120.5600.6570.794
Linear Trend3.24×10−63.24 \times10^{-6}1.0791.0020.7111.1740.9201.3540.880

Table 15. Expanded GIFT-Eval Short full55 panel: Forecast-Change NRMSE by structural feature for the remaining six model configurations.

ModelSFTasksΔ\DeltaMASE95% interval
Chronos-2SF612-0.020[-0.113, 0.052]
Chronos-2SF7120.000[-0.119, 0.113]
Chronos-2SF880.006[-0.009, 0.013]
TimesFM 3.0SF6130.157[0.064, 0.259]
TimesFM 3.0SF7110.051[-0.020, 0.160]
TimesFM 3.0SF870.003[-0.000, 0.009]
Toto 2.0SF613-0.022[-0.163, 0.081]
Toto 2.0SF7120.033[-0.010, 0.107]

Table 16. Expanded GIFT-Eval Short full55 input-removal results. Positive Δ\DeltaMASE favors retaining the auxiliary input; intervals are descriptive task-bootstrap 95% intervals.

SFNNChronos-2Timer-3.5TimesFM 2.5TiREx-2Moirai 2Toto 2.0
SF192.982 / 0.0512.808 / 0.2642.880 / 0.259–3.469 / 0.29312.785 / 0.590
SF2115.309 / 1.1025.533 / 1.2064.016 / 0.961–10.671 / 1.72910.488 / 1.131
SF3107.051 / 0.7877.292 / 1.1303.507 / 0.980–10.616 / 1.5659.955 / 1.296
SF4112.378 / 0.0532.581 / 0.1912.453 / 0.131–4.026 / 0.2767.597 / 0.176
SF5112.851 / 1.4023.039 / 1.7382.936 / 1.322–3.592 / 1.1156.459 / 1.556
SF640.919 / 0.2570.952 / 0.4740.966 / 0.524–1.324 / 0.5711.163 / 0.377
SF767.690 / 0.9928.041 / 1.0708.089 / 1.053–9.902 / 1.05115.988 / 1.189
SF845.151 / 0.5336.848 / 0.4986.849 / 0.373–9.773 / 0.45719.072 / 0.550

Table 17. Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Medium. Lower is better for each component; NRMSE = 1 is the zero-response reference.

SFNNChronos-2Timer-3.5TimesFM 2.5TiREx-2Moirai 2Toto 2.0
SF1103.597 / 0.0623.549 / 0.1913.682 / 0.124–4.609 / 0.21316.115 / 0.257
SF2116.252 / 0.9146.780 / 1.1684.890 / 0.966–9.542 / 1.63821.869 / 0.998
SF3104.860 / 0.8598.021 / 1.2235.632 / 0.961–9.717 / 1.19023.432 / 1.119
SF4112.934 / 0.0643.100 / 0.1753.253 / 0.106–4.442 / 0.32616.994 / 0.265
SF5113.131 / 1.0513.087 / 1.1183.185 / 1.069–4.636 / 1.68316.407 / 1.184
SF641.039 / 0.3851.144 / 0.7391.113 / 0.772–1.749 / 1.0841.324 / 0.663
SF766.669 / 0.8937.005 / 0.9957.446 / 1.058–10.128 / 1.01936.761 / 1.326
SF839.909 / 0.39713.459 / 0.5949.512 / 0.335–14.300 / 0.49053.238 / 0.857

Table 18. Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Long. Lower is better for each component; NRMSE = 1 is the zero-response reference.

SFNNChronos-2Timer-3.5TimesFM 2.5TiREx-2Moirai 2Toto 2.0
SF1181.938 / 0.1761.807 / 0.1811.469 / 0.1111.736 / 0.1301.625 / 0.0871.591 / 0.124
SF2202.656 / 1.2023.003 / 1.1432.582 / 0.9542.703 / 1.0423.144 / 1.2131.839 / 0.809
SF3202.190 / 0.8122.140 / 0.7561.744 / 0.7342.040 / 0.7942.045 / 0.7971.659 / 0.663
SF4201.913 / 0.2131.917 / 0.1631.562 / 0.1201.853 / 0.1361.647 / 0.1031.603 / 0.110
SF5182.188 / 0.9341.987 / 0.9631.712 / 0.8642.154 / 0.8942.055 / 1.0621.731 / 0.852
SF691.766 / 0.2862.044 / 0.3591.630 / 0.3011.819 / 0.2911.791 / 0.3801.663 / 0.295
SF791.766 / 0.2862.100 / 0.7821.678 / 0.7631.693 / 0.7171.711 / 0.8341.826 / 0.752
SF883.073 / 0.7763.046 / 0.6422.480 / 0.6912.972 / 0.6513.025 / 0.6742.303 / 0.551

Table 19. Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on fev-bench Mini-20. Lower is better for each component; NRMSE = 1 is the zero-response reference.

SuiteModelsρR,P\rho_{R,P}ρR,N\rho_{R,N}ρP,N\rho_{P,N}
GIFT-Eval Short60.943-0.0860.029
GIFT-Eval Medium50.7000.8000.800
GIFT-Eval Long50.9000.8000.900
fev-bench Mini-2060.6570.5430.771

Table 20. Spearman rank correlations among Reference MASE (R), Probe MASE (P), and NRMSE (N) for the displayed model sets; the table reports point correlations, with interval estimation treated separately.

SFUnanimous / 5Mean modalMin modalSetting-modal models
SF151.001.00Chronos-2 (5)
SF200.820.70TimesFM 2.5 (1), Toto 2.0 (4)
SF351.001.00Chronos-2 (5)
SF400.680.50TiREx-2 (3), TimesFM 2.5 (1), Chronos-2 (1)
SF500.740.50TimesFM 2.5 (4), Chronos-2 (1)
SF600.640.50Chronos-2 (5)
SF700.600.50TiREx-2 (4), Chronos-2 (1)
SF820.860.70TimesFM 2.5 (5)

Table 21. Local-winner stability across ten GIFT-Eval Short seeds for the structural features. Unanimous settings retain one lowest-NRMSE model in all seeds; modal fractions are descriptive.

Table 22: Complete Auxiliary-Input Gap results for the evaluated suite–model–feature cells. Each estimate is MASEablated−MASEfull\mathrm{MASE}_{\mathrm{ablated}}-\mathrm{MASE}_{\mathrm{full}} with a descriptive task-bootstrap 95% interval; positive values favor the full auxiliary input.

SuiteSFModelTasksΔMASE\Delta\mathrm{MASE} [95% interval]Task medianPositive tasks
GIFT-Eval ShortSF6Chronos-240.038 [0.009, 0.068]0.0423/4
GIFT-Eval ShortSF6TiREx-24-0.004 [-0.006, -0.001]-0.0051/4
GIFT-Eval ShortSF6Toto 2.040.005 [-0.012, 0.030]-0.0042/4
GIFT-Eval ShortSF7Chronos-280.012 [-0.001, 0.030]0.0026/8
GIFT-Eval ShortSF7TiREx-280.003 [-0.002, 0.008]0.0015/8
GIFT-Eval ShortSF7Toto 2.080.031 [-0.002, 0.077]-0.0014/8
GIFT-Eval ShortSF8Chronos-26-0.001 [-0.012, 0.010]0.0014/6
GIFT-Eval ShortSF8TiREx-260.001 [−1.21×10−4-1.21 \times10^{-4}, 0.003]2.29×10−42.29 \times10^{-4}4/6
GIFT-Eval MediumSF6Chronos-24-0.007 [-0.082, 0.050]0.0123/4
GIFT-Eval MediumSF6Toto 2.04-0.024 [-0.095, 0.038]-0.0142/4
GIFT-Eval MediumSF7Chronos-260.005 [-0.048, 0.048]0.0114/6
GIFT-Eval MediumSF7Toto 2.06-0.039 [-0.093, 0.002]-0.0161/6
GIFT-Eval MediumSF8Chronos-240.011 [-0.002, 0.033]0.0012/4
GIFT-Eval LongSF6Chronos-240.034 [-0.064, 0.153]0.0113/4
GIFT-Eval LongSF6Toto 2.04-0.012 [-0.029, 0.005]-0.0121/4
GIFT-Eval LongSF7Chronos-26-0.006 [-0.092, 0.082]-0.0142/6
GIFT-Eval LongSF7Toto 2.060.195 [-0.034, 0.629]-0.0083/6
GIFT-Eval LongSF8Chronos-23-0.005 [-0.010, -0.002]-0.0030/3
fev-bench Mini-20SF6Chronos-280.048 [-0.005, 0.128]0.0036/8
fev-bench Mini-20SF6TiREx-28-0.020 [-0.056, 0.001]-0.0032/8
fev-bench Mini-20SF6Toto 2.080.135 [-0.010, 0.402]-0.0014/8
fev-bench Mini-20SF7Chronos-280.013 [0.004, 0.024]0.0097/8
fev-bench Mini-20SF7TiREx-28-0.010 [-0.029, 0.003]0.0015/8
fev-bench Mini-20SF7Toto 2.080.505 [0.002, 1.495]0.0147/8
fev-bench Mini-20SF8Chronos-282.935 [-0.299, 8.982]0.1505/8
fev-bench Mini-20SF8TimesFM 2.560.037 [0.004, 0.079]0.0285/6
fev-bench Mini-20SF8TiREx-280.038 [-0.530, 0.706]0.0045/8

Table 22.

Table 23: Complete single-realization Chronos-2 trajectories for official quantile-loss and Forecast-Change NRMSE objectives. Values use the panel aggregation on training seed 2026092406 and the same 11,269 samples. This block reports one training seed, so repeated-seed uncertainty is a separate analysis.

StepOfficial MASEOfficial FC-NRMSEFC-NRMSE-objective MASEFC-NRMSE-objective
00.7250.6410.7250.641
1k0.6810.5930.7780.533
2k0.6650.5560.8040.488
3k0.6550.5820.8160.473
4k0.6450.5870.8150.483
5k0.6410.6060.8140.477
6k0.6350.6050.8190.475
7k0.6320.6070.8200.476
8k0.6300.6140.8190.478
9k0.6290.6160.8200.471
10k0.6280.6150.8200.473

Table 23.

Reproduction Levels and Procedures

This subsection defines three reproducibility levels: checking the consistency of reported summaries, reconstructing the tables and figures from compact numerical summaries, and repeating the complete data construction and forecasting evaluation. The first two levels are the reconstruction scope described by the compact records; the third states the inputs required for a fresh end-to-end run.

Reproduction levels

LevelDeclared environmentPurpose
Summary consistencyLinux or macOS; CPU; less than 1 GB memory and diskChecks row counts, dimensions, and numerical agreement for representative tables and figures
Table and figure reconstructionLinux or macOS; Python 3.11 or later; 4 CPU cores, 8 GB RAM, and 5 GB free disk recommendedRecomputes the four analysis blocks reported in the paper from compact numerical summaries
End-to-end evaluationLinux x86-64; Python 3.12; CUDA 12.8-compatible NVIDIA driver; 32 GB VRAM per worker; 16 CPU cores and 64 GB RAM recommendedDefines the environment for reconstructing treatments, evaluating the models, and recomputing the reported analyses

Table 27. Reproduction levels and resource requirements.

Reconstruction procedure

Summary consistency checks compare the numerical summaries with the displayed tables and figures. Table and figure reconstruction recomputes the four main evaluation summaries, the auxiliary-input comparison, the stability analysis, and the fine-tuning curves from compact numerical inputs. End-to-end evaluation is the broader procedure: it requires the benchmark data, model checkpoints, full predictions, and computational resources needed to repeat profiling, augmentation, inspection, forecasting, and metric aggregation.

The model-shortlisting package provides a CPU-only reconstruction route from the frozen window-level score cache. It rebuilds the source splits, leave-family rankings, pooled sensitivity grid, tables, figures, and verification without CaFE, GPU hardware, model weights, raw time series, or network access. Its numerical checks reproduce the frozen package outputs for the declared reconstruction scope.

Chronos-2 objective comparison

The Chronos-2 comparison uses the same 11,269-sample corpus for the standard quantile-loss and response-focused objectives. Both branches use the same learning rate, model modules, training seed, schedule, context lengths, and evaluation conditions, with checkpoints every 1,000 steps through 10,000 steps. The comparison therefore isolates the change in training objective for this realization.

References

  1. [1]Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. GIFT-Eval: A benchmark for general time series forecasting model evaluation. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024. URL https://openreview.net/forum?id=Z2cMO0ANfX.arxiv.org/abs/2410.10393
  2. [2]Yihao Ang, Qiang Huang, Yifan Bao, Anthony K. H. Tung, and Zhiyong Huang. TSGBench: Time series generation benchmark. Proceedings of the VLDB Endowment, 17(3):305–318, 2023. doi: 10.14778/3632093.3632097. URL https://www.vldb.org/pvldb/vol17/p305-huang.pdf.DOI
  3. [3]Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=gerNCVqgtR.arxiv.org/abs/2403.07815
  4. [4]Abdul Fatir Ansari, Oleksandr Shchur, Jaris Kük­en, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Pablo Guerron, Tony Hu, Junming Yin, Nick Erickson, Prateek Mutalik Desai, Hao Wang, Huzefa Rangwala, George Karypis, Yuyang Wang, and Michael Bohlke-Schneider. Chronos-2: From univariate to universal forecasting, 2025. URL https://arxiv.org/abs/2510.15821.
  5. [5]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pp. 10148–10167, Vienna, Austria, July 2024. PMLR. URL https://proceedings.mlr.press/v235/das24c.html.arxiv.org/abs/2310.10688
  6. [6]Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. TimeVAE: A variational auto-encoder for multivariate time series generation, 2021. URL https://arxiv.org/abs/2111.08095.
  7. [7]Cyril Garcia and Guillaume Remy. Statistical benchmarking of transformer models in low signal-to-noise time-series forecasting, 2026. URL https://arxiv.org/abs/2602.09869.
  8. [8]Azul Garza, Renée Rosillo, Rodrigo Mendoza-Smith, David Salinas, Andrew Robert Williams, Arjun Ashok, Mononito Goswami, and José Martín Juárez. Impermanent: A live benchmark for temporal generalization in time series forecasting. In ICLR 2026 Workshop on Time Series in the Age of Large Models, 2026. URL https://openreview.net/forum?id=EOoinHRJOP.
  9. [9]Google Research. TimesFM model card, 2025. URL https://huggingface.co/google/timesfm-2.5-200m-pytorch. Checkpoint: timesfm-2.5-200m-pytorch.
  10. [10]Syamantak Datta Gupta. Evaluating time series foundation models on noisy periodic time series, 2025. URL https://arxiv.org/abs/2501.00889.
  11. [11]Md Rakibul Haque, Vishwa Goudar, Shireen Elhabian, and Warren Woodrich Pettine. TimeSynth: A framework for uncovering systematic biases in time series forecasting, 2026. URL https://arxiv.org/abs/2602.11413.
  12. [12]Rob J. Hyndman and Anne B. Koehler. Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4):679–688, 2006. doi: 10.1016/j.ijforecast.2006.03.001. URL https://doi.org/10.1016/j.ijforecast.2006.03.001.
  13. [13]Yanfei Kang, Rob J. Hyndman, and Feng Li. GRATIS: GeneRativE Time Series with diverse and controllable characteristics. Statistical Analysis and Data Mining: The ASA Data Science Journal, 13(4):354–376, 2020. doi: 10.1002/sam.11461. URL https://doi.org/10.1002/sam.11461.
  14. [14]Emaad Khwaja, Chris Lettieri, Gerald Woo, Eden Belouadah, Marc Cenac, Guillaume Jarry, Enguerrand Paquin, Xunyi Zhao, Viktoriya Zhukov, Othmane Abou-Amal, Chenghao Liu, Ameet Talwalkar, and David Asker. Toto 2.0: Time series forecasting enters the scaling era, 2026. URL https://arxiv.org/abs/2605.20119.
  15. [15]Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 95–104, Ann Arbor, MI, USA, 2018. Association for Computing Machinery. doi: 10.1145/3209978.3210006. URL https://doi.org/10.1145/3209978.3210006.
  16. [16]Zhe Li, Xiangfei Qiu, Peng Chen, Yihang Wang, Hanyin Cheng, Yang Shu, Jilin Hu, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, and Bin Yang. TSFM-Bench: A comprehensive and unified benchmark of foundation models for time series forecasting. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 5595–5606, New York, NY, USA, 2025. Association for Computing Machinery. doi: 10.1145/3711896.3737442. URL https://doi.org/10.1145/3711896.3737442.
  17. [17]Chenghao Liu, Taha Aksu, Juncheng Liu, Xu Liu, Hanshu Yan, Quang Pham, Silvio Savarese, Doyen Sahoo, Caiming Xiong, and Junnan Li. Moirai 2.0: When less is more for time series forecasting, 2025. URL https://arxiv.org/abs/2511.11698. Evaluated checkpoint: Moirai-2.0-R-Small; model card: https://huggingface.co/Salesforce/moirai-2.0-R-small.
  18. [18]Yong Liu, Xingjian Su, Shiyu Wang, Haoran Zhang, Haixuan Liu, Yuxuan Wang, Zhou Ye, Yang Xiang, Jianmin Wang, and Mingsheng Long. Timer-s1: A billion-scale time series foundation model with serial scaling, 2026. URL https://arxiv.org/abs/2603.04791. Evaluated checkpoint: Timer-S1; model card: https://huggingface.co/thuml/Timer-S1.
  19. [19]Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54–74, 2020. doi: 10.1016/j.ijforecast.2019.04.014. URL https://doi.org/10.1016/j.ijforecast.2019.04.014.
  20. [20]Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4):1346–1364, 2022. doi: 10.1016/j.ijforecast.2021.11.013. URL https://doi.org/10.1016/j.ijforecast.2021.11.013.
  21. [21]Marcel Meyer, Sascha Benjamin Kaltenpoth, Henrik Albers, Kevin Zalipski, and Oliver Müller. TS-Arena – a live forecast pre-registration platform. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, pp. 9558–9568. Association for Computing Machinery, 2026. doi: 10.1145/3770855.3817515. URL https://doi.org/10.1145/3770855.3817515.

Paper details

vlnc:W_fs27xd8v
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents