betaFeedback

WorksComputer Science

Breaking feature collapse in self-supervised time series encoders

Abstract

General-purpose time series encoders promise a single backbone that generalises across tasks and domains; however, they currently do not scale. We trace this down to a critical point of failure: under the predominant masked data modelling supervision, the encoder's output tokens partially collapse. This is the natural consequence of supervising encoders only in the input space, which leaves no constraint against redundant token dimensions in the feature space. We introduce OTISv2, which suppresses the collapse by complementing the reconstruction objective in input space with two objectives in the feature space: a self-distillation loss against an exponential-moving-average teacher, and a KoLeo regulariser. Under matched training, the resulting encoder reduces the cosine similarity and lifts the effective rank of its output tokens relative to its predecessor, and it scales monotonically from 7.1 M to 116 M parameters where its predecessor declines. It matches or surpasses existing general-purpose time series encoders across 143 uni- and multi-variate datasets. To foster further research, we release our code and pre-trained weights.

Introduction

In recent years, general-purpose backbones have transformed natural language processing (NLP) and computer vision (CV). In CV, for instance, the DINO family [3, 26, 30] provides dense features that transfer, without retraining, to classification, segmentation, depth estimation, and retrieval across natural, satellite, and medical imaging. Time series analysis is now undergoing the same transition: general-purpose encoders like UniTS [8], MOMENT [11], and OTIS [34] have been proposed as backbones that can serve various tasks across domains as diverse as electrocardiography (ECG), electroencephalography (EEG), audio, and weather. These backbones are pre-trained via masked data modelling: patches of their input are hidden, they learn to reconstruct them, and dense features that transfer are assumed to follow. The most effective family on this frontier is currently OTIS, which matches much larger families using only 7.1 M parameters by explicitly tailoring tokenisation, masking, and the reconstruction objective to time series properties.

Observation. On closer inspection, however, general-purpose time series encoders currently do not scale. At the frontier, OTIS-7.1M comes within 0.6 percentage points (pp) of MOMENT-308M in downstream performance across 143 datasets, despite being 43× smaller; a tiny encoder rivalling a much larger one, the opposite of the scaling trend observed in NLP and CV. Within each family the picture is the same: OTIS gains only 0.8 pp from 43 M to 128 M parameters, MOMENT 0.4 pp from 85 M to 308 M, and UniTS even loses 0.2 pp from 1.1 M to 3.5 M (Figure 1). A universal backbone should improve noticeably with capacity; current general-purpose time series encoders do not.

Figure 1. General-purpose time series encoders saturate at scale. Average accuracy over 23 UEA and 120 UCR datasets. Across three general-purpose encoder families, gains through parameter scaling saturate. OTIS-7.1M nearly matches the 43× larger MOMENT-308M.

Analysis. We trace this scaling failure down to the feature space: the encoder’s output tokens partially collapse, being highly similar to one another and spanning only a fraction of the available dimensions (Figure 2). This is a natural consequence of supervising the encoder only in input space: masked data modelling rewards per-patch reconstruction accuracy but places no explicit constraint on the feature space, so nothing prevents the encoder from routing information through a small set of token dimensions. The resulting cap on the effective rank contributes to the scaling plateau, and the few surviving dimensions are easy for specialised encoders to match on narrow benchmarks.

Figure 2. Feature quality and its consequences for downstream performance. Pre-training of OTIS and OTISv2 with the same data, width, depth, optimiser, and schedule. Both use C=6ND=6.82×1016C = 6ND = 6.82 \times10^{16} FLOPs at N=7.1N = 7.1 M parameters, D=1.60D = 1.60 B training tokens, and DC=32D_C = 32 M unique tokens. (Left) Patch-token cosine similarity (↓\downarrow). (Centre) RankMe (↑\uparrow) normalised by the encoder width d=192d = 192. (Right) UEA 5-shot balanced accuracy on 23 datasets (↑\uparrow). Token similarity drops in both encoders within the first 200 M tokens, then stays at 0.46 in OTIS but declines to 0.32 in OTISv2, whose 5-shot lead grows to 5 to 7 pp by 1.60 B tokens.

Solution. To prevent this feature collapse, we add two feature-space objectives that operate directly on the encoder and its output tokens, complementing the existing reconstruction objective on the decoder: (1) a self-distillation loss in the spirit of DINO [3] and iBOT [44], which matches student output tokens to those of an exponential-moving-average teacher, supplying a rich signal that explicitly shapes the feature space; and (2) a KoLeo regulariser [28] following DINOv2 [26], which spreads the [CLS] features of different samples over the unit hypersphere and thus penalises near-duplicate features across samples. The result is an objective that now supervises both the input space and feature space of an encoder, which is a prerequisite for positive scaling effects.

Outcomes. This yields OTISv2, a 7.1M encoder which matches or surpasses general-purpose encoders up to 43× larger across 23 multi-variate UEA [2] and 120 uni-variate UCR [5] datasets (§ 4.1); whose token similarity falls and whose effective rank rises relative to its predecessor under matched training (§ 4.2); and which converts parameters ranging from 7.1 M to 116 M into downstream performance gains, where its predecessor declines (§ 4.5).

Related Work

Self-Supervised Learning for Time Series. Early time series work relied on contrastive learning, aligning augmented views of a sample in the feature space [19, 41, 38, 37, 33], but identifying effective augmentations is highly domain-dependent and limits contrastive approaches across heterogeneous corpora. Masked data modelling has therefore emerged as the dominant paradigm: by reconstructing masked segments, current methods such as PatchTST [25], UniTS [8], MOMENT [11], and OTIS [34] learn features without hand-crafted augmentations. However, their training signal lives entirely in the input space, supervising the encoder only indirectly towards semantically consistent features.

Supervision in Feature Space. In computer vision, general-purpose encoders are directly supervised in the feature space: DINO [3] supervises a student’s [CLS] token features to match the output distribution of an exponential-moving-average teacher, iBOT [44] extends this training signal to patch tokens, and DINOv2 [26] combines the two at scale and adds a KoLeo regulariser [28] to spread features over the unit hypersphere. OTISv2 brings these feature-space objectives to time series, motivated by concrete failure modes of existing general-purpose time series encoders.

General-Purpose Backbones for Time Series. Recent research aims to unify time series analysis under a single backbone. UniTS [8] uses task prompts within a 3.5 M backbone, risking learning fixed prompt-to-task mappings rather than generalisable features. MOMENT [11] pairs a 308 M encoder with a linear 8 k decoder, a capacity mismatch that may weaken the training signal. OTIS [34] reaches competitive performance at only 7.1 M parameters, but requires domain labels at tokenisation.

Scaling and Feature Collapse. Progress in time series analysis has been driven by the scaling laws hypothesis [18, 15], increasing the parameter counts of backbones to address the heterogeneity of multi-domain time series data. Feature collapse under self-supervised learning is a known cause of stalled scaling in other modalities, and the remedies that have proven effective at scale act directly on the encoder’s output: teacher-based centring [3, 44, 26, 30] and entropy-maximising spreads such as KoLeo [28, 26, 30]. We show that the same pathology manifests in self-supervised learning for time series and contributes to the observed scaling saturation in general-purpose time series encoders.

Methodology

To build OTISv2, we start from the OTIS recipe (§ 3.1) and extend it in two ways. First, to remove the dependency on domain labels at tokenisation, we replace its domain-specific variate embeddings and absolute position embeddings with register tokens and axial RoPE (§ 3.2). Second, to counteract the partial collapse of output tokens under masked data modelling, we add a self-distillation loss and a KoLeo regulariser that also supervise the encoder in feature space (§ 3.3).

Architecture

OTISv2 builds on the Transformer [36] encoder and decoder of OTIS [34]. The encoder f(⋅)f(\cdot) has 7.1 M parameters, 12 blocks of width d=192d=192, three heads, and a 768-wide MLP. The decoder g(⋅)g(\cdot) has 1.7 M parameters and is used only during pre-training, with four blocks of width 160, five heads, and a 640-wide MLP. Each variate is split into non-overlapping patches of p=24p=24 time points, so a sample with VV variates and Tˉ\bar{T} time points yields Np=V⋅T′N_p=V\cdot T' tokens, where T′=Tˉ/pT'=\bar{T}/p. The pre-training context spans Tˉ=1,008\bar{T}=1{,}008 time points, which gives T′=42T'=42 patches per variate.

Two changes act inside every block. A SwiGLU MLP [29] replaces the GELU MLP of OTIS. The attention and MLP output projections are zero-initialised, so each block starts as the identity [13, 26].

Tokenisation

OTIS adds two position embeddings to each patch token. A fixed sinusoidal embedding encodes time, and a learnable embedding encodes variates within a domain. The learnable variate embedding thus ties the tokeniser to domain labels. OTISv2 removes this dependency, so no domain label is needed at any stage. To let the encoder store global information outside the patch tokens, we instead add register tokens [4] to the input sequence. The patch tokens can thus retain the local waveform information. Axial rotary position embedding (RoPE) [31, 14] replaces both absolute position embeddings, so that position enters attention only through the relative offset between patches.

Register Tokens. Vision Transformers [7] can repurpose patch tokens to store and process global information, at the cost of the local information these tokens carry. Register tokens [4] provide dedicated slots for such global computation. We include them to separate global from local information. The registers can hold information about the sample or its domain, while the patch tokens are free to describe the local waveform structure. This separation matters for OTISv2 in particular, since it has no domain-specific variate embeddings to carry such information. We append R=4R=4 learnable register tokens, denoted [reg], to the input sequence after patch embedding, alongside the [CLS] token. Like the [CLS] token, they are initialised from N(0,0.0222)\mathcal{N}(0,0.022^2) and have no grid position. At the output, they are discarded, so downstream tasks utilise only [CLS] and patch tokens.

Axial Rotary Position Embedding. Rotary position embedding (RoPE) [31] rotates each query and key vector by angles that depend on its absolute position. For a query and key qn,kn′∈Rdh\mathbf{q}_n,\mathbf{k}_{n'} \in\mathbb{R}^{d_h} of an attention head at 1D integer positions n,n′n,n', it groups consecutive dimensions into pairs, treats the jj-th pair as a complex number with frequency ωj\omega_j, and maps qn,j↦qn,jeinωj\mathbf{q}_{n,j} \mapsto\mathbf{q}_{n,j}e^{in\omega_j} and kn′,j↦kn′,jein′ωj\mathbf{k}_{n',j} \mapsto\mathbf{k}_{n',j}e^{in'\omega_j}. The attention logit, proportional to Re⁡[∑jqn,jkn′,j∗ei(n−n′)ωj]\operatorname{Re}\left[\sum_j \mathbf{q}_{n,j}\mathbf{k}_{n',j}^{*}e^{i(n-n')\omega_j}\right], depends only on the relative offset n−n′n-n'. Axial RoPE [14] extends this to 2D inputs by applying RoPE along each axis on a separate half of the head dimension. It makes the encoder independent of the sequence length and the variate count. A relative encoding of time lets the encoder process series of any length, without interpolating position embeddings at inference. Likewise, a relative encoding of the variate index distinguishes the variates of a sample without tying them to a domain, and it extends to any number of variates. With dh=64d_h=64, the first half of the dimensions rotate with the variate index v∈{0,…,V−1}v \in\{0,\ldots,V-1\} of a patch and the last half with its time index t∈{0,…,T′−1}t \in\{0,\ldots,T'-1\}. Each half thus has 32 dimensions, or 16 pairs, with frequencies ωj=Ω−j/16\omega_j=\Omega^{-j/16} for j=0,…,15j=0,\ldots,15 and the base Ω=2×105\Omega=2\times10^5. The [CLS] and [reg] tokens have no grid position and are not rotated.

Pre-Training

Following OTIS, we pre-train the encoder via masked data modelling. Specifically, a subset of patches is masked, the encoder processes only the visible patches, and a decoder reconstructs all patches in input space. We keep the dual masking strategy [34] of OTIS, which applies post-fix masking with probability π=0.33\pi=0.33 and random masking otherwise. We use the masking ratio ρs=0.7\rho_s=0.7 for random masking and ρs=0.5\rho_s=0.5 for post-fix masking, and perturb both with uniform noise from U(0,0.15)\mathcal{U}(0,0.15). Let xi,x^i∈R1×p\mathbf{x}_i,\hat{\mathbf{x}}_i \in\mathbb{R}^{1\times p} denote the ii-th input patch and its reconstruction, respectively. The reconstruction loss optimised is the mean squared error between reconstruction and input,

LMSE=1Npp∑i=1Np∥xi−x^i∥22∈[0,∞).(1)\mathcal{L}_{\mathrm{MSE}}=\frac{1}{N_p p}\sum_{i=1}^{N_p}\left\lVert\mathbf{x}_i-\hat{\mathbf{x}}_i\right\rVert_2^2\in[0,\infty). \tag*{(1)}

Additionally, a shape mismatch between reconstruction and input is penalised with a normalised cross-correlation (NCC) regulariser. For two multi-variate time series X,X^∈RV×Tˉ\mathbf{X},\hat{\mathbf{X}}\in\mathbb{R}^{V\times\bar{T}} it is defined as

LNCC=1−NCC⁡(X,X^)∈[0,2],NCC⁡(X,X^)=1VTˉ∑v=1V∑t=1Tˉxv,t−μvσvx^v,t−μ^vσ^v,(2)\mathcal{L}_{\mathrm{NCC}}=1-\operatorname{NCC}(\mathbf{X},\hat{\mathbf{X}})\in[0,2],\quad\operatorname{NCC}(\mathbf{X},\hat{\mathbf{X}})=\frac{1}{V\bar{T}}\sum_{v=1}^{V}\sum_{t=1}^{\bar{T}}\frac{x_{v,t}-\mu_v}{\sigma_v}\frac{\hat{x}_{v,t}-\hat{\mu}_v}{\hat{\sigma}_v}, \tag*{(2)}

where μv,σv\mu_v,\sigma_v and μ^v,σ^v\hat{\mu}_v,\hat{\sigma}_v denote the mean and standard deviation of variate vv in X\mathbf{X} and X^\hat{\mathbf{X}} over their Tˉ\bar{T} time points. Both objectives, LMSE\mathcal{L}_{\mathrm{MSE}} and LNCC\mathcal{L}_{\mathrm{NCC}}, are inherited from OTIS and supervise the encoder in input space. The following two objectives add supervision in feature space.

Self-Distillation. Inspired by DINO [3] and iBOT [44], we add self-distillation to supervise both the [CLS] and patch tokens of the encoder. A student fθsf_{\theta_s} and a teacher fθtf_{\theta_t} share the encoder architecture. The teacher receives no gradients and its weights are an exponential moving average (EMA) of the student weights, θt←mθt+(1−m)θs\theta_t\leftarrow m\theta_t+(1-m)\theta_s, where momentum mm follows a cosine schedule from 0.996 to 1. Given a sample, the student processes two independently masked views u,u′u,u' with masking ratio ρs\rho_s and sees only the visible patches. The decoder gg reconstructs view uu only. The teacher processes less masked views with masking ratio ρt≤ρs\rho_t\leq\rho_s, whose visible patches are a superset of the student’s. Both fθsf_{\theta_s} and fθtf_{\theta_t} have two projection heads, one each for the [CLS] and patch tokens, each with K=8,192K=8{,}192 outputs. We denote them by hsh_s and hth_t and mark the token type of their distributions with the superscript [CLS] or patch. Like fθtf_{\theta_t}, hth_t is an EMA of hsh_s with the same momentum mm. For an output token z∈Rdz\in\mathbb{R}^d of the respective encoder, the student and teacher distributions are

ps=softmax⁡(hs(z)/τs)∈RK,pt=softmax⁡((ht(z)−c)/τt)∈RK,(3)\mathbf{p}_s=\operatorname{softmax}(h_s(\mathbf{z})/\tau_s)\in\mathbb{R}^K,\qquad\mathbf{p}_t=\operatorname{softmax}((h_t(\mathbf{z})-\mathbf{c})/\tau_t)\in\mathbb{R}^K, \tag*{(3)}

where cc is an EMA of the teacher outputs ht(z)h_t(z) with momentum 0.9, kept separately for [CLS] and patch tokens. Centring by cc prevents one dimension from dominating, and a teacher temperature τt<τs=0.1\tau_t < \tau_s = 0.1 prevents collapse to the uniform distribution. Sample-level and patch-level losses are

LCLS=−12[ptCLS(u′)⊤log⁡psCLS(u)+ptCLS(u)⊤log⁡psCLS(u′)]∈[0,∞),(4)\mathcal{L}_{\mathrm{CLS}}=-\frac{1}{2}\left[\mathbf{p}_t^{\mathrm{CLS}}(u')^\top\log\mathbf{p}_s^{\mathrm{CLS}}(u)+\mathbf{p}_t^{\mathrm{CLS}}(u)^\top\log\mathbf{p}_s^{\mathrm{CLS}}(u')\right]\in[0,\infty), \tag*{(4)}
Lpatch=−12∑w∈{u,u′}1∣Vw∣∑i∈Vwpt,ipatch(w)⊤log⁡ps,ipatch(w)∈[0,∞),(5)\mathcal{L}_{\mathrm{patch}}=-\frac{1}{2}\sum_{w\in\{u,u'\}}\frac{1}{\lvert\mathcal{V}_w\rvert}\sum_{i\in\mathcal{V}_w}\mathbf{p}_{t,i}^{\mathrm{patch}}(w)^\top\log\mathbf{p}_{s,i}^{\mathrm{patch}}(w)\in[0,\infty), \tag*{(5)}

where Vw\mathcal{V}_w indexes the visible patches of view ww, and pt(w)\mathbf{p}_t(w) is computed from the less masked teacher view of ww. The LCLS\mathcal{L}_{\mathrm{CLS}} term pulls the sample-level representations of different views together. The Lpatch\mathcal{L}_{\mathrm{patch}} term instead pulls each visible patch token of the student towards the teacher token at the same position, which the teacher computes from the same or more context. This gives dense supervision in feature space that masked data modelling in input space cannot provide.

KoLeo Regularisation. Self-distillation reduces intra-sample redundancies. On its own, it does not prevent inter-sample redundancies, since the student can still map different inputs to a narrow cone in feature space. We therefore add a KoLeo regulariser [28], which spreads features uniformly over the unit hypersphere. It builds on the Kozachenko–Leonenko estimator of differential entropy. Given ℓ2\ell_2-normalised student [CLS] tokens z1,…,zng∈Rdz_1,\ldots,z_{n_g}\in\mathbb{R}^d of a group of ng=32n_g=32 masked samples,

LKoLeo=−1ng∑i=1nglog⁡(ri)∈[−log⁡2,∞),ri=min⁡j∈{1,…,ng}∖{i}∥zi−zj∥2,(6)\mathcal{L}_{\mathrm{KoLeo}}=-\frac{1}{n_g}\sum_{i=1}^{n_g}\log(r_i)\in[-\log2,\infty),\qquad r_i=\min_{j\in\{1,\ldots,n_g\}\setminus\{i\}}\lVert z_i-z_j\rVert_2, \tag*{(6)}

rewards a larger distance rir_i from each sample ii to its nearest neighbour jj and thus directly penalises near-duplicate features. We average this loss over all groups of a batch.

Training Objective. The full pre-training objective of OTISv2 combines the input-space objectives of OTIS with the feature-space objectives introduced above,

L=LMSE+λNCCLNCC+λSD(LCLS+Lpatch)+λKoLeoLKoLeo.(7)\mathcal{L}=\mathcal{L}_{\mathrm{MSE}}+\lambda_{\mathrm{NCC}}\mathcal{L}_{\mathrm{NCC}}+\lambda_{\mathrm{SD}}\left(\mathcal{L}_{\mathrm{CLS}}+\mathcal{L}_{\mathrm{patch}}\right)+\lambda_{\mathrm{KoLeo}}\mathcal{L}_{\mathrm{KoLeo}}. \tag*{(7)}

We empirically set λNCC=0.1\lambda_{\mathrm{NCC}}=0.1, λSD=0.1\lambda_{\mathrm{SD}}=0.1, and λKoLeo=0.025\lambda_{\mathrm{KoLeo}}=0.025. All losses are averaged over the batch.

Data

We curate a diverse corpus of 756 k samples and 45B time points (App. § 8) covering 11 domains across healthcare (ECG, EEG), audio, engineering (electromechanics), natural sciences (temperature, weather), infrastructure (cloud, energy, traffic), and finance (banking, economics). This breadth exposes OTISv2 to heterogeneous variate correlations and temporal dynamics. Variate counts range from 1 to 19, and sequence lengths from 720 to 480 k time points. To align with the growing interest in deploying general-purpose encoders for healthcare applications [27, 40], our pre-training corpus includes the most widely used physiological time series. In particular, it contains 400 k 12-lead ECGs (24 B time points) and 3.4 k 10–20 system EEG recordings (3.5 B time points).

Experiments

We pre-train OTISv2 for 50 epochs on two NVIDIA A100 GPUs with AdamW [22], effective batch size 384, peak learning rate 1.2×10−31.2\times10^{-3} with a warmup-stable-decay schedule [16], weight decay 0.025, layer-wise learning rate decay 0.975, drop path 0.1 [17], and gradient clipping at 3. Compute costs and hyperparameters are detailed in Appendix § 9.

We evaluate OTISv2 along five axes. First, we compare its downstream performance on 143 datasets from the UEA [2] and UCR [5] archives (App. § 10) against nine general-purpose encoders (App. § 11, § 4.1). Second, we track the feature quality of OTIS and OTISv2 during matched pre-training to test whether feature-space supervision mitigates feature collapse (§ 4.2). Third, we vary feature quality under a fixed objective to test whether it determines downstream performance (§ 4.3). Fourth, we conduct a cumulative ablation from OTIS to OTISv2 to attribute the changes in feature quality and downstream performance to the architecture, the training stabilisers, and the objective (§ 4.4). Fifth, we scale the encoder from 7.1 M to 116 M parameters under data constraints to test whether mitigating feature collapse eventually unlocks positive scaling effects (§ 4.5).

Metrics. We pair downstream performance with two feature quality measures (Appendix § 12). Patch-token cosine similarity (or token similarity for short, ↓) is the average pairwise cosine similarity between the ℓ2\ell_2-normalised output patch tokens of a single time series, quantifying token redundancy within a sample. RankMe [9] (↑) measures the effective rank of the mean-pooled patch tokens, reflecting the dimensionality spanned across samples. Bounded by the encoder width dd, RankMe is compared only across encoders of the same width. Both are measured on frozen checkpoints, except in § 4.2 and § 4.5, which record them during training and compare them only within the same figure (Appendix § 12). Downstream performance is measured via balanced accuracy (↑) under three evaluation protocols: 5-shot prototypical classification, kk-NN classification, and linear probing. Avg is the balanced accuracy averaged over all datasets and evaluation protocols, such that each dataset carries equal weight.

Downstream Performance of Frozen General-Purpose Encoders

Setup. We evaluate how well time series encoders generalise to unseen datasets without fine-tuning. Table 1 compares OTISv2-7.1M with nine public encoders of the UniTS, MOMENT, and OTIS families. As these encoders were trained under different conditions, including training data, token budgets, and compute, this comparison serves solely as a benchmark. Complementarily, Sections 4.2 to 4.5 test how our proposed architectural changes and objectives affect pre-training under matched conditions.

Results. OTISv2-7.1M matches or surpasses general-purpose encoders up to 43×43\times larger. With 70.12%, it yields the best average performance. Only OTIS-128M comes within 0.2 pp (69.97%) with 18× more parameters and 1.9× more training tokens. The largest encoder, MOMENT-308M, stays 1.83 pp below despite 43× more parameters and 1.2× more compute. Over OTIS-7.1M of the same size, OTISv2 gains 2.47 pp on average and 0.6 to 4.5 pp on every archive and protocol, with 3.8× fewer training tokens. OTIS gains about as much only by growing from 7.1M to 128M parameters (2.32 pp), so at a fixed size the recipe of OTISv2 is worth an 18-fold increase in parameters.

OTISv2 has the highest accuracy under all three protocols on UEA and two of the three on UCR, and its lead is largest under the 5-shot evaluation protocol (1.30 pp on UEA and 0.40 pp on UCR over the next-best encoder). Its frozen features thus separate classes from few labels, on multi-variate and uni-variate data alike. Only on the UCR linear probe do five encoders score 0.1 to 1.0 pp higher. These are exactly the five wider encoders (d=384d = 384 to 1,024, against 192), so here the trained linear head appears to profit from encoder width.

The margin of OTISv2 over MOMENT and UniTS grows with the number of variates VV. On all eight datasets with V≥8V \ge8, OTISv2 outperforms MOMENT-308M and UniTS-3.5M and matches OTIS-128M, scoring higher on four. Every encoder scores lower on these datasets than on uni-variate ones, but the OTIS family loses the least accuracy: OTIS-128M 4.2 pp, OTISv2 5.1 pp, UniTS-3.5M 9.9 pp, and MOMENT-308M 12.7 pp. Four of the eight datasets have 24 to 144 variates, more than the 19 seen in pre-training. Against MOMENT-308M, the margin also grows with high-frequency content (Spearman ρ=0.28\rho=0.28), whereas neither dataset difficulty nor similarity to our corpus predicts it. The advantage of OTISv2 thus holds on data unlike its pre-training data, in both variate count and signal properties.

Under matched training, OTISv2 has lower token similarity and higher RankMe than OTIS (0.50 against 0.56 and 75 against 52, § 4.4). The released encoders do not show this order (0.60 against 0.55 and 62 against 68), as they also differ in training data and training length (App. § 11). Token similarity and RankMe thus compare recipes only between runs trained alike.

Feature Collapse With and Without Feature-Space Supervision

Setup. OTIS is supervised only in input space, whereas OTISv2 also supervises its feature space through self-distillation and KoLeo regularisation. If this supervision mitigates feature collapse, OTISv2 should have lower token similarity and higher RankMe than OTIS. To test this, we pre-train both encoders from scratch under matched conditions: the same data, token budget, width, depth, optimiser, and schedule. The training curves are depicted in Figure 2. The two encoders still differ in architecture and training stabilisers, which § 4.4 separates from the feature-space objectives.

Results. OTISv2 keeps lowering token similarity after OTIS stalls, and its downstream lead grows with training. In both encoders, token similarity drops within the first 200 M training tokens. OTIS then stays at 0.46, whereas OTISv2 declines further to 0.32. The RankMe of OTISv2 also rises faster and levels off higher. In OTIS, the collapse thus sets in early, and further training does not undo it. The UEA 5-shot accuracy of the two encoders is at parity after 32 M tokens, and OTISv2 leads by 5 to 7 pp after 1.60 B tokens. More training thus widens the lead of OTISv2 instead of closing it.

The difference is not limited to the output tokens. After training, OTISv2 has lower token similarity and higher RankMe at every encoder block, and its accuracy lead builds up over the first four blocks (App. § 13). Neither encoder develops the high-norm artefact tokens of large vision transformers [4] (App. § 14). Single ECG samples of the released encoders illustrate the difference: in OTISv2, different patches attend to distinct sets of patches, whereas in OTIS their attention maps look alike (App. § 15).

Linking Feature Quality to Downstream Performance

Setup. Whether mitigating feature collapse causes the downstream lead of OTISv2 is still open, as the objective and the degree of collapse change together. We therefore keep the objective fixed and vary the degree of collapse by changing only the optimisation. If downstream performance then follows, there is a causal link between collapse and performance. We pre-train OTISv2-7.1M at the ten combinations of effective batch size BeffB_{\mathrm{eff}} and peak learning rate η\eta in Table 2, keeping the objective, architecture, data, training token budget D=221D=221 M (as in § 4.5), and compute fixed.

BeffB_{\mathrm{eff}} / η\etaSim (↓)RankMe (↑)5-shot (↑)BeffB_{\mathrm{eff}} / η\etaSim (↓)RankMe (↑)5-shot (↑)
384 / 3×10−43 \times10^{-4}0.52851.252.18384 / 1.2×10−31.2 \times10^{-3}0.51962.153.22
192 / 3×10−43 \times10^{-4}0.48758.853.2696 / 3×10−43 \times10^{-4}0.48162.654.29
384 / 2.4×10−32.4 \times10^{-3}0.55659.352.99192 / 1.2×10−31.2 \times10^{-3}0.51662.953.69
384 / 6×10−46 \times10^{-4}0.50559.853.00192 / 6×10−46 \times10^{-4}0.48964.354.27
96 / 1.2×10−31.2 \times10^{-3}0.53859.854.0996 / 6×10−46 \times10^{-4}0.49464.855.01

Table 2. Feature quality and downstream performance under a fixed objective. OTISv2-7.1M pre-trained at ten combinations of effective batch size BeffB_{\mathrm{eff}} and peak learning rate η\eta, ordered by RankMe. RankMe, patch-token cosine similarity (Sim), and UEA 5-shot accuracy (5-shot) are means over three seeds. RankMe correlates with 5-shot (r=+0.83r=+0.83), but Sim does not (r=−0.48r=-0.48).

Results. Downstream accuracy follows the effective rank even under a fixed objective. Across the ten combinations, RankMe spans 51 to 65, and UEA 5-shot accuracy follows it with a Pearson correlation of r=+0.83r=+0.83. When the 5-shot protocol uses the mean-pooled patch tokens instead of the [CLS] token, the correlation is r=+0.78r=+0.78, so the link does not depend on which embedding is classified. As the objective is fixed, the loss of rank itself limits downstream performance.

Only RankMe tracks accuracy on both data archives. Token similarity spans only 0.48 to 0.56 and does not track UEA 5-shot accuracy (r=−0.48r=-0.48). On the uni-variate UCR archive, both measures track 5-shot accuracy (r=+0.66r=+0.66 with RankMe and r=−0.85r=-0.85 with token similarity), although accuracy varies by only 1.8 pp across the combinations. The run with the highest RankMe, Beff=96B_{\mathrm{eff}}=96 at η=6×10−4\eta=6\times10^{-4}, also has the highest UEA 5-shot accuracy (55.01%). It takes the most optimisation steps at a moderate learning rate, whereas at every batch size, raising the learning rate beyond the best value lowers both RankMe and accuracy. At Beff=384B_{\mathrm{eff}}=384, at which the scaling study of § 4.5 holds fixed, the peak learning rate of the released OTISv2, 1.2×10−31.2\times10^{-3}, is the best setting. As RankMe needs no labels, it can thus guide the choice of optimisation settings before any downstream evaluation.

Cumulative Ablation of Architecture, Objective, and Stabilisation

Setup. OTISv2 changes the architecture, objective, and training stabilisers of OTIS. To isolate the contribution of each component, we start from the OTIS recipe and add one component per row of Table 3, eventually yielding OTISv2. The first three rows make the architecture independent of domain labels, sequence length, and variate count: shared registers in place of domain-specific variate embeddings, and axial RoPE in place of absolute position embeddings. The next four rows add training stabilisers: zero-initialised residual projections, SwiGLU MLPs, drop path, and layer-wise learning rate decay. The last three rows add self-distillation, gradient clipping, and KoLeo regularisation. If supervising the feature space mitigates feature collapse, only these last three rows should lower token similarity below that of OTIS.

Table 3: Cumulative ablation from OTIS to OTISv2. One component added per row, starting from the OTIS pre-training recipe towards the OTISv2 recipe. All rows use C=6ND=6.82×1016C=6ND=6.82\times10^{16} FLOPs at N=7.1N=7.1 M parameters, D=1.60D=1.60 B training tokens, and DC=32D_C=32 M unique tokens. Mean scores over three pre-training seeds are reported. Arrows give the change from the previous row.

EncoderPatch-token cosine sim (↓\downarrow)RankMe (↑\uparrow)UEA 5-shot (↑\uparrow)UEA k-NN (↑\uparrow)UEA linear (↑\uparrow)UCR 5-shot (↑\uparrow)UCR k-NN (↑\uparrow)UCR linear (↑\uparrow)Avg weighted (↑\uparrow)
OTIS-7.1M0.565252.4856.5664.3858.6261.7574.1163.69
+domain token0.59 ↑0.03\uparrow0.0351 ↓1\downarrow151.08 ↓1.40\downarrow1.4056.78 ↑0.22\uparrow0.2264.31 ↓0.07\downarrow0.0757.01 ↓1.61\downarrow1.6160.53 ↓1.22\downarrow1.2272.56 ↓1.55\downarrow1.5562.40 ↓1.29\downarrow1.29
+register tokens0.59 ↓0.00\downarrow0.0050 ↓1\downarrow152.78 ↑1.70\uparrow1.7057.60 ↑0.82\uparrow0.8265.51 ↑1.20\uparrow1.2058.37 ↑1.36\uparrow1.3662.04 ↑1.51\uparrow1.5173.58 ↑1.02\uparrow1.0263.69 ↑1.29\uparrow1.29
+Axial RoPE0.66 ↑0.07\uparrow0.0756 ↑6\uparrow653.12 ↑0.34\uparrow0.3457.97 ↑0.37\uparrow0.3765.98 ↑0.47\uparrow0.4758.75 ↑0.38\uparrow0.3862.26 ↑0.22\uparrow0.2274.83 ↑1.25\uparrow1.2564.27 ↑0.58\uparrow0.58
+zero-init residuals0.64 ↓0.02\downarrow0.0254 ↓2\downarrow253.44 ↑0.32\uparrow0.3257.58 ↓0.39\downarrow0.3966.10 ↑0.12\uparrow0.1256.47 ↓2.28\downarrow2.2858.77 ↓3.49\downarrow3.4973.49 ↓1.34\downarrow1.3462.29 ↓1.98\downarrow1.98
+SwiGLU0.62 ↓0.02\downarrow0.0257 ↑3\uparrow350.41 ↓3.03\downarrow3.0355.72 ↓1.86\downarrow1.8664.65 ↓1.45\downarrow1.4556.50 ↑0.03\uparrow0.0360.60 ↑1.83\uparrow1.8373.28 ↓0.21\downarrow0.2162.41 ↑0.12\uparrow0.12
+drop path0.64 ↑0.02\uparrow0.0258 ↑1\uparrow152.26 ↑1.85\uparrow1.8557.20 ↑1.48\uparrow1.4865.58 ↑0.93\uparrow0.9357.99 ↑1.49\uparrow1.4960.62 ↑0.02\uparrow0.0273.75 ↑0.47\uparrow0.4763.19 ↑0.78\uparrow0.78
+layer-wise LR decay0.62 ↓0.02\downarrow0.0260 ↑2\uparrow252.46 ↑0.20\uparrow0.2056.64 ↓0.56\downarrow0.5665.35 ↓0.23\downarrow0.2357.54 ↓0.45\downarrow0.4561.10 ↑0.48\uparrow0.4873.66 ↓0.09\downarrow0.0963.14 ↓0.05\downarrow0.05
+self-distillation0.53 ↓0.09\downarrow0.0968 ↑8\uparrow855.52 ↑3.06\uparrow3.0659.79 ↑3.15\uparrow3.1567.39 ↑2.04\uparrow2.0459.61 ↑2.07\uparrow2.0763.91 ↑2.81\uparrow2.8175.71 ↑2.05\uparrow2.0565.52 ↑2.38\uparrow2.38
+gradient clipping0.55 ↑0.02\uparrow0.0267 ↓1\downarrow155.46 ↓0.06\downarrow0.0659.55 ↓0.24\downarrow0.2466.93 ↓0.46\downarrow0.4659.72 ↑0.11\uparrow0.1163.96 ↑0.05\uparrow0.0575.98 ↑0.27\uparrow0.2765.60 ↑0.08\uparrow0.08
+KoLeo = OTISv2-7.1M0.50 ↓0.05\downarrow0.0575 ↑8\uparrow857.52 ↑2.06\uparrow2.0663.10 ↑3.55\uparrow3.5567.44 ↑0.51\uparrow0.5161.32 ↑1.60\uparrow1.6068.23 ↑4.27\uparrow4.2776.15 ↑0.17\uparrow0.1767.62 ↑2.02\uparrow2.02

Table 3.

Results. The feature-space objectives supply the entire gain from OTIS to OTISv2. Together with gradient clipping, they lower token similarity from 0.62 to 0.50, raise RankMe from 60 to 75, and lift the average accuracy over all datasets and protocols by 4.48 pp, more than the total gain of 3.93 pp. The other seven rows together lose 0.55 pp: the architectural rows add 0.58 pp, and the training stabilisers cost 1.13 pp. Self-distillation lowers token similarity by 0.09, raises RankMe by 8, and lifts UEA 5-shot and k-NN accuracy by 3.06 and 3.15 pp. KoLeo lowers token similarity by another 0.05, raises RankMe by another 8, and lifts both accuracies by 2.06 and 3.55 pp. Although KoLeo acts only on the [CLS] token, it lowers patch-token cosine similarity, as [CLS] and patch tokens share the encoder weights. Gradient clipping leaves the average accuracy unchanged (+0.08 pp). As only the last three rows take token similarity below the 0.56 of OTIS, the lower token similarity of OTISv2 in § 4.2 and its downstream gain both stem from supervising the feature space.

The architectural rows make the encoder independent of domain labels, sequence length, and variate count without lowering accuracy. Shared variate embeddings with a domain token lower the average accuracy by 1.29 pp, and registers in place of this token recover the loss (+1.29 pp) at unchanged token similarity. With registers, the encoder thus applies to any domain and still reaches the same average accuracy as OTIS (63.69 %). Axial RoPE raises token similarity more than any other row (+0.07), but, among the architectural rows, it alone raises RankMe (+6) and yields the largest gain on the UCR linear probe (+1.25 pp). A rise in token similarity thus need not mean a loss of rank.

Scaling Encoder Size Under Data Constraints

Setup. Larger OTIS encoders improve little over smaller ones (Fig. 1). If feature collapse causes this plateau, mitigating it should let larger encoders convert their parameters into downstream performance. As time series corpora are small, Figure 3 compares both encoders across sizes under data constraints. We train both encoders at four sizes NN (App. I) on the same training token budget D=221D = 221 M, so that compute is fixed within each size. At each size, we vary the unique data budget DCD_C in eight steps from 221 M to 7 M tokens, i.e. from 1 to 32 epochs of repetition. As DD is the same for every size, tokens per parameter fall from 31 at 7.1 M to 1.9 at 116 M. Both encoders use the same settings at every size: effective batch size 384, peak learning rate 1.2×10−31.2 \times10^{-3}, and a warmup-stable-decay schedule. We approximate compute as C=6NDC = 6ND [18, 24], which understates the cost of OTISv2 by 1.05 to 2.1×\times over our size range (App. B). Each run uses a single seed. From four runs at 7.1 M

Compute (C=6NDC = 6ND)Parameters (NN)Training tokens (DD)Data budget (DCD_C)
9.4×10159.4 \times10^{15}7.1 M221 M{221,111,74,55,44,32,16,7}\{221, 111, 74, 55, 44, 32, 16, 7\} M
2.0×10162.0 \times10^{16}14.8 M221 M{221,111,74,55,44,32,16,7}\{221, 111, 74, 55, 44, 32, 16, 7\} M
5.3×10165.3 \times10^{16}40 M221 M{221,111,74,55,44,32,16,7}\{221, 111, 74, 55, 44, 32, 16, 7\} M
1.5×10171.5 \times10^{17}116 M221 M{221,111,74,55,44,32,16,7}\{221, 111, 74, 55, 44, 32, 16, 7\} M

Table 3 (PDF p. 9).

repeated with three seeds, the UEA 5-shot difference between two runs varies by about 0.6 pp across seeds, and we read only effects beyond this seed band.

Results. OTISv2 gains with size where OTIS declines. At all 32 combinations of size and data budget, OTISv2 has lower token similarity, higher RankMe, and higher UEA 5-shot accuracy than OTIS, with a margin of 3.6 to 10.3 pp. Averaged over the eight data budgets, the UEA 5-shot accuracy of OTISv2 rises at every step in size, from 52.3% at 7.1 M to 54.3% at 116 M, whereas that of OTIS falls at every step, from 47.9% to 45.5%. The gap between them thus doubles, from 4.4 to 8.8 pp.

OTIS collapses as it grows: its token similarity rises from 0.72 to 0.95, and its RankMe, normalised by the encoder width dd, falls from 0.35 to 0.05. OTISv2 loses some rank (0.55 to 0.38), but its similarity stays between 0.39 and 0.49. Both encoders see nearly the same tokens per parameter at every size, so undertraining alone does not explain the collapse of OTIS. As the loss of rank limits downstream performance (§ 4.3), this collapse is the likely cause of the decline of OTIS with size.

Most of the accuracy gain of OTISv2 comes with the first doubling of parameters. From 7.1 M to 14.8 M, OTISv2 improves at all eight budgets beyond the seed band (+0.8 to +2.5 pp), whereas OTIS improves at only two. Beyond 14.8 M, the gains of OTISv2 are small and change sign between data budgets, but both larger encoders stay above the 7.1 M encoder at all eight budgets. At 40 M, OTIS instead scores below its own 14.8 M encoder at seven of the eight budgets, and at 116 M below its own 7.1 M encoder at all eight. The released OTIS encoders gain 3.2 pp in UEA 5-shot accuracy from 7.1 M to 128 M parameters (Table 1), but with 91× to 185× our 221 M training tokens. Larger OTISv2 encoders thus pay off on far fewer tokens than larger OTIS encoders. As the gains beyond 14.8 M stay small at this token budget, the plateau sets in at a larger size rather than disappearing.

EncoderEncoder width ddCompute C=6NDC = 6ND [FLOPs]Tokens/param TPP =D/N= D/NPatch-token cosine sim (↓)RankMe (↑)UEA 5-shot (↑)UEA kk-NN (↑)UEA linear (↑)UCR 5-shot (↑)UCR kk-NN (↑)UCR linear (↑)Avg weighted (↑)
UniTS-0.4M324.3×1016†4.3 \times10^{16\dagger}45,000†45{,}000^\dagger0.231638.4846.0250.9249.7157.3661.2954.35
UniTS-1.1M§646.0×1016†6.0 \times10^{16\dagger}8,200†8{,}200^\dagger0.093246.4652.7856.4553.9762.9366.0359.52
UniTS-3.5M1283.8×1017†3.8 \times10^{17\dagger}5,200†5{,}200^\dagger0.274644.8251.7758.1752.9261.4368.1059.33
MOMENT-19M5123.5×1016†3.5 \times10^{16\dagger}16†16^\dagger0.842548.1354.7763.6060.8769.4177.4167.02
MOMENT-85M7681.6×1017†1.6 \times10^{17\dagger}3.6†3.6^\dagger0.834949.4656.4363.6362.2870.2477.7567.91
MOMENT-308M1,0245.7×1017†5.7 \times10^{17\dagger}1.0†1.0^\dagger0.5618152.8458.3764.3862.8070.3977.2868.29
OTIS-7.1M1921.8×10181.8 \times10^{18}5,7000.556855.5364.4267.7359.3669.8876.6367.65
OTIS-43M3845.1×10185.1 \times10^{18}4700.679957.5364.8069.2861.4571.5277.6669.19
OTIS-128M5761.6×10191.6 \times10^{19}1600.5116258.7065.7869.1962.3772.4678.1869.97
OTISv2-7.1M1924.6×10174.6 \times10^{17}1,5000.606260.0066.2269.8463.2072.7077.1870.12

Table 1. Feature quality and downstream performance of frozen general-purpose time series encoders. Balanced accuracy on 23 UEA and 120 UCR datasets, with best and second-best scores highlighted. DD counts patch tokens seen during pre-training, including repetition. OTISv2-7.1M matches or surpasses encoders up to 43×43\times larger on average.

†Not reported by the original works; reconstructed from their released pre-training recipes and configurations (Appendix § 11).

‡C=6NDC = 6ND [18] is reported for cross-family comparability only (Appendix § 9).

§Fully supervised, not pre-trained by masked data modelling (Appendix § 11).

Repeating data up to 32 times does not lower the UEA 5-shot accuracy of OTISv2. At each size, the mean accuracy of the OTISv2 runs with 14 and 32 epochs differs from that of runs with 1 and 2 epochs by only −0.5 to +0.1 pp, inside the seed band, against −1.4 to +1.0 pp for OTIS. As DD and the number of optimisation steps are the same at each size, the comparison isolates repetition itself. For language models, in contrast, the value of repeated tokens decays beyond about four epochs [24]. At this scale, 7 M unique tokens repeated 32 times thus serve about as well as 221 M unique tokens.

Conclusion

In this work, we identify feature collapse as a proximate cause of the scaling plateau in general-purpose time series encoders. We complement the input-space objectives of masked data modelling with two feature-space objectives: a self-distillation loss and a KoLeo regulariser. This mitigates the collapse, and a single frozen OTISv2–7.1 M encoder surpasses general-purpose encoders by 43× larger in average accuracy over 143 UEA and UCR datasets. Our analysis yields three insights. First, the loss of effective rank limits downstream performance, as accuracy follows the effective rank even under a fixed objective. Second, under matched training, the feature-space objectives supply the entire accuracy gain over OTIS, whereas the architectural changes mainly make the encoder independent of domain labels, sequence length, and variate count. Third, once the collapse is mitigated, larger encoders gain accuracy, whereas larger OTIS encoders lose it. At our token budget, however, the plateau sets in at a larger size rather than disappearing.

Limitations. Our evaluation focuses on discriminative tasks, and beyond zero-shot forecasting (App. § 18), the effect of OTISv2 on generative tasks remains to be studied. Furthermore, the pre-training corpus, although diverse, over-represents physiological signals, so generalisation to unseen time series domains needs further evaluation. The absolute scale we explore, at most 116 M parameters and 221 M training tokens in the scaling study, is small compared to computer vision and natural language processing. Whether the recipe extends to billion-parameter time series encoders, and where the plateau sets in at larger token budgets, remain open. The patch width p=24p=24 limits performance on very short uni-variate series of 96 time points or fewer, where the encoder sees at most four tokens. OTISv2 is also more sensitive than OTIS to additive noise and, less so, to spikes, missing segments, and variate dropout, in part because its [CLS] embedding, which carries most of the downstream advantage, is more sensitive than its mean-pooled patch tokens (App. § 19). Finally, self-distillation with the exponential-moving-average teacher raises the memory needed for training by about 2.4×, which limits batch and encoder size on a given GPU, but it does not affect inference cost.

Acknowledgments and Disclosure of Funding

We thank Laura Kraus for encouraging the submission of this work and for believing in its potential.

References

Pre-Training Corpus Details

Table 4 summarises the pre-training corpus of OTISv2. The corpus holds 756,267 samples with 45.2 billion time points from 11 domains. More than half of the samples (53 %) are physiological signals, namely 400 k 12-lead ECGs and 3.4 k EEGs in the 10–20 system, to support general-purpose encoders for healthcare applications [27, 40]. In terms of time points, ECG (24.0 B) and audio (15.6 B) account for 88 % of the corpus. Samples have 1 to 19 variates and 720 to 480 k time points, covering a wide range of correlations between variates and of temporal dynamics. The nine sources other than GIFT-Eval [1] are shared with OTIS and pre-processed in the same way [34]. The four GIFT-Eval domains are new to OTISv2 and are drawn from the GIFT-Eval pre-training collection. For these domains, we discard series shorter than 1,024 time points, split longer ones into chunks of at most 100 k time points (50 k for energy and weather), set missing values to zero, normalise each variate to zero mean and unit variance, and clip the result to [−10,10][-10, 10].

DomainSourceSamplesVariates VVTime points TTV⋅TV \cdot TFrequency
ECGMIMIC-IV-ECG [12]400,000125,00024.0 B500 Hz
EEGTDBRAIN [35]2,6921960,0003.07 B500 Hz
SEED [42]6751937,000475 M200 Hz
AudioAudioSet-20K [10]19,6141–224,576–480,00015.6 B44.1 kHz
ElectromechanicsFD-A [21]8,18415,12041.9 M64 kHz
TemperatureDWD [6]202,7121720146 M278 µHz
CloudGIFT-Eval [1]29,2731–21,025–8,064167 M0.28–3.33 mHz
EnergyGIFT-Eval [1]21,8541–31,036–50,0001.24 B0.28–250 mHz
TrafficGIFT-Eval [1]67,9841–31,037–96,400378 M0.28–3.33 mHz
WeatherGIFT-Eval [1]3,0531–111,302–50,00087.8 M12–278 µHz
BankingNN5 [32]111179187.8 k12 µHz
EconomicsFRED-MD [23]107172877.9 k386 nHz
Exchange [20]817,58860.7 k12 µHz
Total756,26745.2 B

Table 4. Pre-training corpus. Eleven domains from healthcare, audio, engineering, natural sciences, infrastructure, and finance, with 756,267 samples and 45.2 B time points in total. V⋅TV \cdot T counts the time points of all variates, summed over samples, and Frequency denotes the sampling rate.

Computational Details

Table 5 reports the compute used for pre-training, and Table 6 lists the pre-training hyperparameters of OTISv2-7.1M. OTISv2-7.1M is pre-trained on two NVIDIA A100-80GB GPUs. The 14.8 M, 40 M, and 116 M variants are trained only in the scaling study of § 4.5, each on a single GPU. Their entries in Table 5 thus report the total cost of the eight runs per size, one per data budget. Throughout the paper, we approximate compute as C=6NDC = 6ND. Counting the matrix multiplications and convolutions of a full training step with the FLOP counter of PyTorch, for a sample of average size (288 patch tokens) and at the mean masking ratios of the released recipe, the actual cost of OTISv2 per training token is 2.09, 1.52, 1.19, and 1.05 times 6N6N for the 7.1 M, 14.8 M, 40 M, and 116 M encoders. The decoder, the projection heads, and the teacher add compute that 6ND6ND ignores, whereas the encoder processes only the visible patches, so the gap shrinks as the encoder grows. Each downstream evaluation runs on a single GPU with at least 46 GB of memory.

EncoderParametersGPU CountGPU Hours†GPU Type
OTISv2-7.1M7.1 M2∼61NVIDIA A100-80GB
OTISv2-14.8M‡14.8 M1∼12NVIDIA A100-80GB
OTISv2-40M‡40 M1∼18NVIDIA A100-80GB
OTISv2-116M‡116 M1∼27NVIDIA A100-80GB

Table 5. Computational resources used for pre-training OTISv2.

HyperparameterValue
OptimiserAdamW [22], β=(0.9,0.95)\beta=(0.9,0.95)
Base learning rate (peak = base ×Beff/32\times B_{\mathrm{eff}}/32)1×10−41 \times10^{-4}
Peak / minimum learning rate1.2×10−3/1.2×10−41.2 \times10^{-3}/1.2 \times10^{-4}
Learning rate schedulewarmup-stable-decay, 10 % warm-up, 10 % cosine decay
Layer-wise learning rate decay0.975
Epochs50
Batch size (per GPU / effective)192 / 384
Weight decay0.025
Drop path0.1
Gradient clipping3
Patch width pp / context T‾\overline{T}24 / 1,008
Register tokens RR4
Post-fix masking probability π\pi0.33
Masking ratio of student ρs\rho_s (random / post-fix)0.7 / 0.5, each +U(0,0.15)+\mathcal{U}(0,0.15) per batch
Masking ratio of teacher ρt\rho_t0.5
NCC weight λNCC\lambda_{\mathrm{NCC}}0.1
Self-distillation weight λSD\lambda_{\mathrm{SD}} (on LCLS+Lpatch\mathcal{L}_{\mathrm{CLS}}+\mathcal{L}_{\mathrm{patch}})0.1
Projection heads ([CLS] and patch)3-layer MLP, hidden size 2,048
Bottleneck size / head outputs KK256 / 8,192
Student temperature τs\tau_s0.1
Teacher temperature τt\tau_t0.04→0.0550.04 \to0.055 over the first 4 % of training
Centring momentum0.9
EMA teacher momentum mm0.996→10.996 \to1, cosine schedule
Last head layer frozenfirst 2 % of training
KoLeo weight λKoLeo\lambda_{\mathrm{KoLeo}} (on [CLS])0.025
KoLeo group size32

Table 6. Pre-training hyperparameters of OTISv2-7.1M.

Benchmark Details

We evaluate on 143 classification datasets from two public archives and use their official splits. The UEA archive [2] contributes 23 datasets: 22 multi-variate datasets with 2 to 144 variates and 26 to 1,751 time points, and Phoneme, which we use in its uni-variate UCR version (1,024 time points). Phoneme therefore appears in both archives, so the 143 datasets comprise 142 distinct ones. The UEA datasets cover speech and audio (JapaneseVowels, SpokenArabicDigits, Phoneme, Heartbeat), motion and gesture (NATOPS, RacketSports, BasicMotions, Cricket, Libras, UWaveGestureLibrary, ArticularyWordRecognition, Epilepsy), EEG and MEG (FaceDetection, FingerMovements,

HandMovementDirection, SelfRegulationSCP1, SelfRegulationSCP2), ECG (AtrialFibrillation), handwriting (Handwriting, CharacterTrajectories), and other sensors (ERing, EthanolConcentration, LSST). The UCR archive [5] contributes 120 uni-variate datasets with 15 to 2,844 time points and 2 to 60 classes, covering image outlines, sensors, ECG, motion, simulations, traffic, and spectroscopy. Of the eleven variable-length UCR datasets, we use the equal-length versions of the three GestureMidAir datasets and exclude the other eight, whose equal-length versions pad the training and test splits to different lengths. We linearly interpolate the missing values of four datasets in time. Time series shorter than the patch width p=24p=24 are right-padded with zeros. As axial RoPE encodes only relative positions, longer time series are embedded at native length, up to their last full patch.

Baseline Details

Table 7 summarises the baselines. All baselines are loaded from their public checkpoints and evaluated with the frozen-encoder protocols of Appendix E, without fine-tuning.

EncoderStrategyEncoder width ddParametersPre-training time points
UniTS-0.4M [8]MDM + task prompts320.4 M0.13 B0.13\,\mathrm{B}2
UniTS-1.1M [8]Supervised multi-task3641.1 M0.13 B0.13\,\mathrm{B}2
UniTS-3.5M [8]MDM + task prompts1283.5 M0.13 B0.13\,\mathrm{B}2
MOMENT-19M [11]MDM51219 M1.2 B
MOMENT-85M [11]MDM76885 M1.2 B
MOMENT-308M [11]MDM1,024308 M1.2 B
OTIS-7.1M [34]MDM1927.1 M42 B
OTIS-43M [34]MDM38443 M42 B
OTIS-128M [34]MDM576128 M42 B
OTISv2-7.1M (ours)MDM + SD + KoLeo1927.1 M45 B

Table 7. General-purpose time series encoders. Pre-training strategy (MDM, masked data modelling; SD, self-distillation), encoder width, parameter count, and pre-training corpus size.

Corpora and training tokens. OTIS and OTISv2 are pre-trained on different corpora that share nine sources from seven domains (App. § 8). OTIS uses 612,394 samples (41.7 B time points). OTISv2 adds the four GIFT-Eval [1] domains and the samples that OTIS held out for validation, for a total of 756,267 samples (45.2 B time points). With a context of T=1,008T=1{,}008 time points and patch width p=24p=24, one epoch on the two corpora yields 206.6 M and 219.0 M patch tokens, respectively. The gap in training tokens DD in Table 1 thus comes almost entirely from repetition. The released OTIS-7.1M weights correspond to epoch 198 (D=40.9D=40.9 B), and those of OTIS-43M and OTIS-128M to epochs 97 and 99 (D=20.0D=20.0 and 20.5 B), compared with 50 epochs for OTISv2 (D=10.9D=10.9 B). The scaling study of § 4.5 derives its data budgets from 221 M tokens per epoch, the count obtained if every variate filled the full context, whereas shorter samples yield fewer patches. MOMENT is pre-trained for two epochs on its 1.2 B time points at patch width p=8p=8, which gives D=308D=308 M. UniTS reports no corpus size, so we reconstruct it from its released pre-training configuration. Its 38 task configurations draw on 25 sources with 0.13 B time points in total. Each forecasting source is sampled with sliding windows that advance by one time step, at four context lengths and for ten epochs, so each of its time points is revisited about 10310^3 times. This gives D=18.1D=18.1 B patch tokens at patch width p=16p=16, so UniTS repeats its small corpus far more often than any other encoder. The supervised UniTS-1.1M trains on the same 38 tasks for five epochs instead of ten, which gives D=9.1D=9.1 B.

Evaluation Protocols

We follow the frozen-encoder protocols of OTIS [34] with three changes: k=5k=5 instead of 20 neighbours, three instead of five 5-shot episodes, and standardised instead of ℓ2\ell_2-normalised features for the linear probe, and apply them alike to every encoder. All metrics in Table 1 and 3, and the similarity and RankMe of Table 2 and Figure 4, are computed on frozen checkpoints, without masking or augmentation.

Figure 4. The advantage of OTISv2 builds up over depth. Token similarity, RankMe, and 5-shot accuracy at every block of the matched runs of OTIS and OTISv2 (N=7.1 MN = 7.1\,\mathrm{M}, D=1.60 BD = 1.60\,\mathrm{B}, DC=32 MD_C = 32\,\mathrm{M}, 12 blocks, d=192d = 192). Lines show the mean over three pre-training seeds, and shading one standard deviation. Similarity and RankMe follow the protocol of Table 3 (App. § 12), and at block 12 they equal its first and last rows. (Left) Patch-token cosine similarity. (Centre) RankMe normalised by dd. (Right) UEA 5-shot balanced accuracy on the [CLS] token of each block.

Feature quality. We draw a fixed set of 25,600 samples, and every encoder within a table uses the same samples. Table 1 and Table 2 draw from the pre-training corpus (App. § 8), and Table 3 and Figure 4 from the 101,858-sample subset on which their runs were pre-trained. Each domain contributes an equal share of these samples. Each sample keeps its native variate count and at most its first 1,008 time points, so no input is padded. OTIS receives freshly initialised variate embeddings for the four domains on which it was not pre-trained. MOMENT processes each sample in consecutive windows of 512 time points. RankMe [9] is computed on the 25,600×d25{,}600 \times d matrix of sample embeddings, each the mean of its output patch tokens. Patch-token cosine similarity is the mean off-diagonal cosine similarity between the ℓ2\ell_2-normalised output patch tokens of one sample, averaged over samples with equal weight. Both exclude special tokens such as [CLS], [reg], and prompt tokens. Over three independent draws, similarity varies by at most 0.001 and RankMe by at most 0.9, so larger differences between encoders are not caused by the choice of samples. Token similarity and RankMe recorded during training (Figure 2 and Figure 3) are computed differently. They use masked training batches, pool similarity over all token pairs of a batch, and compute RankMe on the patch tokens of a batch rather than on sample embeddings. Training batches pad samples to a common variate count, and the padded tokens are either excluded or corrected by a constant offset per run, measured at its final checkpoint. These values are therefore compared only within their own figure.

Figure 3. Scaling encoder size under data constraints. Columns. The four IsoFLOP (NN, DD) groups listed in the table. Rows. (Top) Patch-token cosine similarity (↓\downarrow). (Centre) RankMe (↑\uparrow) normalised by the encoder width dd. (Bottom) UEA 5-shot balanced accuracy on 23 datasets, using mean-pooled patch tokens (↑\uparrow). Within each column, the eight data budgets DCD_C correspond to {1,2,3,4,5,7,14,32}\{1, 2, 3, 4, 5, 7, 14, 32\} epochs of repetition (colour). OTIS is dotted, OTISv2 is solid. At every size and data budget, OTISv2 has lower token similarity, higher RankMe, and higher accuracy than OTIS.

Embeddings. We use the [CLS] token for OTISv2, as self-distillation supervises it, and the mean of the patch tokens for all other encoders. MOMENT splits series longer than 512 time points into consecutive windows of 512 time points and averages the valid patch tokens of all windows; shorter series are right-padded and masked. OTIS interpolates its time embedding beyond its trained context of 1,008 time points and receives freshly initialised variate embeddings for every unseen dataset. UniTS has no prompt and task tokens for unseen datasets, so these are freshly initialised.

5-shot prototypical classification. This protocol tests how well a few labelled samples describe each class. In each episode, every class contributes k=5k=5 support samples drawn from the training split without replacement, or all of its samples if it has fewer. The mean of the support embeddings of a class forms its prototype. Prototypes and test embeddings are ℓ2\ell_2-normalised, and each test sample takes the label of the prototype with the highest cosine similarity. We average the balanced accuracy over three episodes with new support sets. A fixed seed gives every encoder the same support sets.

kk-NN classification. This protocol tests whether samples of the same class lie close together in the embedding space, using the full training split and no fitted parameters. Following DINO [3], train and test embeddings are ℓ2\ell_2-normalised, and each test sample retrieves its k=5k=5 nearest training samples by cosine similarity, with kk capped at the size of the training split. Neighbour ii votes for its class with weight wi=exp⁡(cos⁡i/τ)w_i=\exp(\operatorname{cos}_i/\tau) and τ=0.07\tau=0.07, and the class that collects the largest total weight is predicted. We keep kk small, as some datasets have fewer than five training samples in a class and a larger neighbourhood would mix classes.

Linear probing. This protocol tests whether the classes are linearly separable in the embedding space. Embeddings are standardised with training-split statistics. We fit an ℓ2\ell_2-regularised multinomial logistic regression with L-BFGS. The regularisation strength is chosen from C∈{10−2,10−1,1,10,102}C \in\{10^{-2}, 10^{-1}, 1, 10, 10^2\} by balanced accuracy on a stratified 20% hold-out split of the training data. Datasets with fewer than 50 training samples or fewer than five per class use stratified cross-validation instead. The final probe is then trained on the full training split.

Per-Block Representation Quality

Setup. Table 3 measures the encoders only at their output, so it leaves open whether the advantage of OTISv2 forms in the last blocks or builds up over depth. If it formed only at the output, the two encoders would differ only in the last blocks. Figure 4 therefore probes the matched 7.1 M runs of OTIS and OTISv2 (the first and last rows of Table 3, three pre-training seeds each) at the output of every block, which we pass through the final normalisation layer of the encoder. Token similarity and RankMe follow Appendix E on the samples of Table 3, so block 12 reproduces the values of that table. 5-shot accuracy uses the [CLS] token of each block on the 23 UEA datasets.

Results. In both encoders, RankMe rises with depth. Normalised by the encoder width dd, the RankMe of OTISv2 is higher at every block, rising from 0.16 at block 1 to 0.39 at block 12, compared with 0.08 to 0.27 for OTIS. OTISv2 also has lower token similarity at every block, between 0.48 and 0.52, whereas that of OTIS stays between 0.64 and 0.66 up to block 6 and then falls to 0.56 at block 12. Both orderings hold at every block in each of the three seeds, so OTISv2 mitigates feature collapse throughout the encoder, not only at its output. 5-shot accuracy also rises with depth in both encoders and levels off over the last blocks of OTISv2 (57.3% and 57.2% at blocks 11 and 12). At block 1, both encoders score almost the same (44.0% for OTISv2 against 43.6% for OTIS). From block 2 onwards, OTISv2 leads in every seed, and its margin grows from 3.1 pp at block 2 to 6.3 pp at block 4, then stays between 5.8 and 7.0 pp. The advantage of OTISv2 thus builds up over the first blocks and is not limited to the last block.

Patch-Token Norms

Setup. Large Vision Transformers [7] develop high-norm artefact tokens that store global information [4]. If such tokens appeared in our encoders, a separate population of patch tokens with much larger norms would emerge. To test this, Figure 5 records the ℓ2\ell_2 norms of the patch tokens at the output of every block, before the final normalisation layer, for the matched 7.1 M runs of OTIS and OTISv2 (one pre-training seed each) on 4,096 pre-training samples with all patches visible.

Figure 5. Neither encoder develops high-norm artefact tokens during pre-training. Patch-token ℓ2\ell_2 norms of the matched runs of OTIS and OTISv2 (N=7.1 MN = 7.1\,\mathrm{M}, D=1.60 BD = 1.60\,\mathrm{B}, DC=32 MD_C = 32\,\mathrm{M}), one pre-training seed each. Colour gives the density. (Left two) Last encoder block over training. (Right two) All encoder blocks at the end of pre-training.

Results. Neither encoder develops a separate high-norm population, so artefact tokens do not explain the higher token similarity of OTIS. The encoders differ only in how their norms grow. At the last block, the norms at the start of training lie between 30 and 100 for OTIS and between 10 and 20 for OTISv2. After training, the norms of OTIS already lie in a narrow band of 200 to 300 at the first block, whereas those of OTISv2 start at about half these values and grow linearly with depth. Both end in a band of 150 to 300 at the last block. OTISv2 thus defers norm growth to later blocks rather than suppressing it. As the two runs differ in architecture, training stabilisers, and objective, this comparison does not show which change causes the difference in norm growth.

Single-Sample Encoder Inspection on 12-Lead ECG

To illustrate how the two encoders represent a single sample, we inspect the last block of the frozen OTIS-7.1M and OTISv2-7.1M encoders of Table 1 on four 12-lead ECGs from the pre-training corpus. Each sample is embedded without masking as a grid of 12 leads by 42 time patches, 504 patches in total. Table 8 summarises the norm of each patch token after the final normalisation layer, a joint PCA of the patch tokens, and the attention that each patch receives.

OTISOTISv2
Token ℓ2\ell_2 norm range[6.65, 10.34][11.50, 20.47]
Token ℓ2\ell_2 norm spread (max / min)1.56×1.78×
Spearman ρ\rho(token norm, raw patch energy)−0.18+0.39
max ∣ρ∣|\rho|(PC1–PC3, raw patch energy)0.62 (PC2)0.45 (PC2)
PCA explained variance, top-3 sum0.350.38
Attention max received (uniform = 0.00198)0.0132 (6.7×)0.0187 (9.4×)
Effective keys per patch query321220
Correlation between patch-query maps (mean)0.720.09

Table 8. Last-block properties on four 12-lead ECG samples (504 patches per sample). Norms are taken after the final normalisation layer, and attention is averaged over the patch queries of each head. Patch energy is the standard deviation of the raw signal in a patch, and the effective keys of a query are the exponential of the entropy of its attention. The two checkpoints are not matched, so these contrasts are illustrative only (App. § 15).

Caveat. The two checkpoints are not matched. OTIS was pre-trained on its own corpus (App. § 11) with 38 variate embeddings and no registers, and OTISv2 on the corpus of Appendix § 8 with four registers. As the corpus, variate layout, and training length differ in addition to the recipe, this section only illustrates single samples and supports no claim of the paper.

Token norms. After the final normalisation layer, the token norms of OTIS span 6.65 to 10.34 and those of OTISv2 span 11.50 to 20.47 (Figure 6), so the largest norm exceeds the smallest by only 1.56× and 1.78×. Neither encoder thus produces outlier tokens, in line with Appendix § 14, whose norms are taken before this layer. The norm correlates only weakly with patch energy, the standard deviation of its raw signal, negatively for OTIS (Spearman ρ=−0.18\rho=-0.18) and positively for OTISv2 (ρ=0.39\rho=0.39).

(a) (a) OTIS.

(b) (b) OTISv2.

Figure 6. Both encoders keep token norms in a narrow band without outliers. Per-patch token ℓ2\ell_2 norms after the final normalisation layer on four 12-lead ECGs. In each subfigure, the top row shows the raw signal, and the bottom row shows the norm of each patch with the signal overlaid, on a colour scale per encoder shared across the four samples.

PCA. The leading components of OTIS vary smoothly across neighbouring patches (Figure 7). One is aligned with the QRS complexes, the large deflections of each heartbeat, and correlates with patch energy (ρ=0.62\rho=0.62), and another varies mainly between leads. Those of OTISv2 vary more from patch to patch, with weaker structure across leads and time (Figure 8). On these four samples, the top three components explain a similar share of the variance in OTIS and OTISv2 (35 % and 38 %). Neighbouring patch tokens of OTISv2 thus differ more, consistent with its lower token similarity.

Figure 7. The leading patch-token components of OTIS vary smoothly across neighbouring patches. Joint PCA of the last-block patch tokens of OTIS over the four samples. Row 1 shows the raw signal, and rows 2 to 4 show PC1 to PC3 with the signal overlaid, on a symmetric colour scale shared across samples.

Figure 8. The leading patch-token components of OTISv2 vary more from patch to patch. Joint PCA of the last-block patch tokens of OTISv2, with the same layout as Figure 7.

Attention. Uniform attention would give each patch 1/504=0.001981/504=0.00198 of the attention. Neither encoder attends uniformly (Figure 9 and Figure 10). In both encoders, one head attends to the QRS-aligned patches, and the 10 % of patches with the highest energy receive 17 % of the attention in OTIS and 23 % in OTISv2, against 10 % under uniform attention. In OTISv2, each of the other two heads concentrates on a few leads. The most attended patch receives 6.7× uniform attention in OTIS and 9.4× in OTISv2. The attention of a patch query spreads over 321 and 220 effective keys, respectively, measured as the exponential of its entropy, so OTISv2 is only moderately more concentrated. The attention maps of single queries show the clearest difference (Figure 11). In OTIS, four random patch queries produce similar maps focused on QRS-aligned patches (pairwise correlation 0.66 to 0.86), and the [CLS] token focuses on one QRS-aligned column of patches (17× uniform attention). In OTISv2, each patch query attends to its own set of patches (pairwise correlation −0.06 to 0.23) and puts 17 to 24 % of its attention on its own lead, against 8 % under uniform attention, while the [CLS] token attends diffusely. On these samples, the patch queries of OTIS thus draw on the same few patches, whereas each query of OTISv2 gathers its own context.

Figure 9. OTIS attends to QRS-aligned patches in one head. Mean last-block attention that each patch receives from all query patches, per head, with the signal overlaid. Row 1 shows the raw signal, and rows 2 to 4 show the three heads.

Figure 10. OTISv2 attends to QRS-aligned patches in one head and to a few leads in the other two. Per-head last-block attention of OTISv2, with the same layout as Figure 9.

Figure 11. Each OTISv2 query attends to its own set of patches. Head-averaged last-block attention of the [CLS] token and four random patch queries on one 12-lead ECG. Column 1 shows the signal with the query positions marked by coloured crosses, column 2 the attention of [CLS], and columns 3 to 6 the attention of each patch query.

Encoder Configurations

Table 9 lists the four configurations of the encoder f(⋅)f(\cdot). The released OTISv2 has 7.1 M parameters, and the scaling study of Section 4.5 uses all four sizes, the 14.8 M, 40 M, and 116 M variants only there. All variants keep the per-head dimension at 64 and the MLP size at four times the encoder width, and grow in both width and depth.

EncoderBlocksEncoder width ddMLP sizeHeadsdhd_hParameters
OTISv2–7.1M121927683647.1 M
OTISv2–14.8M142561,02446414.8 M
OTISv2–40M243201,28056440 M
OTISv2–116M364481,792764116 M

Table 9. Encoder configurations of OTISv2.

Zero-Shot Forecasting

Zero-Shot Forecasting

Setup. OTISv2 is evaluated as an encoder for discriminative tasks, but its pre-training decoder can already forecast, as it reconstructs masked patches at the end of a series. If the feature-space objectives improve the representation as a whole, this forecast should also improve over OTIS without any training. Table 10 therefore compares the released OTIS and OTISv2 encoders on seven standard forecasting datasets [43, 39, 20], with encoder and decoder frozen, so that no parameter is trained or tuned. Each test window gives the encoder a lookback of 336 time points and masks the next 96, 192, or 336 time points, which the decoder reconstructs; for the short weekly ILI series, the lookback is 96 and the horizons are 24, 48, and 72. ETT, Weather, and ILI enter as multi-variate series, and Electricity as one uni-variate series per client. Each variate is normalised with the statistics of the training split, test windows advance by one time point, and we report the mean squared error (MSE) over the horizon, averaged over the three horizons. We do not compare MOMENT and UniTS, as the released MOMENT checkpoints contain no reconstruction head and UniTS was pre-trained on four of the seven datasets. We exclude the common Traffic dataset, as our pre-training corpus contains 803 of its 862 series, including 95 % of its test period, through the GIFT-Eval collection [1].

EncoderETTh1ETTh2ETTm1ETTm2WeatherElectricityILIAvg
OTIS-7.1M1.9161.6112.2301.5000.5650.7977.2952.273
OTIS-43M1.1610.4791.4090.4790.3940.8235.4141.451
OTIS-128M1.0710.3171.0680.3810.3830.8665.8681.422
OTISv2-7.1M1.0480.4271.5060.5460.2570.6175.0741.354

Table 10. Without any training, OTISv2 has the lowest average forecasting error. Zero-shot MSE (↓) of the frozen released encoders and their pre-training decoders, averaged over the horizons 96, 192, and 336 (24, 48, and 72 for ILI). Bold marks the best encoder per dataset.

Results. OTISv2 forecasts better than OTIS of the same size on all seven datasets, and its average MSE of 1.354 is the lowest, below the 1.422 of the 18×\times larger OTIS-128M. OTISv2 also has the lowest mean absolute error (0.719 against 0.761). It forecasts best on ETTh1, Weather, Electricity, and ILI, whereas OTIS-128M forecasts best on ETTh2, ETTm1, and ETTm2. The gains of the feature-space objectives thus transfer to a generative task without any training, although thorough forecasting evaluation remains future work.

Robustness to Input Corruptions

Setup. The benchmarks of Table 1 contain clean recordings, whereas real sensors add noise, spikes, gaps, and failures. If the frozen embeddings were robust, accuracy under mild corruptions would stay close to its clean value. Table 11 therefore corrupts the test splits of the 23 UEA datasets, keeps the training splits clean, and evaluates the encoders of Table 1 with the three protocols of Appendix § 12. We apply four corruptions at a mild severity: Gaussian noise with 5 % of the standard deviation of each variate, spikes of ±5\pm5 standard deviations on 0.5 % of the time points, three missing segments that cover 10 % of one variate per sample and are filled with the mean of its remaining values, and variate dropout, which removes one variate per dataset. All encoders receive the same corrupted inputs, which are not re-normalised. Besides the [CLS] embedding of OTISv2, we also evaluate its mean-pooled patch tokens, the embedding used for all other encoders.

EncoderCleanNoiseSpikesMissingDropout{sup}†`
5-shot prototypical classification
MOMENT-308M52.8452.2050.0452.8352.82
OTIS-7.1M55.5355.4953.4355.0253.75
OTIS-43M57.5357.4855.4957.2855.41
OTIS-128M58.7058.6356.9758.1956.61
OTISv2-7.1M60.0056.8256.9858.3956.04
OTISv2-7.1M, mean-pooled55.7754.6753.4555.0853.06
kk-NN classification
MOMENT-308M58.3757.8556.0958.0458.34
OTIS-7.1M64.4264.2861.3663.5159.88
OTIS-43M64.8065.0362.5764.5561.91
OTIS-128M65.7865.6963.9664.9662.26
OTISv2-7.1M66.1761.8561.9663.3961.51
OTISv2-7.1M, mean-pooled62.3260.8659.1461.8059.13
Linear probing
MOMENT-308M64.3762.9758.0363.1264.54
OTIS-7.1M67.7366.9562.6066.3061.96
OTIS-43M69.2769.4166.4168.7163.95
OTIS-128M69.1869.2166.4368.7364.04
OTISv2-7.1M69.8566.1064.4666.1463.35
OTISv2-7.1M, mean-pooled68.5165.8063.0766.4462.93

Table 11. OTISv2 loses more accuracy under corruptions, mostly through its [CLS] embedding. Balanced accuracy (↑\uparrow) on the 23 UEA datasets with clean training splits and corrupted test splits. Bold marks the best encoder per protocol and condition. UniTS and the two smaller MOMENT encoders, which lead in no setting, are omitted, and the clean accuracies reproduce Table 1 within 0.1 pp.

†On the 22 multi-variate datasets.

Results. Under every corruption and protocol, OTISv2 loses more accuracy than each OTIS encoder. Its UEA 5-shot accuracy drops by 1.6 to 5.7 pp, against 0.1 to 4.0 pp for OTIS-128M, with the largest gap under noise (3.18 against 0.07 pp). OTISv2 still matches OTIS-128M under spikes (56.98 % against 56.97 %) and leads under missing segments (58.39 % against 58.19 %), but trails it by 1.81 pp under noise and 0.58 pp under variate dropout. Under kk-NN classification and linear probing, OTIS-128M, OTIS-43M, or MOMENT-308M leads every corrupted setting, by 0.75 to 3.84 pp over OTISv2. Most of this sensitivity enters through the [CLS] embedding, as the mean-pooled patch tokens of OTISv2 lose less accuracy in 11 of the 12 settings, for example 1.10 instead of 3.18 pp under noise in 5-shot classification. The encoder itself is also more sensitive to noise, as even its mean-pooled patch tokens lose more than those of OTIS-7.1M (1.10 against 0.04 pp). Three of the 23 datasets, Handwriting, CharacterTrajectories, and SpokenArabicDigits, account for 85 % of the additional relative loss of OTISv2 under noise in 5-shot classification. The selective [CLS] summary that yields OTISv2 its clean advantage thus also passes input faults on more strongly, so for noisy inputs its mean-pooled patch tokens are the more robust embedding.

NeurIPS Paper Checklist

  1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

  1. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

  1. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A] Justification: This work is a fully empirical study.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]

Justification: The paper provides open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

  1. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

  1. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports appropriate information about statistical significance of the experiments.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.).

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Experimental details are provided in the respective section.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [N/A]

Justification: Not applicable.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: Not applicable.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets (code and pre-trained OTISv2 weights) will be released with documentation upon acceptance, as stated in the abstract and conclusion.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: Not applicable.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: Not applicable.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: Not applicable.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]T. Aksu, G. Woo, J. Liu, X. Liu, C. Liu, S. Savarese, C. Xiong, and D. Sahoo. GIFT-Eval: A benchmark for general time series forecasting model evaluation. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024.arxiv.org/abs/2410.10393
  2. [2]A. Bagnall, H. A. Dau, J. Lines, M. Flynn, J. Large, A. Bostrom, P. Southam, and E. Keogh. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075, 2018.
  3. [3]M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In International Conference on Computer Vision, 2021.arxiv.org/abs/2104.14294
  4. [4]T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. In International Conference on Learning Representations, 2024.arxiv.org/abs/2309.16588
  5. [5]H. A. Dau, A. Bagnall, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. A. Ratanamahatana, and E. Keogh. The UCR time series archive. IEEE/CAA Journal of Automatica Sinica, 2019.arxiv.org/abs/1810.07758
  6. [6]Deutscher Wetterdienst. Temperature dataset. DWD, 2024. https://www.dwd.de/DE/leistungen/klimadatendeutschland/klarchivtagmonat.html.
  7. [7]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  8. [8]S. Gao, T. Koker, O. Queen, T. Hartvigsen, T. Tsiligkaridis, and M. Zitnik. UniTS: A unified multi-task time series model. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2403.00131
  9. [9]Q. Garrido, R. Balestriero, L. Najman, and Y. LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning, 2023.
  10. [10]J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
  11. [11]M. Goswami, K. Szafer, A. Choudhry, Y. Cai, S. Li, and A. Dubrawski. MOMENT: A family of open time-series foundation models. In International Conference on Machine Learning, 2024.
  12. [12]B. Gow, T. Pollard, L. A. Nathanson, A. Johnson, B. Moody, C. Fernandes, N. Greenbaum, J. W. Waks, P. Eslami, T. Carbonati, A. Chaudhari, E. Herbst, D. Moukheiber, S. Berkowitz, R. Mark, and S. Horng. MIMIC-IV-ECG: Diagnostic electrocardiogram matched subset. PhysioNet, 2023.DOI
  13. [13]P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.arxiv.org/abs/1706.02677
  14. [14]B. Heo, S. Park, D. Han, and S. Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, 2024.arxiv.org/abs/2403.13298
  15. [15]J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2203.15556
  16. [16]S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, X. Zhang, Z. L. Thai, K. Zhang, C. Wang, Y. Yao, C. Zhao, J. Zhou, J. Cai, Z. Zhai, N. Ding, C. Jia, G. Zeng, D. Li, Z. Liu, and M. Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In Conference on Language Modeling, 2024.arxiv.org/abs/2404.06395
  17. [17]G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016.arxiv.org/abs/1603.09382
  18. [18]J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
  19. [19]D. Kiyasseh, T. Zhu, and D. A. Clifton. CLOCS: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning, 2021.arxiv.org/abs/2005.13249
  20. [20]G. Lai, W.-C. Chang, Y. Yang, and H. Liu. Modeling long- and short-term temporal patterns with deep neural networks. In International ACM SIGIR Conference on Research and Development in Information Retrieval, 2018.arxiv.org/abs/1703.07015
  21. [21]C. Lessmeier, J. K. Kimotho, D. Zimmer, and W. Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference, 2016.DOI
  22. [22]I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  23. [23]M. W. McCracken and S. Ng. FRED-MD: A monthly database for macroeconomic research. Journal of Business & Economic Statistics, 2016.
  24. [24]N. Muennighoff, A. M. Rush, B. Barak, T. Le Scao, A. Piktus, N. Tazi, S. Pyysalo, T. Wolf, and C. Raffel. Scaling data-constrained language models. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.16264
  25. [25]Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.arxiv.org/abs/2211.14730
  26. [26]M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2304.07193
  27. [27]K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi, et al. Capabilities of Gemini models in medicine. arXiv preprint arXiv:2404.18416, 2024.arxiv.org/abs/2404.18416
  28. [28]A. Sablayrolles, M. Douze, C. Schmid, and H. Jégou. Spreading vectors for similarity search. In International Conference on Learning Representations, 2019.arxiv.org/abs/1806.03198
  29. [29]N. Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.arxiv.org/abs/2002.05202
  30. [30]O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. DINOv3. Transactions on Machine Learning Research, 2026.arxiv.org/abs/2508.10104
  31. [31]J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
  32. [32]S. B. Taieb, G. Bontempi, A. F. Atiya, and A. Sorjamaa. A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition. Expert Systems with Applications, 2012.
  33. [33]Ö. Turgut, P. Müller, P. Hager, S. Shit, S. Starck, M. J. Menten, E. Martens, and D. Rueckert. Unlocking the diagnostic potential of electrocardiograms through information transfer from cardiac magnetic resonance imaging. Medical Image Analysis, 101:103451, 2025.DOI
  34. [34]Ö. Turgut, P. Müller, M. J. Menten, and D. Rueckert. OTIS: Learning high-quality time series features with tiny encoders. Transactions on Machine Learning Research, 2026.
  35. [35]H. Van Dijk, G. Van Wingen, D. Denys, S. Olbrich, R. Van Ruth, and M. Arns. The two decades brainclinics research archive for insights in neurophysiology (TDBRAIN) database. Scientific Data, 2022.DOI
  36. [36]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1706.03762
  37. [37]Z. Wang, X. Xu, W. Zhang, G. Trajcevski, T. Zhong, and F. Zhou. Learning latent seasonal-trend representations for time series forecasting. In Advances in Neural Information Processing Systems, 2022.DOI
  38. [38]G. Woo, C. Liu, D. Sahoo, A. Kumar, and S. Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In International Conference on Learning Representations, 2022.arxiv.org/abs/2202.01575
  39. [39]H. Wu, J. Xu, J. Wang, and M. Long. Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.13008
  40. [40]K. Zhang, R. Zhou, E. Adhikarla, Z. Yan, Y. Liu, J. Yu, Z. Liu, X. Chen, B. D. Davison, H. Ren, et al. A generalist vision–language foundation model for diverse biomedical tasks. Nature Medicine, 2024.DOI
  41. [41]X. Zhang, Z. Zhao, T. Tsiligkaridis, and M. Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2206.08496
  42. [42]W.-L. Zheng and B.-L. Lu. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 2015.DOI
  43. [43]H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Conference on Artificial Intelligence, 2021.arxiv.org/abs/2012.07436
  44. [44]J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.arxiv.org/abs/2111.07832

Paper details

vlnc:W_2tqyq6py
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents