WorksComputer Science
Self-Supervised Latent Action-Conditioned World Modeling for Time Series
Abstract
Time-series data play an important role in various tasks, such as weather forecasting, web traffic analysis, and speech recognition. Recently, pioneering studies have increasingly adopted Joint-Embedding Predictive Architecture (JEPA) based encoders to learn time-series representations through latent-space prediction. However, they often overlook the underlying factors governing transitions between time-series observations. In this work, we hypothesize that explicitly modeling the information behind changes in time series can facilitate time-series modeling. Building on this hypothesis, we propose ACTS-WM, a self-supervised latent action-conditioned world model framework for time-series analysis. First, we introduce a latent action modeling approach that infers the underlying information behind time-series transitions as latent actions, without requiring explicit action annotations. We then develop a JEPA-based encoder and an action-conditioned world model to predict target time-series representations conditioned on the learned latent actions and task information. We evaluate ACTS-WM across 4 representative tasks on 22 datasets from diverse domains, covering univariate and multivariate forecasting and classification. Comprehensive experiments demonstrate that ACTS-WM can improve downstream performance over pretrained JEPA-based encoders by learning complementary representations.
CCS Concepts
• Information systems → Data mining; • Computing methodologies → Learning latent representations.
ACM Reference Format:
Chia-Han Kuo*, Chengshuai Zhao*, Huan Liu, and Lo Pang-Yun Ting. 2027. Self-Supervised Latent Action-Conditioned World Modeling for Time Series. In . ACM, New York, NY, USA, 12 pages. https://doi.org/XXXXXXXX. XXXXXXXX
Introduction
Time-series data play a pivotal role across modern data-driven systems, ranging from weather forecasting and energy monitoring to real-time speech recognition [14, 39, 49]. In Web ecosystems, modern online services continuously generate massive volumes of time-series data, including Web traffic, user activity logs, and network monitoring signals [34, 47], providing abundant resources for web knowledge discovery. Recently, time-series foundation models [7, 19, 26] have emerged as a state-of-the-art technique to leverage substantial time-series data. By embedding temporal data into a condensed latent space, they learn general-purpose representations supporting various tasks, including forecasting and classification.1
Equal contribution.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Conference’17, Washington, DC, USA
© 2027 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/XXXXXXX.XXXXXXX To improve generalization across tasks and domains, recent studies have increasingly explored self-supervised representation learning for time series [13, 30]. Among these approaches, Joint-Embedding Predictive Architectures (JEPA) [24], which learn representations by predicting target embeddings from contextual observations in latent space, demonstrate effectiveness across downstream time-series tasks [10, 12, 23].
Despite this progress, existing approaches primarily learn representations from observed temporal patterns without explicitly modeling the information underlying how time series change over time. Such information can characterize temporal transitions that are predictive of future dynamics while revealing invariant semantics shared across observations, thereby providing informative signals for both forecasting and classification. For example, pioneering studies in visual control and robotics have explored learning latent actions [36, 45] from observed transitions. Inspired by this intuition, we hypothesize that explicitly modeling the information underlying temporal changes can provide richer supervisory signals and facilitate more effective time-series representation learning.
However, modeling the information underlying temporal changes in time series presents several challenges. (i) Unlike conventional settings such as robotics [3, 21, 31, 44], where an agent interacts with its environment through explicit control actions and observes the resulting state transitions, time-series data are typically collected as passive observations without corresponding action signals, as shown in Figure 1. As a result, the latent factors that drive transitions between observations are not directly available and must instead be inferred from the observed temporal evolution itself. (ii) Further, multivariate time series can contain varying numbers of channels across datasets, making it difficult to model temporal transitions within a unified framework. These challenges call for a general mechanism that can infer latent transition factors from observed changes while remaining robust to heterogeneous and dataset-specific channel configurations.
Figure 1. Model comparison. (a) Conventional action-conditioned world models in robotics rely on explicit action signals. (b) JEPA-based time-series encoders learn representations without explicit action conditioning. (c) ACTS-WM learns latent actions from observational time series, enabling action-conditioned world modeling for time-series analysis without requiring explicit action signals.
To address these challenges, we propose ACTS-WM (Action-Conditioned Time-Series World Modeling), a self-supervised latent action-conditioned world modeling framework for time-series analysis. ACTS-WM explicitly learns latent actions from observed time series and then leverages an action-conditioned world model for downstream forecasting and classification. Specifically, ACTS-WM consists of two stages. First, we introduce self-supervised latent action modeling to learn discrete latent actions from observed time-series transitions without requiring action annotations. We further learn to infer these actions from observed time series alone, enabling their use when target time series are unavailable. Second, we pretrain a JEPA-based time-series encoder to learn informative latent representations of time-series observations. We then develop an action-conditioned world model that incorporates the learned latent actions to model transitions between observations in the encoder’s latent representation space. Through task-conditioned modeling, ACTS-WM is able to provide complementary latent representations to support downstream forecasting and classification within a unified framework.
Key Takeaways. Our contributions are summarized as follows:
We propose a novel perspective for time-series representation learning. We argue that the latent factors underlying transitions between observations provide valuable supervisory signals for learning transferable representations.
Building on the hypothesis, we develop a task-conditioned latent action modeling approach that captures temporal evolution and variations across tasks of the same class.
We propose ACTS-WM. To the best of our knowledge, it is the first self-supervised latent action-conditioned world-modeling framework tailored for time-series analysis.
To validate the hypothesis, we conduct extensive experiments across univariate and multivariate forecasting and classification tasks on 22 time-series datasets from diverse domains. The results demonstrate the effectiveness of ACTS-WM in providing complementary representations, while further ablation studies and qualitative analyses reveal meaningful transition patterns captured by the learned latent actions.
Related Work
Time Series Analysis
For time-series analysis models, we explore time series foundation models and JEPA-based time-series models. Recent studies have explored various approaches to time-series foundation models. These studies introduce novel architectures that achieve remarkable performance on multi-domain benchmarks. MOMENT [19] and UniTS [15] learn expressive representations that can be directly adapted to various downstream tasks. Toto [7], Moirai [26, 27, 42], and Time-MoE [38], also learn expressive representations but focus on forecasting ability.
In parallel, Joint-Embedding Predictive Architectures (JEPA) [24] have gained increasing attention in time-series analysis. TS-JEPA [12] and CHARM [10] adapt the JEPA architecture to various time-series analysis tasks. HEPA [32] and SC-JEPA [23] learn to predict future latent representations and apply them to rare-event prediction and anomaly detection. Lat-PTN [40] further combines JEPA with zero-shot forecasting capabilities. Beyond representation learning, JEPA-based time-series world models have explored incorporating additional action signals, such as mechanical states and control inputs [33, 37] or external information from other modalities [8].
World Models
World models aim to capture the dynamics of the physical world and support a wide range of downstream tasks. Dreamer 4 [22] learns a scalable latent dynamics model, while DIAMOND [2] adopts a diffusion model as a pixel-space world model. Both then train policies via reinforcement learning in imagination. Extending the JEPA framework, V-JEPA 2-AC [3] plans in latent space to control robot arms for picking and placing objects given a goal image. H-JEPA [48] further introduces a hierarchy of latent world models that enables long-horizon planning at multiple timescales. Video-generation-based world models such as COSMOS [1] and Genie [6] generate physically plausible, action-controllable videos. Some works aim to combine policy models and world models. PaW [28] employs a single pretrained large language model (LLM) to jointly train a policy and a world model, while CoLA-World [41] jointly trains a latent action model (LAM) with a pretrained video world model, allowing the two to co-evolve.
Latent Action Models
To learn dynamics from observational data without explicit action labels, recent studies [16, 36, 41, 45] have explored latent action learning to infer action representations directly from observed transitions. These studies employ inverse dynamics models (IDMs) or latent action encoders to infer latent actions from observed transitions, and forward dynamics models (FDMs) or latent action decoders to predict subsequent observations conditioned on the inferred actions. LAPO [36], LAPA [45] and CoLA-World [41] further quantize the learned latent actions into a discrete action space.
Despite these advances, most time-series models do not explicitly model the underlying information governing transitions between observations. Existing world models primarily focus on visual and language domains, while time-series world models often rely on additional action signals, such as system states or control inputs. Although latent action models enable modeling transitions without explicit action labels, they have mainly been explored in video-based settings. In our work, ACTS-WM aims to learn latent actions directly from time-series observations and uses them to condition world modeling without requiring action annotations.
Preliminaries and Problem Definition
Problem Formulation
Let denote a time series with channels and time steps, where corresponds to the multivariate setting. Given time-series observations collected from multiple domains, we define the pretraining dataset as: , where denotes the observed time series, and denotes the target time series. For instance, in forecasting tasks, represents the subsequent temporal segment following . The task tag indicates whether the observation pair corresponds to a forecasting task with a specified prediction horizon or a classification task.
Given the pretraining dataset , ACTS-WM aims to learn latent actions that capture the underlying transition information between time-series observations without requiring explicit action annotations. At the evaluation phase, given an unseen time series from either a pretraining domain or a previously unseen domain, along with a task tag , ACTS-WM infers a latent action and uses it to predict the latent representations of the target time series. These predicted representations are subsequently used for downstream forecasting and classification.
Inverse and Forward Dynamics Models
Inverse and Forward dynamics models are two main components in recent latent action models to learn latent actions from video data, as discussed in § 2.3. The inverse dynamics model (IDM) infers latent actions from past observations and future observation , while the forward dynamics model (FDM) reconstructs the future observation given the past observations and a latent action obtained from IDM. These processes are formulated as:
In studies [36, 41, 45], vector quantization [20] is further applied to latent actions before passing them to the FDM, serving as an information bottleneck that prevents the IDM from simply copying future observations into the action representation.
In our work, we build upon these approaches and extend latent action modeling to time-series data.
Methodology
Our ACTS-WM framework consist of two stages, as shown in Figure 2: (1) Latent Action Modeling learns discrete latent actions from observed time series (Sec. 4.1), and (2) Action-Conditioned Time Series World Model uses the learned actions and a JEPA-based encoder to predict representations of target time series (Sec. 4.2).
Figure 2. Overview of ACTS-WM.
Latent Action Modeling in Time Series
In our work, the underlying information that explains the transition between observed and target time-series observations is regarded as a latent action. Building upon the latent action learning framework described in § 3.2, we introduce several designs to learn latent actions from time-series data.
Given an observation pair , we first encode the time-series observations and task information separately. These representations are then incorporated into the inverse and forward dynamics models to learn latent actions.
Channel and Temporal Fusions
To accommodate varying numbers of channels and sequence lengths in time-series data, we develop a fusion module consisting of two components: Channel Fusion and Temporal Fusion. For Channel Fusion, we first fuse features from each channel independently using shared convolutional layers, followed by multi-head self-attention and learnable-query attention pooling to capture and aggregate cross-channel dependencies. Temporal Fusion then employs dilated temporal convolutional networks (TCNs) [5, 25] to capture multi-scale temporal patterns. Given a time series , its encoding process is formulated as:
where is the dimension of the resulting representation. The extracted features are subsequently used for latent action modeling.
Task Embedding
To distinguish different types of transitions, including forecasting with different prediction horizons and classification, we introduce learnable Task Embeddings. Given a task tag , its embedding is obtained as:
where denotes the one-hot vector corresponding to the task tag , and is a learnable embedding table with embedding dimension . Each task tag is assigned a unique index to construct its one-hot vector. The embedding table is optimized jointly with the model parameters through the training objective, allowing the model to learn task-specific representations.
Inverse and Forward Dynamics Models
Building upon the preceding designs, we redesign the inverse dynamics model (IDM) to infer a continuous latent action from an observation pair . Since and may have different sequence lengths, we employ the fusion module (§ 4.1.1) to map them into fixed-dimensional representations of the same dimension. These representations are then combined with the task embedding (§ 4.1.2) to infer the latent action :
where and are learnable projection matrices, and denotes concatenation. The inferred action represents the transition of the observation pair and is shared across all channels.
Following [36, 41, 45], the continuous action is then quantized into a discrete latent action using vector quantization (details in § 4.1.4). Correspondingly, we redesign the forward dynamics model (FDM) to reconstruct the target observation from and the quantized latent action , formulated as:
where denotes the reconstructed target observation. Specifically, we employ a one-dimensional U-Net [35] with shared weights across channels to accommodate varying numbers of channels. The FDM incorporates the quantized latent action , the fused representation , and the task embedding as conditioning information.
Finally, the IDM and FDM are jointly trained to minimize the reconstruction loss for time series and a quantization loss that encourages the continuous latent action to remain close to its quantized representation using the following objective:
where the first term measures the reconstruction error, and the second term denotes the quantization loss weighted by . Here, denotes the stop-gradient operator. This objective encourages the IDM to encode information relevant to the transition toward in the latent action.
Latent Action Selection
We describe the latent action quantization process and how latent actions are obtained at the evaluation phase by using the latent policy.
Latent Action Quantization. Following previous studies [36, 41, 45], vector quantization is applied to the continuous latent action to introduce an information bottleneck. Specifically, a codebook is constructed consisting of code vectors, each with dimension , and the continuous latent action is divided into slots. Each slot is quantized by selecting its nearest code vector from the codebook, and the selected vectors are concatenated to form the quantized latent action . The codebook vectors are updated using an exponential moving average, while determines the number of available codes and determines the number of code selections per action.
Latent Policy. Following LAPO [36], since is unavailable at the evaluation phase, we train a latent policy to infer latent actions from observed time series and the task tag alone. With the pretrained IDM frozen, the latent policy is trained to regress the continuous latent actions produced by the IDM from observation pairs. At the evaluation phase, the policy’s output is quantized using the learned codebook to obtain the latent action.
Action-Conditioned Time-Series World Model
After learning latent actions from time-series observations, we develop an action-conditioned world model by adapting a JEPA-based encoder for time-series representation learning and training a predictor to infer representations of target observations.
JEPA-based Encoder
We build upon V-JEPA 2 [3], which learns predictive spatiotemporal representations from video data, making it suitable for time-series modeling. To adapt it to time-series data, we introduce one-dimensional temporal patches and channel-independent encoding with shared weights, allowing the model to handle varying numbers of channels. The adapted encoder maps a time series to its latent representation: , where is the number of temporal patches and is the representation dimension. During pretraining, we employ interior-block and suffix masking and retain the exponential moving average (EMA) target encoder of V-JEPA 2, which provides stable representations for masked prediction.
To further account for dependencies across channels, we introduce a multi-channel consistency term. Given the predicted and target representations and , respectively, we define the multi-channel distance function as:
where and denote the predicted and target representations of channel at the masked positions, respectively, and weights the multi-channel consistency term.
Given a binary temporal mask indicating the masked patch positions and its complement (unmasked positions), our JEPA-based encoder and predictor are jointly trained to minimize:
where denotes the representations encoded from the unmasked patches, predicts the representations at the masked positions, and denotes the corresponding target representations obtained from the EMA target encoder.
By optimizing Eq. (7), the JEPA-based encoder learns latent representations of time-series observations for subsequent world modeling.
Action-Conditioned Predictor
V-JEPA 2-AC [3] is a latent action-conditioned world model post-trained from V-JEPA 2. Building upon this approach, we adapt its action-conditioned predictor (with parameters ) to model transitions between time-series representations. Given the JEPA-based encoder , the quantized latent action , and the task tag , the predictor estimates the representation of target observation as: . We then optimize the action-conditioned predictor using the multi-channel distance function introduced in Eq. (6):
Overall, the training objectives of ACTS-WM are as follows:
where and are parameters of latent action modeling and JEPA pretraining, respectively.
Finally, by optimizing Eq. (9), ACTS-WM learns to predict target representations for time-series observations across different domains, sequence lengths, and numbers of channels, supporting different downstream tasks. While ACTS-WM is developed based on V-JEPA 2, the framework can also be integrated with other JEPA-based encoders and general time-series representation models.
Experiments
Experimental Setup
Dataset
We collect univariate and multivariate time-series datasets from various sources, including the UEA [4] and UCR [9] archives, forecasting benchmarks from Autoformer [43], WebTraffic [11], and WikiTraffic [17, 18]. We use 21 datasets for ACTS-WM pretraining and 22 for downstream evaluation. Dataset details and usage are summarized in Table 4, while preprocessing and data splits are described in Appendix § 8.§ 2.
| Dataset | Pretrain | Evaluation | #T | #C |
| Univariate classification* | ||||
| Computers | ✓ | 180,000 | 1 | |
| Earthquakes | ✓ | 164,864 | 1 | |
| EOGHorizontalSignal | ✓ | 452,500 | 1 | |
| EthanolLevel | ✓ | 882,504 | 1 | |
| InsectWingbeatSound | ✓ | 56,320 | 1 | |
| SemgHandMovementCh2 | ✓ | 675,000 | 1 | |
| ShapesAll | ✓ | 307,200 | 1 | |
| UWaveGestureLibraryX | ✓ | 282,240 | 1 | |
| WormsTwoClass | ✓ | 162,900 | 1 | |
| WebTraffic | ✓ | 504,000 | 1 | |
| Multivariate classification* | ||||
| SelfRegulationSCP2 | ✓ | 230,400 | 7 | |
| SpokenArabicDigits | ✓ | 263,224 | 13 | |
| ArticularyWordRecognition | ✓ | ✓ | 39,600 | 9 |
| BasicMotions | ✓ | ✓ | 4,000 | 6 |
| CharacterTrajectories | ✓ | ✓ | 169,218 | 3 |
| FingerMovements | ✓ | ✓ | 15,800 | 28 |
| AtrialFibrillation | ✓ | 9,600 | 2 | |
| Cricket | ✓ | 129,276 | 6 | |
| Epilepsy | ✓ | 28,222 | 3 | |
| ERing | ✓ | 1,950 | 4 | |
| HandMovementDirection | ✓ | 64,000 | 10 | |
| Handwriting | ✓ | 22,800 | 3 | |
| Libras | ✓ | 8,100 | 2 | |
| LSST | ✓ | 88,524 | 6 | |
| NATOPS | ✓ | 9,180 | 24 | |
| RacketSports | ✓ | 4,530 | 6 | |
| SelfRegulationSCP1 | ✓ | 240,128 | 6 | |
| UWaveGestureLibrary | ✓ | 37,800 | 3 | |
| Forecasting | ||||
| ETTh1 | ✓ | ✓ | 17,420 | 7 |
| ETTh2 | ✓ | ✓ | 17,420 | 7 |
| ETTm1 | ✓ | ✓ | 69,680 | 7 |
| ETTm2 | ✓ | ✓ | 69,680 | 7 |
| Weather | ✓ | ✓ | 52,696 | 21 |
| WikiTraffic* | ✓ | 56,254 | 1 |
Table 4. Datasets used for ACTS-WM pretraining and downstream evaluation. Univariate and multivariate classification datasets are from UCR [9] and UEA [4], respectively, except WebTraffic [11]. Forecasting datasets are from Autoformer [43], except WikiTraffic [17, 18]. (#T refers to the number of time steps, and #C refers to the number of channels).
Baselines
We compare ACTS-WM with publicly available JEPA-based time-series encoders: TS-JEPA [12] and SC-JEPA [23]. We also evaluate our JEPA-based encoder (§ 4.2.1), adapted from V-JEPA 2 to time series, denoted as “JEPA”.
For downstream evaluation, we employ task-specific heads for all methods: Linear, Ridge, and NLH (non-linear head) for forecasting, and Linear, MLP, and SVM for classification. Using these heads, we evaluate three representation settings for each JEPA-based encoder:
“Base” uses only the pretrained encoder’s representations.
“Pred. Only” uses only the representations of the target time series predicted by ACTS-WM.
“Obs.+Pred.” concatenates the representation of the observed time series from the JEPA-based encoder and the representation of the target time series by ACTS-WM.
Since TS-JEPA supports only univariate inputs, we evaluate it only on univariate tasks. Detailed settings for baselines and downstream heads are provided in Appendix § 8.§ 3.
Evaluation and Implementation Details
For evaluation, we use accuracy (Acc.) and F1-score (F1) for classification, and MSE and MAE on normalized targets for forecasting.
The key hyperparameter settings are as follows. For forecasting, the input length and prediction horizon are set to 320 and 48 time steps, respectively. For latent action modeling, we set (§ 4.1.1), (§ 4.1.2), and (Eq. (5)). The VQ codebook (§ 4.1.4) contains code vectors of dimension , with code selections per action. For the JEPA-based encoder (§ 4.2.1), we set .
For the main experiments, we present representative results using selected downstream heads, averaged over three runs. All experiments were conducted on two NVIDIA GB10 Grace Blackwell Superchips. Detailed settings are provided in Appendix § 8.§ 2, and full results are provided in Appendix § 8.§ 4.
Performance Comparison
Tables 1 and 2 represent the results. Key observations are as follows:
For the forecasting tasks in Table 1, ACTS-WM consistently improves forecasting performance compared to the base model of JEPA-based encoders (“Base”), particularly when using predicted target representations alone (“Pred. Only”). With our JEPA, the “Pred. Only” variant reduces MSE on all 11 forecasting settings and achieves the lowest MSE across all five multivariate forecasting datasets. Similar improvements are observed with TS-JEPA and SC-JEPA, although their effectiveness varies across datasets and representation settings. Notably, on WikiTraffic,
Table 1: Time-series forecasting results averaged across three runs. Datasets marked with ‡ are not included in pre-training ACTS-WM. The best result is in bold, and ★ marks results with ACTS-WM that outperform the original base models. The win rate is the percentage of datasets where ACTS-WM outperforms the base model. Win rates above 50% are shaded.
| Dataset | Metric | Base JEPAACTS | Base TS-JEPA | Base SC-JEPA | + ACTS-WM (Pred. Only) JEPAACTS | + ACTS-WM (Pred. Only) TS-JEPA | + ACTS-WM (Pred. Only) SC-JEPA | + ACTS-WM (Obs.+Pred.) JEPAACTS | + ACTS-WM (Obs.+Pred.) TS-JEPA | + ACTS-WM (Obs.+Pred.) SC-JEPA |
| Univariate Time Series Forecasting | ||||||||||
| WikiTraffic‡ | MSE | 1.167 | 1.047 | 1.413 | 1.063★ | 0.972★ | 1.363★ | 1.164★ | 1.040★ | 1.906 |
| MAE | 0.705 | 0.664 | 0.807 | 0.652★ | 0.610★ | 0.773★ | 0.704★ | 0.677 | 0.981 | |
| Weather | MSE | 0.007 | 0.002 | 0.003 | 0.003★ | 0.001★ | 0.003 | 0.007 | 0.002 | 0.003 |
| MAE | 0.045 | 0.031 | 0.038 | 0.042★ | 0.025★ | 0.038 | 0.046 | 0.028★ | 0.044 | |
| ETTh1 | MSE | 0.100 | 0.097 | 1.631 | 0.084★ | 0.157 | 0.728★ | 0.086★ | 0.091★ | 1.739 |
| MAE | 0.248 | 0.240 | 1.142 | 0.220★ | 0.310 | 0.710★ | 0.227★ | 0.234★ | 1.093★ | |
| ETTh2 | MSE | 0.478 | 0.402 | 1.478 | 0.351★ | 0.351★ | 1.358★ | 0.444★ | 0.396★ | 2.140 |
| MAE | 0.516 | 0.508 | 1.018 | 0.467★ | 0.456★ | 0.985★ | 0.497★ | 0.500★ | 1.197 | |
| ETTm1 | MSE | 0.044 | 0.045 | 1.312 | 0.035★ | 0.045 | 0.793★ | 0.043★ | 0.054 | 0.775★ |
| MAE | 0.158 | 0.160 | 0.990 | 0.135★ | 0.158★ | 0.753★ | 0.155★ | 0.177 | 0.736★ | |
| ETTm2 | MSE | 0.184 | 0.197 | 1.602 | 0.138★ | 0.184★ | 1.673 | 0.175★ | 0.133★ | 1.073★ |
| MAE | 0.332 | 0.352 | 1.063 | 0.286★ | 0.332★ | 1.043★ | 0.324★ | 0.270★ | 0.865★ | |
| Win rate vs. Base | MSE | – | – | – | 100% | 67% | 67% | 83% | 67% | 33% |
| MAE | – | – | – | 100% | 83% | 83% | 83% | 67% | 50% | |
| Multivariate Time Series Forecasting | ||||||||||
| Weather | MSE | 0.257 | – | 0.590 | 0.238★ | – | 0.471★ | 0.259 | – | 0.514★ |
| MAE | 0.225 | – | 0.500 | 0.238 | – | 0.418★ | 0.225 | – | 0.456★ | |
| ETTh1 | MSE | 1.218 | – | 1.321 | 0.678★ | – | 1.129★ | 1.196★ | – | 1.209★ |
| MAE | 0.742 | – | 0.888 | 0.557★ | – | 0.825★ | 0.737★ | – | 0.855★ | |
| ETTh2 | MSE | 0.366 | – | 1.890 | 0.276★ | – | 1.395★ | 0.297★ | – | 2.084 |
| MAE | 0.423 | – | 1.080 | 0.376★ | – | 0.922★ | 0.396★ | – | 1.142 | |
| ETTm1 | MSE | 0.597 | – | 1.434 | 0.489★ | – | 1.105★ | 0.564★ | – | 1.599 |
| MAE | 0.513 | – | 0.924 | 0.456★ | – | 0.805★ | 0.503★ | – | 0.987 | |
| ETTm2 | MSE | 0.161 | – | 1.913 | 0.152★ | – | 1.596★ | 0.156★ | – | 1.740★ |
| MAE | 0.281 | – | 1.090 | 0.270★ | – | 0.973★ | 0.276★ | – | 1.031★ | |
| Win rate vs. Base | MSE | – | – | – | 100% | – | 100% | 80% | – | 60% |
| MAE | – | – | – | 80% | – | 100% | 80% | – | 60% |
Table 1.
Table 2: Time-series classification results averaged over three runs. Datasets marked with ‡ are not included in pre-training ACTS-WM. The best result is in bold, and ★ marks results with ACTS-WM that outperform the original base models. The win rate is the percentage of datasets where ACTS-WM outperforms the base model. Win rates above 50% are shaded.
| Dataset | Metric | Base JEPAACTS | Base TS-JEPA | Base SC-JEPA | + ACTS-WM (Pred. Only) JEPAACTS | + ACTS-WM (Pred. Only) TS-JEPA | + ACTS-WM (Pred. Only) SC-JEPA | + ACTS-WM (Obs.+Pred.) JEPAACTS | + ACTS-WM (Obs.+Pred.) TS-JEPA | + ACTS-WM (Obs.+Pred.) SC-JEPA |
| Univariate Time Series Classification | ||||||||||
| WebTraffic‡ | Acc. | 0.152 | 0.194 | 0.265 | 0.168★ | 0.158 | 0.215 | 0.158★ | 0.202★ | 0.213 |
| Earthquakes‡ | Acc. | 0.715 | 0.748 | 0.667 | 0.748★ | 0.748 | 0.736★ | 0.739★ | 0.748 | 0.719★ |
| UWaveGestureLibraryX‡ | Acc. | 0.633 | 0.635 | 0.588 | 0.663★ | 0.701★ | 0.563 | 0.653★ | 0.700★ | 0.577 |
| ShapesAll‡ | Acc. | 0.474 | 0.652 | 0.622 | 0.521★ | 0.655★ | 0.637★ | 0.506★ | 0.661★ | 0.631★ |
| WormsTwoClass‡ | Acc. | 0.541 | 0.545 | 0.532 | 0.554★ | 0.571★ | 0.623★ | 0.558★ | 0.545 | 0.636★ |
| Computers‡ | Acc. | 0.592 | 0.464 | 0.652 | 0.592 | 0.623★ | 0.692★ | 0.640★ | 0.500★ | 0.692★ |
| InsectWingbeatSound‡ | Acc. | 0.444 | 0.610 | 0.616 | 0.494★ | 0.493 | 0.522 | 0.478★ | 0.609 | 0.551 |
| EOGHorizontalSignal‡ | Acc. | 0.355 | 0.382 | 0.180 | 0.328 | 0.388★ | 0.162 | 0.322 | 0.400★ | 0.181★ |
| EthanolLevel‡ | Acc. | 0.360 | 0.298 | 0.457 | 0.246 | 0.270 | 0.379 | 0.396★ | 0.268 | 0.377 |
| SemgHandMovementCh2‡ | Acc. | 0.347 | 0.496 | 0.331 | 0.484★ | 0.559★ | 0.220 | 0.520★ | 0.516★ | 0.239 |
| Win rate vs. Base | Acc. | – | – | – | 70% | 60% | 40% | 90% | 60% | 50% |
| Multivariate Time Series Classification | ||||||||||
| ArticularyWordRecognition | Acc. | 0.847 | – | 0.948 | 0.897★ | – | 0.944 | 0.853★ | – | 0.949★ |
| BasicMotions | Acc. | 0.775 | – | 0.492 | 0.850★ | – | 0.792★ | 0.900★ | – | 0.758★ |
| CharacterTrajectories | Acc. | 0.957 | – | 0.912 | 0.934 | – | 0.902 | 0.961★ | – | 0.915★ |
| FingerMovements | Acc. | 0.463 | – | 0.500 | 0.480★ | – | 0.507★ | 0.463 | – | 0.503★ |
| SelfRegulationSCP2‡ | Acc. | 0.507 | – | 0.494 | 0.528★ | – | 0.511★ | 0.535★ | – | 0.489 |
| SpokenArabicDigits‡ | Acc. | 0.946 | – | 0.946 | 0.898 | – | 0.943 | 0.938 | – | 0.954★ |
| Win rate vs. Base | Acc. | – | – | – | 67% | – | 50% | 67% | – | 83% |
Table 2.
which is excluded from ACTS-WM pretraining, “Pred. Only” improves both MSE and MAE for all three JEPA-based encoders. Overall, “Pred. Only” generally outperforms “Obs.+Pred.” in forecasting, suggesting that the predicted target representations can improve time-series prediction.
For classification, combining observed and predicted representations generally yields stronger results. With our JEPA, “Obs.+Pred.” improves accuracy over Base on 9 out of 10 univariate classification datasets, all of which are excluded from ACTS-WM pretraining. For multivariate classification, JEPA with “Obs.+Pred.” achieves the best accuracy on 3 out of 6 datasets, while SC-JEPA with “Obs.+Pred.” improves upon its Base on 5 out of 6 datasets. These results suggest that predicted representations of ACTS-WM can complement observed representations for classification. Furthermore, the improvements on datasets excluded from ACTS-WM pretraining indicate the potential of the learned world model to generalize across datasets.
Ablation Study
We conduct ablation studies to evaluate the proposed designs in § 4.1.1 and § 4.1.2, and latent action selection in § 4.1.4.
Component Ablation. Table 3 evaluates the contributions of designs in § 4.1.1 and § 4.1.2. For the average performance, the full ACTS-WM achieves the best in three of the four task categories. Task embeddings are particularly important for multivariate forecasting, as removing them increases the average MSE from 0.472 to 0.788. Temporal fusion also contributes to multivariate tasks, with its removal increasing forecasting MSE from 0.472 to 0.550. In contrast, removing channel fusion has a small impact on multivariate forecasting and slightly improves univariate classification accuracy. Overall, the effectiveness of each component varies across different tasks and datasets.
| Removed component | ACTS-WM (JEPA, Obs.+Pred.) | |||||||
| Task embedding | ✓ ✓ ✓ ✓ | |||||||
| Channel fusion | ✓ ✓ ✓ ✓ | |||||||
| Temporal fusion | ✓ ✓ ✓ ✓ | |||||||
| Dataset | Full | (1) | (2) | (3) | (4) | (5) | (6) | (7) |
| Univariate Forecasting (MSE) | ||||||||
| Weather | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 |
| ETTh2 | 0.452 | 0.479 | 0.484 | 0.478 | 0.468 | 0.477 | 0.487 | 0.478 |
| ETTm1 | 0.050 | 0.048 | 0.051 | 0.049 | 0.052 | 0.051 | 0.050 | 0.054 |
| ETTm2 | 0.188 | 0.193 | 0.184 | 0.183 | 0.180 | 0.189 | 0.192 | 0.182 |
| Average | 0.174 | 0.181 | 0.181 | 0.179 | 0.176 | 0.181 | 0.184 | 0.180 |
| Multivariate Forecasting (MSE) | ||||||||
| ETTh1 | 0.905 | 1.103 | 0.901 | 0.987 | 0.938 | 1.102 | 1.101 | 1.098 |
| ETTh2 | 0.256 | 0.690 | 0.286 | 0.323 | 0.271 | 0.691 | 0.692 | 0.689 |
| ETTm1 | 0.576 | 0.787 | 0.564 | 0.689 | 0.599 | 0.789 | 0.800 | 0.799 |
| ETTm2 | 0.150 | 0.572 | 0.152 | 0.200 | 0.168 | 0.571 | 0.575 | 0.570 |
| Average | 0.472 | 0.788 | 0.476 | 0.550 | 0.494 | 0.788 | 0.792 | 0.789 |
| Univariate Classification (Acc.) | ||||||||
| Earthquakes | 0.737 | 0.723 | 0.741 | 0.745 | 0.665 | 0.741 | 0.741 | 0.719 |
| UWaveGestureLibraryX | 0.645 | 0.642 | 0.663 | 0.645 | 0.643 | 0.636 | 0.636 | 0.655 |
| ShapesAll | 0.525 | 0.515 | 0.518 | 0.498 | 0.521 | 0.526 | 0.492 | 0.486 |
| EOGHorizontalSignal | 0.360 | 0.365 | 0.370 | 0.376 | 0.394 | 0.359 | 0.369 | 0.345 |
| Average | 0.567 | 0.561 | 0.573 | 0.566 | 0.556 | 0.566 | 0.559 | 0.551 |
| Multivariate Classification (Acc.) | ||||||||
| ArticularyWordRecognition | 0.920 | 0.915 | 0.912 | 0.912 | 0.903 | 0.900 | 0.895 | 0.905 |
| BasicMotions | 0.788 | 0.725 | 0.712 | 0.662 | 0.663 | 0.713 | 0.738 | 0.712 |
| CharacterTrajectories | 0.920 | 0.932 | 0.930 | 0.918 | 0.930 | 0.917 | 0.920 | 0.936 |
| SpokenArabicDigits | 0.929 | 0.934 | 0.929 | 0.941 | 0.942 | 0.931 | 0.926 | 0.935 |
| Average | 0.889 | 0.877 | 0.871 | 0.858 | 0.859 | 0.865 | 0.870 | 0.872 |
Table 3. Ablation results of ACTS-WM components using JEPA with the Obs.+Pred. variant, averaged over two selective downstream heads per dataset. The best results are highlighted in bold.
Effect of Learned Latent Actions. To evaluate latent action selection (§ 4.1.4), we compare ACTS-WM using learned latent actions with randomly assigned actions in Figure 3. Replacing learned actions with random actions consistently degrades performance
Figure 3. Comparison of learned and randomly assigned latent actions in ACTS-WM using JEPA.

Figure 4. Parameter analysis for the codebook size and prediction horizons.
across all four task categories. For forecasting, “Pred. Only” achieves the lowest errors, while random actions increase both MSE and MAE. For classification, “Pred. Only” is sensitive to random actions, with substantial decreases in accuracy and F1-score. Nevertheless, using random actions generally degrades performance, often below the Base JEPA encoder, particularly in classification tasks. These findings suggest that learned latent actions contribute meaningful information and that combining observed and predicted representations can improve robustness.
Parameter Analysis
Effect of Codebook Size. We investigate the effect of codebook size on world model training over 100 epochs. In fig:4a, the training loss decreases rapidly during the first 10 epochs across all settings. However, differences emerge toward the end of training. Increasing the codebook size from 2 to 64 generally reduces the final training loss, with the lowest loss achieved at 64. Further increasing the size to 128 slightly increases the loss. These results suggest that an action space that is too small () may fail to capture sufficient transition information, while an excessively large action space () may also hinder representation prediction.
Prediction Horizon. fig:4b compares forecasting performance across prediction horizons of . The mean MSE increases with longer horizons across all JEPA-based encoders in both univariate and multivariate settings. This degradation is particularly pronounced for SC-JEPA, especially under the “Obs.+Pred.” setting. Overall, these results highlight the increasing difficulty of modeling longer-term time-series transitions.
Interpretation of Learned Latent Actions
To investigate what the learned latent actions capture, we visualize the continuous latent representations inferred by the IDM before quantization (Eq. 3) using UMAP projection [29], as shown. in Figure 5. We select representative forecasting and classification datasets with sufficient evaluation data samples.
Figure 5. (a) Forecasting datasets. Colors indicate quartiles of relative mean differences between observed and target time series, ordered from smallest to largest. (b) Classification datasets. Different colors represent different class labels within each dataset. UMAP projection of continuous latent actions inferred by the IDM before quantization. Each point represents a latent action inferred from an observation pair.
For forecasting (Figure 5(a)), the first (smallest difference) and fourth quartiles (largest difference) exhibit distinct distributions across several datasets, suggesting that the learned latent actions capture information related to the magnitude of changes between observed and future time series. For classification, Figure 5(b) reveals class-dependent structures in several datasets, although some classes exhibit overlapping distributions. These observations suggest that the IDM captures transition characteristics associated with different time-series classes.
Overall, the visualizations provide qualitative evidence that the learned latent actions encode meaningful information about time-series transitions across forecasting and classification tasks.
Conclusion and Future Work
In this work, we proposed ACTS-WM, a self-supervised latent action-conditioned world modeling framework for time-series analysis. ACTS-WM learns discrete latent actions used to condition a JEPA-based world model for predicting target representations. Experiments on 22 datasets covering univariate and multivariate forecasting and classification show improvements over pretrained JEPA-based encoders on many datasets. Further analyses highlight the benefits of learned latent actions and action conditioning. Future work will focus on improving the latent policy’s ability to predict latent actions.
References
Appendix
Dataset for Pretraining and Evaluation
All datasets are listed in Table 4.
Detailed Data Preprocessing and Settings
Preprocessing. For the datasets and variant lengths in the forecasting tasks, we impute the last value to the maximum length among all samples. Following [46], for univariate forecasting tasks, the target value is set as oil temperature for ETDatasets and co2 for the Weather dataset.
To prevent data leakage, we use only the training split during pretraining for datasets included in downstream evaluation. For other datasets, both training and test splits are used for pretraining. For forecasting tasks, datasets are divided into training, validation, and test sets at a ratio of 65:5:30. For classification tasks, we retain the original sequence lengths and predefined train/test splits, reserving 10% of the training set for validation.
Settings for JEPA-based Encoders
For pretraining ACTS-WM with our , we use 21 datasets, including 16 from the UEA archive [4], 4 ETT datasets, and the Weather dataset. The ETT and Weather datasets are widely used forecasting benchmarks available through the Autoformer repository [43].
Since TS-JEPA and SC-JEPA require fixed-length inputs, we pretrain ACTS-WM separately with each encoder. Specifically, we use 12 univariate datasets for TS-JEPA, which supports only univariate inputs, and 22 datasets for SC-JEPA.
For downstream evaluation, we use 22 datasets covering forecasting and classification. These include 9 UCR datasets [9] and WebTraffic [11] for univariate classification, 6 UEA datasets for multivariate classification, and 6 forecasting datasets: ETTh1, ETTh2, ETTm1, ETTm2, Weather, and WikiTraffic [17, 18].
Model Settings
Downstream Head Settings
We adapt the JEPA encoders and our ACTS-WM to downstream tasks using six different downstream heads. We use three classification heads: linear head, SVM head, and MLP head; and three forecasting heads: linear head, ridge regression head, and NLH head. For forecasting, linear head and NLH head follow the configurations described in CHARM [10]. The ridge head is a per-channel ridge regression, mapping the flattened tokens to the forecast horizon. It is solved in closed form, with the regularization strength selected based on validation performance.
Hyperparameter Settings
The key hyperparameters of ACTS-WM are configured as follows.
For latent action modeling, we set (§ 4.1.1), (§ 4.1.2), and (Eq. (5)). The VQ codebook (§ 4.1.4) contains code vectors of dimension , with code selections per action. For the JEPA-based encoder (§ 4.2.1), we set .
For the settings of JEPA-encoders, We train the encoder for 200 epochs, with the learning rate initially set to , linearly warmed up to over the first 5% of training epochs, then gradually reduced to with a cosine learning rate schedule. For other JEPA-based time series encoders, we pretrain TS-JEPA [12] and SC-JEPA [23] following the configurations specified in their respective papers.
Full Results and Additional Experiments
Performance over Downstream Heads
The results of each downstream head are reported in Table 5 and Table 6. For forecasting, the improvements brought by ACTS-WM are mainly observed with the linear head and ridge head. With these two heads, ACTS-WM outperforms base model on most
| Head | Model | WebT‡ | Earthq‡ | UWaveX‡ | Shapes‡ | Worms‡ | Comp‡ | Insect‡ | EOGH‡ | Ethanol‡ | Semg‡ | AWR | BM | CT | FM | SCP2‡ | SAD‡ | Uni. | Multi. |
| Linear | JEPA₍ACTS₎ Base | 0.168 | 0.715 | 0.633 | 0.506 | 0.541 | 0.528 | 0.523 | 0.355 | 0.362 | 0.495 | 0.929 | 0.750 | 0.932 | 0.457 | 0.520 | 0.946 | – | – |
| + ACTS-WM (Pred.) | 0.174★ | 0.748★ | 0.663★ | 0.538★ | 0.554★ | 0.547★ | 0.460 | 0.328 | 0.250 | 0.413 | 0.951★ | 0.642 | 0.783 | 0.427 | 0.506 | 0.898 | 60% | 17% | |
| + ACTS-WM (Obs.+Pred.) | 0.182★ | 0.739★ | 0.653★ | 0.511★ | 0.558★ | 0.492 | 0.505 | 0.322 | 0.324 | 0.498★ | 0.873 | 0.858★ | 0.875 | 0.483★ | 0.522★ | 0.938 | 60% | 50% | |
| TS-JEPA Base | 0.195 | 0.748 | 0.635 | 0.588 | 0.524 | 0.464 | 0.623 | 0.382 | 0.298 | 0.506 | – | – | – | – | – | – | – | – | |
| + ACTS-WM (Pred.) | 0.114 | 0.748 | 0.701★ | 0.537 | 0.532★ | 0.623★ | 0.495 | 0.388★ | 0.270 | 0.388 | – | – | – | – | – | – | 40% | – | |
| + ACTS-WM (Obs.+Pred.) | 0.200★ | 0.746 | 0.700★ | 0.560 | 0.511 | 0.500★ | 0.591 | 0.400★ | 0.268 | 0.512★ | – | – | – | – | – | – | 50% | – | |
| SC-JEPA Base | 0.251 | 0.667 | 0.588 | 0.652 | 0.494 | 0.643 | 0.616 | 0.180 | 0.457 | 0.331 | 0.948 | 0.467 | 0.912 | 0.500 | 0.500 | 0.946 | – | – | |
| + ACTS-WM (Pred.) | 0.181 | 0.736★ | 0.563 | 0.665★ | 0.602★ | 0.673★ | 0.522 | 0.162 | 0.379 | 0.220 | 0.944 | 0.725★ | 0.902 | 0.507★ | 0.480 | 0.943 | 40% | 33% | |
| + ACTS-WM (Obs.+Pred.) | 0.196 | 0.719★ | 0.577 | 0.653★ | 0.571★ | 0.668★ | 0.551 | 0.181★ | 0.377 | 0.239 | 0.949★ | 0.725★ | 0.915★ | 0.503★ | 0.504★ | 0.954★ | 50% | 100% | |
| MLP | JEPA₍ACTS₎ Base | 0.155 | 0.743 | 0.646 | 0.474 | 0.550 | 0.553 | 0.444 | 0.398 | 0.431 | 0.540 | 0.958 | 0.775 | 0.957 | 0.463 | 0.507 | 0.950 | – | – |
| + ACTS-WM (Pred.) | 0.151 | 0.748★ | 0.665★ | 0.521★ | 0.524 | 0.588★ | 0.494★ | 0.332 | 0.257 | 0.551★ | 0.962★ | 0.658 | 0.934 | 0.480★ | 0.528★ | 0.562 | 60% | 50% | |
| + ACTS-WM (Obs.+Pred.) | 0.173★ | 0.753★ | 0.663★ | 0.506★ | 0.554★ | 0.545 | 0.478★ | 0.379 | 0.435★ | 0.514 | 0.960★ | 0.858★ | 0.961★ | 0.463 | 0.535★ | 0.948 | 70% | 67% | |
| TS-JEPA Base | 0.229 | 0.748 | 0.697 | 0.652 | 0.550 | 0.508 | 0.610 | 0.420 | 0.576 | 0.496 | – | – | – | – | – | – | – | – | |
| + ACTS-WM (Pred.) | 0.191 | 0.748 | 0.711★ | 0.655★ | 0.519 | 0.623★ | 0.493 | 0.437★ | 0.259 | 0.559★ | – | – | – | – | – | – | 50% | – | |
| + ACTS-WM (Obs.+Pred.) | 0.226 | 0.746 | 0.718★ | 0.661★ | 0.576★ | 0.523★ | 0.609 | 0.424★ | 0.525 | 0.516★ | – | – | – | – | – | – | 60% | – | |
| SC-JEPA Base | 0.265 | 0.707 | 0.591 | 0.622 | 0.498 | 0.660 | 0.599 | 0.159 | 0.447 | 0.326 | 0.937 | 0.492 | 0.942 | 0.507 | 0.487 | 0.967 | – | – | |
| + ACTS-WM (Pred.) | 0.215 | 0.700 | 0.539 | 0.637★ | 0.563★ | 0.680★ | 0.493 | 0.140 | 0.363 | 0.197 | 0.926 | 0.792★ | 0.857 | 0.453 | 0.485 | 0.947 | 30% | 17% | |
| + ACTS-WM (Obs.+Pred.) | 0.213 | 0.676 | 0.547 | 0.631★ | 0.550★ | 0.673★ | 0.470 | 0.148 | 0.372 | 0.206 | 0.928 | 0.758★ | 0.883 | 0.483 | 0.489★ | 0.954 | 30% | 33% | |
| SVM | JEPA₍ACTS₎ Base | 0.152 | 0.748 | 0.651 | 0.453 | 0.597 | 0.592 | 0.508 | 0.354 | 0.360 | 0.347 | 0.847 | 0.775 | 0.912 | 0.490 | 0.522 | 0.930 | – | – |
| + ACTS-WM (Pred.) | 0.168★ | 0.748 | 0.668★ | 0.320 | 0.584 | 0.592 | 0.352 | 0.268 | 0.246 | 0.484★ | 0.897★ | 0.850★ | 0.751 | 0.490 | 0.489 | 0.876 | 30% | 33% | |
| + ACTS-WM (Obs.+Pred.) | 0.158★ | 0.748 | 0.668★ | 0.458★ | 0.558 | 0.640★ | 0.490 | 0.348 | 0.396★ | 0.520★ | 0.853★ | 0.900★ | 0.883 | 0.490 | 0.483 | 0.919 | 60% | 33% | |
| TS-JEPA Base | 0.194 | 0.748 | 0.720 | 0.522 | 0.545 | 0.596 | 0.627 | 0.475 | 0.308 | 0.473 | – | – | – | – | – | – | – | – | |
| + ACTS-WM (Pred.) | 0.158 | 0.748 | 0.715 | 0.455 | 0.571★ | 0.636★ | 0.483 | 0.420 | 0.256 | 0.427 | – | – | – | – | – | – | 20% | – | |
| + ACTS-WM (Obs.+Pred.) | 0.202★ | 0.748 | 0.727★ | 0.503 | 0.545 | 0.592 | 0.601 | 0.464 | 0.276 | 0.567★ | – | – | – | – | – | – | 30% | – | |
| SC-JEPA Base | 0.268 | 0.748 | 0.638 | 0.560 | 0.532 | 0.652 | 0.609 | 0.199 | 0.480 | 0.333 | 0.953 | 0.700 | 0.956 | 0.520 | 0.494 | 0.979 | – | – | |
| + ACTS-WM (Pred.) | 0.192 | 0.748 | 0.557 | 0.490 | 0.623★ | 0.692★ | 0.484 | 0.177 | 0.354 | 0.207 | 0.953 | 0.800★ | 0.910 | 0.460 | 0.511★ | 0.959 | 20% | 33% | |
| + ACTS-WM (Obs.+Pred.) | 0.192 | 0.748 | 0.558 | 0.490 | 0.636★ | 0.692★ | 0.489 | 0.171 | 0.356 | 0.209 | 0.953 | 0.800★ | 0.914 | 0.480 | 0.489 | 0.960 | 20% | 17% |
Table 5. Time-series classification results of each head averaged across three runs. Datasets marked with ‡ are not included in pre-training ACTS-WM. The best result is in bold, and ★ marks results with ACTS-WM that outperforms original base models. The win rate is the percentage of datasets where ACTS-WM outperforms the base model. Win rates above 50% are shaded.
| Head | Model | Wiki‡ | Weather | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Uni. | Multi. |
| Linear | JEPA₍ACTS₎ Base | 1.167 | 0.007 | 0.100 | 0.478 | 0.044 | 0.184 | 0.257 | 1.056 | 0.302 | 0.597 | 0.161 | – | – |
| + ACTS-WM (Pred.) | 1.063★ | 0.003★ | 0.084★ | 0.351★ | 0.035★ | 0.138★ | 0.238★ | 0.822★ | 0.264★ | 0.489★ | 0.152★ | 100% | 100% | |
| + ACTS-WM (Obs.+Pred.) | 1.164★ | 0.007 | 0.086★ | 0.442★ | 0.043★ | 0.175★ | 0.259 | 1.017★ | 0.290★ | 0.564★ | 0.156★ | 83% | 80% | |
| TS-JEPA Base | 1.047 | 0.002 | 0.085 | 0.402 | 0.036 | 0.153 | – | – | – | – | – | – | – | |
| + ACTS-WM (Pred.) | 0.972★ | 0.001★ | 0.097 | 0.351★ | 0.064 | 0.426 | – | – | – | – | – | 50% | – | |
| + ACTS-WM (Obs.+Pred.) | 1.040★ | 0.002 | 0.088 | 0.396★ | 0.045★ | 0.136★ | – | – | – | – | – | 50% | – | |
| SC-JEPA Base | 1.225 | 0.003 | 0.740 | 1.977 | 1.312 | 1.602 | 0.590 | 1.207 | 2.164 | 1.163 | 0.914 | – | – | |
| + ACTS-WM (Pred.) | 1.288 | 0.003 | 0.428★ | 1.651 | 0.793★ | 1.673 | 0.471★ | 1.279 | 2.213 | 1.255 | 0.909★ | 33% | 40% | |
| + ACTS-WM (Obs.+Pred.) | 1.240 | 0.003 | 1.089 | 6.716 | 0.775★ | 1.073★ | 0.514★ | 1.217 | 2.261 | 1.142★ | 0.817★ | 33% | 60% | |
| Ridge | JEPA₍ACTS₎ Base | 1.141 | 0.005 | 0.095 | 0.467 | 0.049 | 0.178 | 0.255 | 1.218 | 0.366 | 0.605 | 0.160 | – | – |
| + ACTS-WM (Pred.) | 1.092★ | 0.005 | 0.094★ | 0.345★ | 0.045★ | 0.139★ | 1.902 | 0.678★ | 0.276★ | 0.495★ | 0.153★ | 83% | 80% | |
| + ACTS-WM (Obs.+Pred.) | 1.141 | 0.005 | 0.093★ | 0.468 | 0.049 | 0.171★ | 0.253★ | 1.196 | 0.297★ | 0.583★ | 0.156★ | 33% | 100% | |
| TS-JEPA Base | 0.976 | 0.005 | 0.097 | 0.319 | 0.045 | 0.197 | – | – | – | – | – | – | – | |
| + ACTS-WM (Pred.) | 0.977 | 0.005 | 0.157 | 0.412 | 0.045 | 0.184★ | – | – | – | – | – | 17% | – | |
| + ACTS-WM (Obs.+Pred.) | 0.968★ | 0.005 | 0.091★ | 0.322 | 0.054 | 0.133★ | – | – | – | – | – | 50% | – | |
| SC-JEPA Base | 1.239 | 0.006 | 0.601 | 1.024 | 1.448 | 1.543 | 0.558 | 1.194 | 1.919 | 1.217 | 0.915 | – | – | |
| + ACTS-WM (Pred.) | 1.270 | 0.006 | 0.946 | 1.448 | 0.964★ | 1.651 | 0.483★ | 1.201 | 2.050 | 1.269 | 0.981 | 17% | 20% | |
| + ACTS-WM (Obs.+Pred.) | 1.272 | 0.006 | 0.946 | 1.448 | 0.915★ | 1.614 | 0.491★ | 1.201 | 2.051 | 1.268 | 0.919 | 17% | 20% | |
| HIN | JEPA₍ACTS₎ Base | 1.268 | 0.016 | 0.104 | 0.278 | 0.077 | 0.186 | 0.278 | 0.573 | 0.233 | 0.542 | 0.162 | – | – |
| + ACTS-WM (Pred.) | 1.393 | 0.042 | 0.671 | 0.712 | 0.638 | 0.665 | 0.554 | 1.084 | 1.136 | 1.018 | 1.096 | 0% | 0% | |
| + ACTS-WM (Obs.+Pred.) | 1.496 | 0.088 | 0.499 | 0.559 | 0.485 | 0.506 | 0.474 | 0.888 | 0.809 | 0.861 | 0.784 | 0% | 0% | |
| TS-JEPA Base | 1.151 | 0.543 | 0.215 | 0.674 | 0.144 | 0.397 | – | – | – | – | – | – | – | |
| + ACTS-WM (Pred.) | 1.372 | 0.099★ | 0.822 | 0.815 | 0.372 | 0.528 | – | – | – | – | – | 17% | – | |
| + ACTS-WM (Obs.+Pred.) | 1.976 | 6.494 | 0.519 | 1.407 | 1.273 | 2.899 | – | – | – | – | – | 0% | – | |
| SC-JEPA Base | 1.413 | 0.148 | 1.631 | 1.478 | 1.822 | 1.625 | 0.848 | 1.321 | 1.890 | 1.434 | 1.913 | – | – | |
| + ACTS-WM (Pred.) | 1.363★ | 0.018★ | 0.728★ | 1.358★ | 1.588★ | 1.354★ | 0.641★ | 1.129★ | 1.395★ | 1.105★ | 1.596★ | 100% | 100% | |
| + ACTS-WM (Obs.+Pred.) | 1.906 | 0.473 | 1.739 | 2.140 | 1.493★ | 4.001 | 1.009 | 1.209★ | 2.084 | 1.599 | 1.740★ | 17% | 40% |
Table 6. Time-series forecasting MSE () of each head averaged across three runs. Datasets marked with ‡ are not included in pre-training ACTS-WM. The best result is in bold, and ★ marks results with ACTS-WM that outperforms original base models. The win rate is the percentage of datasets where ACTS-WM outperforms the base model. Win rates above 50% are shaded.
datasets. Remarkably, the linear head outperforms the base model on all datasets when is integrated with ACTS-WM in the predictor-only setting. Although ACTS-WM does not yield substantial improvements with NLH head for , the NLH head demonstrates its effectiveness with SC-JEPA, which outperforms the base in all datasets.
For classification, outperforms the base model on most datasets in univariate classification. For multivariate classification, SC-JEPA with ACTS-WM outperforms the base model on all datasets when both observation and predictor embeddings are fed into the linear head.
Case study of latent action semantics
In Figure 6, we plot the time-series pairs associated with different action codes.
Figure 6. Case study of latent actions on univariate forecasting with the backbone. Contexts that are assigned the same action code lead to targets with the same kind of change (horizon 48).
For the action code with index (5, 6), the sequences of selected pairs exhibit patterns similar to those in the middle sections of their corresponding sequences. For the action code with index (14, 11), the sequences of all three pairs exhibit an upward trend regardless of whether the tail sections of exhibit an upward trend, a downward trend or a steady state.
Based on these observations, we speculate that the actions may capture some meaningful patterns that can be interpreted in terms of human-understandable concepts.
References
- [1]Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. 2026. Cosmos 3: Unimodal world models for physical ai. arXiv preprint arXiv:2606.02800 (2026).arxiv.org/abs/2606.02800
- [2]Eloi Alonso, Adam Jelley, Vincent Michiel, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. 2024. Diffusion for World Modeling: Visual Details Matter in Atari. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 58757–58791. doi:10.52202/079017-1873DOI
- [3]Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Koukoulas, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. 2025. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985 (2025).arxiv.org/abs/2506.09985
- [4]Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. 2018. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 (2018).
- [5]Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018).arxiv.org/abs/1803.01271
- [6]Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Maria Elisabeth Bechtle, Feryal Behbahani, Stephanie C.Y. Chan, Nicolas Haess, Lucy Gonzalez, Simon Osindero, Sherij Olzair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. 2024. Genie: Generative Interactive Environments. In Forty-first International Conference on Machine Learning. https://openreview.net/forum?id=bJs bSjkOSarxiv.org/abs/2402.15391
- [7]Ben Coh, Emaad Khwaja, Kan Wang, Charles Masson, Elise Ramé, Youssef Doubli, and Othmane Abou-Amal. 2024. Toto: Time Series Optimized Transformer for Observability. https://openreview.net/forum?id=ayupWYa1qD
- [8]Xilin Dai, Hongzhou Chen, Yifan Hu, Yiding Liu, Zewei Dong, and Jiang-Ming Yang. 2026. Pythia: Toward Foundation World Models for Multimodal Time Series. arXiv preprint arXiv:2610.05240 (2026).arxiv.org/abs/2610.05240
- [9]Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. 2019. The UCR time series archive. IEEE/CAA Journal of Automatica Sinica 6, 6 (2019), 1293–1305. doi:10.1109/JAS.2019.1911747DOI
- [10]Utsav Dutta, Gerardo Pastrana, Sina Khoshcheft Pakazad, and Henrik Ohlsson. 2026. Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=INoaqrOXtiarxiv.org/abs/2605.31580
- [11]Joseph Early, Gavin Cheung, Kurt Cutajar, Hanting Xie, Jas Kandola, and Niall Twomey. 2024. Inherently Interpretable Time Series Classification via Multiple Instance Learning. In International Conference on Learning Representations, Vol. 2024. 56800–56828.arxiv.org/abs/2311.10049
- [12]Sofiane ENNADIR, Siavash Golkar, and Leopoldo Sarra. 2024. Joint Embedding to Temporal. In NeurIPS Workshop on Time Series in the Age of Large Models. https://openreview.net/forum?id=Fldbozemby
- [13]Archibald Fraikin, Adrien Bennetot, and Stéphanie Allassonnière. 2024. T-rep: Representation learning for time series using time-embeddings. In International Conference on Learning Representations, Vol. 2024. 31285–31305.arxiv.org/abs/2310.04486
- [14]Yisong Gu, Fei Wang, Zhezhi Shao, Boyu Diao, Lin Wu, Zhulin An, Chengqing Yu, Yujie Li, and Yongjun Xu. 2025. On the Integration of Spatial-Temporal Knowledge: A Lightweight Approach to Atmospheric Time Series Forecasting. In the Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=thHKpIt8q
- [15]Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. 2024. UniTS: A Unified Multi-Task Time Series Model. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 140589–140631. doi:10.52202/079017-4463DOI
- [16]Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. 2025. Ada-World: Learning Adaptable World Models with Latent Actions. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=QQEgZj9Jskarxiv.org/abs/2503.18938
- [17]Rakshitha Godahewa, Christoph Bergmeir, Geoff Webb, Rob Hyndman, and Pablo Montero-Manso. 2022. Extended Wikipedia Web Traffic Daily Dataset (without Missing Values). doi:10.5281/zenodo.7371038DOI
- [18]Rakshitha Wathsadini Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob Hyndman, and Pablo Montero-Manso. 2021. Monash Time Series Forecasting Archive. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=wEc1mgAjU-arxiv.org/abs/2105.06643
- [19]Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. 2024. MOMENT: a family of open time-series foundation models. In Proceedings of the 41st International Conference on Machine Learning. JMLR.org, Article 642, 38 pages.arxiv.org/abs/2402.03885
- [20]Robert Gray. 1984. Vector quantization. IEEE Assp Magazine 1, 2 (1984), 4–29.
- [21]Yanjiang Guo, Lucy Shi, Jianyu Chen, and Chelsea Finn. 2026. Ctrl-world: A controllable generative world model for robot manipulation. In International Conference on Learning Representations. Vol. 2026. 6121–6138.arxiv.org/abs/2510.10125
- [22]Danijar Hafner, Wilson Yan, and Timothy Lillicrap. 2025. Training Agents Inside of Scalable World Models. arXiv:2509.24527 [cs.AI]. https://arxiv.org/abs/2509.24527
- [23]Yanan He, Yunshi Wen, Xin Wang, and Tengfei Ma. 2026. SC-JEPA: Stabilizing Latent Predictive Learning for Time-Series Anomaly Prediction. In Forecasting as a New Frontier of Intelligence. https://openreview.net/forum?id=SNjhsnUvHHarxiv.org/abs/2602.04643
- [24]Yann LeCun et al. 2022. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review 62, 1 (2022), 1–62.
- [25]Yang Lin, Irena Koprinska, and Mashud Rana. 2021. Temporal convolutional attention neural networks for time series forecasting. In 2021 International joint conference on neural networks (IJCNN). IEEE, 1–8.
- [26]Chenghao Liu, Taha Aksu, Juncheng Liu, Xu Liu, Hanshu Yan, Quang Pham, Silvio Savarese, Doyen Sahoo, Caiming Xiong, and Junnan Li. 2025. Moirai 2.0: When less is more for time series forecasting. arXiv preprint arXiv:2511.11698 (2025).arxiv.org/abs/2511.11698
- [27]Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann, Chenghao Liu, Junnan Li, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. 2025. Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=SrEOUSyJcRarxiv.org/abs/2410.10469
- [28]Ning Liu, Haijiang Lin, Shengcai Liu, Haozhe Lv, Yanbin Wei, Lingting Zhu, Shengju Qian, Xin Wang, Ying-Cong Chen, et al. 2026. Policy and World Modeling Co-Training for Language Agents. arXiv preprint arXiv:2606.02388 (2026).arxiv.org/abs/2606.02388
- [29]Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018).
- [30]Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=Jbdc0vTOcolarxiv.org/abs/2211.14730
- [31]Joaquim Ortiz-Haro, Joseph Amigo, Omar Mostafa, Daniel Dugas, Franziska Meier, and Ludovic Righetti. 2026. Worldplanner: Monte carlo tree search and mpc with action-conditioned visual world models. In 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 22166–22173.
- [32]Jonas Petersen, Gian-Alessandro Lombardi, Riccardo Maggioni, Camilla Mazzoleni, Federico Martelli, and Philipp Christian Petersen. 2026. HEPA: A Self-Supervised Horizon-Conditioned Event Predictive Architecture for Time Series. In 2nd ICML Workshop on Foundation Models for Structured Data. https: /openreview.net/forum?id=TdqpVBB8naarxiv.org/abs/2605.11130
- [33]Xiaoqian Qi, Haoye Chai, Yue Wang, and Yong Li. 2026. Beyond Static Forecasting: Unleashing the Power of World Models for Mobile Traffic Extrapolation. arXiv:2604.08199 [cs.NI] https://arxiv.org/abs/2604.08199
- [34]Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing, Mao Yang, Jie Tong, and Qi Zhang. 2019. Time-Series Anomaly Detection Service at Microsoft. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD ’19). Association for Computing Machinery, New York, NY, USA, 3009–3017. doi:10.1145/3292500.3330680DOI
- [35]Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.arxiv.org/abs/1505.04597
- [36]Dominick Schmidt and Minqi Jiang. 2024. Learning to act without actions. In International Conference on Learning Representations, Vol. 2024. 9379–9395.
- [37]Vaisakh Shaj, Saleh GHOLAM ZADEH, Ozan Demir, Luiz Ricardo Douat, and Gerhard Neumann. 2023. Multi Time Scale World Models. In Thirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/forum?id=fY7dShtbmoarxiv.org/abs/2310.18534
- [38]Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. 2025. Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=e1wDDfMfVu
- [39]Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer. 2021. Emformer: Efficient Memory Transformer Based Acoustic Model for Low Latency Streaming Speech Recognition. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 6783–6787. doi:10.1109/ICASSP39728.2021.9414560DOI
- [40]Stijn Verdenius, Andrea Ziro, and Roy L. M. Wang. 2024. Lat-PFN: A Joint Embedding Predictive Architecture for In-context Time-series Forecasting. arXiv:2405.10093 [cs.LG] https://arxiv.org/abs/2405.10093
- [41]Yucen Wang, Fengming Zhang, De-Chuan Zhan, Li Zhao, Kaixin Wang, and Jiang Bian. 2026. Co-Evolving Latent Action World Models. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=iec4FIZyrx
- [42]Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. Unified Training of Universal Time Series Forecasting Transformers. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 53140–53164.arxiv.org/abs/2402.02592
- [43]Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 22419–22430.
- [44]Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. 2023. DayDreamer: World models for physical robot learning. In Conference on robot learning. PMLR, 2226–2240.
- [45]Seonghyeon Ye, Joel Jang, Byeonguk Jeon, Se June Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. 2025. Latent action pretraining from videos. In International Conference on Learning Representations, Vol. 2025. 28213–28239.arxiv.org/abs/2410.11758
- [46]Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. 2022. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI conference on artificial intelligence, Vol. 36. 8980–8987.arxiv.org/abs/2106.10466
- [47]Jianqi Zhang, Jingyao Wang, Wenwen Qiang, Fanjiang Xu, and Changwen Zheng. 2026. Enhancing Large Language Models for Time-Series Forecasting via Vector-Injected In-Context Learning. In Proceedings of the ACM Web Conference 2026 (Dubai, United Arab Emirates) (WWW ’26). Association for Computing Machinery, New York, NY, USA, 7211–7222. doi:10.1145/3774904.3792374DOI
- [48]Wanceng Zhang, Valerie Twer, Michael Rabbat, Yann LeCun, and Randall Balestriero. 2026. H-JEPA: End-to-end learning of hierarchical world models for visual planning. arXiv preprint arXiv:2610.06805 (2026).
- [49]Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 1106–1115.arxiv.org/abs/2012.07436




