WorksComputer Science
Benchmarking System One Models for Time-Series Decision Making: A Case Study of Jev
Abstract
Time-series tasks require both accurate predictions and fast decisions. Advances in large models have introduced decision-oriented models that directly output answers and their probabilities, which we call Structured System 1 models. Their applicability to time-series tasks remains underexplored. Using Jev 1.13.0 as a case study, we evaluate time-series classification, event type prediction, oil temperature direction and stock movement prediction, converting continuous forecasting targets into discrete decisions to fit the model interface. We establish a unified evaluation framework using accuracy and AUC. Results show stronger performance on selected explicit temporal patterns and event type prediction, but near-chance discrimination on many numerical sequence classification tasks and stock movement prediction. Some datasets also reveal a gap between ranking quality and final decision accuracy. These findings motivate improvements in temporal representation, domain adaptation, and decision-score alignment. Standardized sample construction, model calls, and result recording support extending the evaluation to future models.
Introduction
Time series record how a system changes over time and support applications across diverse domains: wearable sensors enable human activity recognition [19], physiological measurements support patient assessment and deterioration prediction [12], operational signals enable equipment fault detection and diagnosis [16], and financial observations support market movement prediction [18]. These applications require time-series analysis to balance accuracy and processing speed. Traditional machine learning methods, including support vector machines, random forests, and XGBoost [7, 5, 6], are typically trained using task-specific features, and their predictive performance depends on feature representations and training data.
Recent advances in large language models and time-series foundation models have introduced new approaches to time-series analysis. Time-LLM maps temporal patches into the representation space of a frozen language model, Chronos quantizes time-series values for pretraining across domains, and Chronos-2 further supports zero-shot multivariate and covariate-informed forecasting [14, 2, 3]. Some studies use natural-language instructions to guide models in making time-series predictions [17].
Although complex temporal dependencies may require stronger modeling and reasoning capabilities, many time-series tasks can be naturally formulated as direct decision problems rather than continuous-valued forecasting. A unified, direct, and structured decision-making approach also offers potential advantages in computational efficiency and reduced generation overhead. These considerations motivate evaluating whether Structured System 1 can effectively interpret time-series information and produce reliable decisions. Inspired by dual-process theories in psychology, related studies use
Preprint. System 1 to describe fast, direct judgments and System 2 to describe deliberative reasoning involving multiple steps [11, 25], as shown in Figure 1(a).
Figure 1. (a) Schematic of System 1 and System 2, illustrating intuitive judgments and deliberative reasoning in dual-process theory. (b) Model taxonomy under our operational definitions: System 1 provides direct answers through structured outputs or text generation, while System 2 uses intermediate reasoning through test-time enhancement or built-in reasoning.
Existing studies differ in how they distinguish System 1 from System 2. Distilling System 2 into System 1 categorizes Chain-of-Thought (CoT) and other methods that generate intermediate reasoning steps as System 2 [25]. The time-series forecasting benchmark ReC4TS instead treats CoT, self-consistency, and self-correction as “System 1 reasoning enhancements,” while categorizing models that undergo dedicated reasoning post-training as System 2 [17].
This paper adopts an operational distinction based on the inference procedure. As shown in Figure 1(b), System 1 produces final answers without generating an intermediate reasoning sequence and is divided into Structured System 1 and Generative System 1 according to the output mechanism. The former directly computes decisions, while the latter generates answers through text decoding, with implications for response speed and output stability. System 2 uses additional inference-time computation for intermediate derivation, search, or reflection through Built-in Reasoning or Test-Time Enhanced approaches, which can be combined. Existing time-series research has focused primarily on System 2 and Generative System 1, with fewer systematic evaluations of Structured System 1.
Structured System 1 models score predefined alternatives, labels, or ordered levels conditioned on a state, a natural-language question, and candidate answers, returning decisions and probabilities through a unified interface. This output format supports time-series classification, anomaly detection, and discretized forecasting. A typical implementation uses an encoder to obtain contextual representations and a classification or scoring head to map them into the target output space, reducing sequential decoding and answer-parsing overhead. BERT combines a pretrained encoder with task-specific output layers for classification and natural language inference, while DistilBERT explores smaller and faster language representation models, providing a foundation for this approach [10, 20].
Jev is a System One model developed by TypeSafe AI, characterized by low latency and low API cost [21]. Independent research has evaluated its decision quality and invocation cost on text classification, natural language inference, reading comprehension, and content moderation [9], while its performance on time-series tasks awaits systematic evaluation. We use Jev as a case study, organizing temporal observations into a unified state-and-question representation and evaluating decision quality using accuracy (ACC) and the area under the receiver operating characteristic curve (AUC) to assess the applicability of Structured System 1 across time-series tasks.
Our contributions are threefold:
We define the task scope and decision mappings for time-series classification, anomaly detection, and adapted forecasting, mapping continuous future values to explicit decision targets.
We evaluate Jev 1.13.0 through binary and multiclass decision interfaces, reporting ACC and AUC and analyzing performance across datasets and task types.
We establish an extensible evaluation framework that standardizes sample construction, model calls, and result recording, enabling subsequent Structured System 1 models to be evaluated under the same task definitions and evaluation protocols.
Problem formulation
Time-series inputs and decision outputs
Following established time-series definitions [13, 15], let
where is the number of ordered observations, is the number of channels, and contains the values at time . A univariate series is the special case . For a decision at time , an observed window of length is
The right endpoint is and values after cannot be used in the input to a future-oriented decision.
Some observations are events at irregular times. We write their history as
where is the event time, is its categorical mark, and is the number of observed events by time . This representation retains the event times instead of imposing a regular sampling interval. We use for the task-specific input, whether a complete series, a window, or an event history.
Let be the state supplied to the model, with denoting a fixed representation procedure. Given a question and a candidate set , a decision model returns
with nonnegative components. Here denotes model parameters and is the number of admissible decisions. The predicted answer is . The three tasks below differ in what is observed, what the label denotes, and when it becomes known.
Time-series classification
For sample , let be an observed series of length and let be its class. The goal is to estimate
The index identifies a sample, its length, and the number of channels within its dataset. A segment cut from a continuous activity record can itself be a classification sample. For the task is binary, for it is multiclass. The label describes the observed segment rather than an unobserved future.
Time-series anomaly detection
Let indicate whether time is anomalous, with 1 denoting an anomaly. Using the historical window ending at , define the anomaly score
where is a prespecified decision threshold and is the indicator function. If anomalies are annotated as intervals , the corresponding point label is . The window supplies evidence, whereas its right endpoint is the object being labeled. This distinction prevents the label “the window contains an anomaly” from silently replacing the label “time is anomalous.” The continuous score can also be used to compute threshold-independent ranking measures.
Decision-oriented time-series forecasting
Conventional forecasting estimates future continuous observations. Let be the forecast horizon and the number of forecast target channels. With , the future target is
where denotes a future step and is a numeric forecasting model. For example, squared-error training minimizes a sum of over training examples, with denoting the Frobenius norm.
Our reformulation fixes a label map before evaluating the model:
Here collects task parameters, such as a horizon or threshold. The future values generate the ground-truth label , but are never included in the input state. The model directly answers a question about an event in the future.
For a concrete example, let be a stock’s closing price at time , and define the -step return, for , by
The binary upward-movement target is
The equivalence follows because dividing by the positive does not change the sign of . The requested quantity is thus , rather than a point estimate of . A three-class version, with a fixed tolerance , uses
The threshold determines the width of the flat class and must remain fixed across systems being compared. Stock movement prediction has previously been posed as a directional task [23], the example here illustrates the general conversion of a continuous forecast target into an explicit decision event.
The reformulation changes the statistical object being estimated. For a scalar future price and a squared-error point forecast , differentiating its conditional loss gives
In contrast, the binary decision depends on . A conditional mean and a probability mass above a threshold summarize different aspects of the future distribution. The same historical window can therefore lead to different optimal numerical and decision outputs.
Mapping tasks to typed answers
Equation (4) gives a common output form. Sequence classification uses the class of the observed sample, anomaly detection uses the status of a specified endpoint, and decision-oriented forecasting uses a future-derived class. Binary questions are evaluated through Jev’s Noul primitive, which returns the probability that a statement is true. Choice selects from a fixed candidate set and returns option probabilities, supporting both binary and multiclass decisions. The label construction and the evaluation unit remain task-specific even though the external decision format is shared.
Method
Jev is a commercial model whose full internal architecture is not publicly documented [21]. Figure 2 illustrates one possible architecture for Structured System 1 decision making using a BERT-based classifier [10]. We use this example to explain the main computational stages and evaluate Jev through its public decision interface.
Figure 2. One possible architecture for a Structured System 1 model, illustrated using BERT. Embedding, encoding, and classification map the input to class probabilities and a predicted class.
Input Embedding
Given an observed time series , a representation procedure constructs the input state . The state records observations chronologically and can include channel names and timestamps. Together with the task question , it is serialized into a token sequence:
Here, [CLS] is the classification token and [SEP] separates the state and question. Numerical observations are represented through the tokens produced by the tokenizer.
BERT combines token, position, and segment embeddings to construct the initial representation of token :
where identifies whether the token belongs to the state or question. For tokens and hidden dimension , these vectors form . Token embeddings represent the input content, position embeddings distinguish token locations, and segment embeddings distinguish the two input components [10].
Contextual Encoder
The embedding sequence passes through a stack of bidirectional Transformer encoder layers:
where is the number of encoder layers. Each layer uses multi-head self-attention and a feedforward network, together with residual connections and layer normalization [22].
For one attention head, the input representations are projected into queries, keys, and values:
The attention output is
where is the key dimension. This operation allows each token to incorporate information from other tokens in the observed input. In the time-series setting, it provides a mechanism for relating observations at different positions and conditioning their interpretation on the task question.
The final classification-token representation,
provides a contextual representation for the decision head. Bidirectional encoding operates over the supplied observations, which contain only the history available for the task.
Classification and Decision Probabilities
For a fixed answer set , a linear classification head maps the contextual representation to class logits:
where and are learned parameters. A softmax converts these logits into a probability distribution:
For binary decisions, a single logit can instead be converted into the positive-class probability:
The fixed-label head illustrates standard BERT classification. To accommodate changing candidate sets, an alternative is to encode each candidate together with the state and question, apply a shared scoring head, and normalize the resulting scores across candidates. This makes the output depend explicitly on the supplied alternatives.
In both cases, the encoder representations are mapped directly to decision probabilities without generating an answer token by token. The predicted class supports ACC evaluation, while the probabilities support AUC evaluation. In our Jev experiments, these outputs are obtained through Noul for binary questions and Choice for multiclass questions.
Experiments
Datasets And Evaluation Metrics
This study selects datasets suitable for evaluating System 1 models, covering time series classification, event type prediction, time series anomaly detection, and forecasting. These datasets include univariate and multivariate numerical sequences, as well as irregular event sequences consisting of timestamps and event types. We evaluate Jev on these datasets using accuracy (ACC) and the area under the receiver operating characteristic curve (AUC).
(1) Time series classification: UCR and UEA archives. The UCR Archive is a representative collection of datasets for univariate time series classification. Its 2018 release contains 128 datasets, covering sensor recordings, physiological signals such as electrocardiograms, motion data, and sequences derived from image outlines [8]. Each sample consists of a univariate sequence and its class label. The number of samples, sequence lengths, and numbers of classes vary substantially across datasets. The task is therefore univariate time series classification: the model reads an entire sequence and predicts its class, under either a binary or multiclass setting. This study uses the archive to examine Jev’s ability to recognize temporal patterns in a single channel.
The UEA Archive focuses on multivariate time series classification. Its 2018 release contains 30 datasets covering activity recognition, motion recognition, physiological signals, and other settings involving multiple observation channels [4]. Whereas each time step in UCR contains a single observation, each UEA sample consists of multiple channels evolving over time and is associated with a sequence-level class label. The model must identify temporal changes within individual channels and exploit relationships between channels. For example, activity recognition may require combining recordings from several motion sensors to distinguish actions with similar local patterns. UCR and UEA thus provide sequence classification evaluations under univariate and multivariate input settings, respectively.
(2) Event type prediction: EasyTPP. EasyTPP is a benchmarking framework for temporal point processes that provides event sequence datasets, model implementations, and standardized evaluation procedures [24]. The five real-world datasets used in its original paper are Amazon, Retweet, Taxi, Taobao, and StackOverflow. They contain user reviews, social reposts, taxi pick-up and drop-off events, e-commerce clicks, and user badge awards, respectively. Each sequence consists of events ordered by occurrence time, with each event containing a timestamp and a discrete event type. The intervals between events are generally irregular.
| Dataset group | Number | Mean ACC | Mean AUC | Median AUC |
| UCR binary | 41 | 0.5411 | 0.5341 | 0.5277 |
| UCR multiclass | 86 | 0.1874 | 0.5187 | 0.5047 |
| UEA | 17 | 0.1566 | 0.5172 | 0.4986 |
| EasyTPP | 5 | 0.5127 | 0.7028 | 0.6684 |
| ETT | 8 | 0.5137 | 0.5900 | 0.5912 |
| Stocks | 3 | 0.5034 | 0.5040 | 0.5042 |
Table 1. Overall results, averaged equally across datasets with available scores.
(3) Direction prediction: ETT and stock market data. The Electricity Transformer Temperature (ETT) dataset contains transformer monitoring records used in the Informer study [26]. Its ETT-small collection includes ETTh1, ETTh2, ETTm1, and ETTm2, with hourly or 15-minute observations of six electrical load variables and oil temperature (OT). The stock market data comprise daily records for AAPL, BA, and CAT released by Al Ridhawi et al. [1]. These datasets cover two forecasting settings: temperature changes in electrical equipment and price movements in financial markets.
This study reformulates continuous-valued forecasting on both datasets as binary direction prediction: the model reads a historical observation window and predicts whether the target value at a specified future horizon exceeds its current value, yielding “increase” and “not increase.” For ETT, the target is OT at a 24-hour horizon, corresponding to 24 hourly steps or 96 15-minute steps. We evaluate univariate (S) and multivariate (MS) input settings, using OT alone or OT together with the six electrical load variables, respectively. For the stock data, the target is the closing price at a fixed number of trading steps ahead. Together, these datasets examine Jev’s ability to use temporal observations to anticipate directional changes, providing information relevant to transformer operational monitoring and directional investment decisions.
Implementation Details
Each sample is represented by an input state, a task question, candidate answers, and a ground-truth label. Observations are arranged chronologically using consistent input settings within each experiment.
We evaluate Jev 1.13.0 with a fixed question template for each task. Binary tasks use Noul to estimate the probability of the positive event, while multiclass tasks use Choice to return a selected answer and candidate probabilities. Predictions and probabilities are recorded alongside ground-truth labels for evaluation.
Experimental Results and Analysis
Table 1 summarizes Jev’s performance across the evaluated dataset groups. Dataset-level results supporting the analysis are presented in Table 2–Table 6. (1) Numerical sequence classification shows limited overall discrimination: the mean AUC is 0.5341 for binary UCR tasks, 0.5187 for multiclass
| Setting | Dataset | ACC | AUC |
| Binary | Chinatown | 0.6793 | 0.6657 |
| DodgerLoopWeekend | 0.7464 | 0.6953 | |
| DodgerLoopGame | 0.5942 | 0.6475 | |
| ItalyPowerDemand | 0.5598 | 0.6597 | |
| GunPoint | 0.5200 | 0.7215 | |
| Wafer | 0.6525 | 0.8056 | |
| ECG200 | 0.5900 | 0.4696 | |
| ECGFiveDays | 0.4901 | 0.4434 | |
| TwoLeadECG | 0.5004 | 0.4629 | |
| Multiclass | SyntheticControl | 0.8100 | 0.9978 |
| SmoothSubspace | 0.9000 | 0.9815 | |
| BME | 0.3933 | 0.7434 | |
| CBF | 0.3889 | 0.7019 | |
| DodgerLoopDay | 0.2125 | 0.5056 | |
| TwoPatterns | 0.1380 | 0.3224 |
Table 2. Selected UCR results used in the analysis.
| Stock | ACC | AUC |
| AAPL | 0.5037 | 0.4950 |
| BA | 0.5000 | 0.5042 |
| CAT | 0.5064 | 0.5129 |
| Mean | 0.5034 | 0.5040 |
Table 6. Binary stock movement prediction results.
| Dataset | ACC | AUC |
| BasicMotions | 0.3000 | 0.7671 |
| Epilepsy | 0.2971 | 0.6622 |
| PenDigits | 0.1335 | 0.5998 |
| FingerMovements | 0.4400 | 0.4138 |
| AtrialFibrillation | 0.3333 | 0.3567 |
Table 3. Selected UEA results used in the analysis.
| Dataset | ACC | AUC |
| Amazon | 0.2795 | 0.6684 |
| Retweet | 0.5718 | 0.6466 |
| Stockflow | 0.4037 | 0.6470 |
| Taobao | 0.6343 | 0.8241 |
| Taxi | 0.6740 | 0.7282 |
| Mean | 0.5127 | 0.7028 |
Table 4. Event type prediction results on EasyTPP datasets.
UCR tasks, and 0.5172 for UEA tasks. Among the 86 multiclass UCR datasets, 63 have AUC values between 0.45 and 0.55, with a median of 0.5047. (2) Strong performance is concentrated in a small subset. SyntheticControl achieves an ACC of 0.8100 and an AUC of 0.9978, while SmoothSubspace achieves 0.9000 and 0.9815. Excluding these two datasets reduces the mean multiclass UCR AUC to 0.5075. These results show that success on selected temporal patterns has not translated into consistent performance across the archive.
The dataset-level results reveal differences associated with temporal structure, target granularity, and signal type. (1) Several stronger results involve recognizable trends or shapes. SyntheticControl contains classes defined by cycles, trends, and level shifts, while SmoothSubspace distinguishes the position of a continuous segment [8]. BME and CBF also achieve AUC values of 0.7434 and 0.7019. Chinatown and ItalyPowerDemand reach 0.6657 and 0.6597, suggesting some ability to recognize patterns in everyday activity and demand. However, Wafer reaches 0.8056 despite its specialized industrial setting, whereas the synthetic TwoPatterns dataset reaches only 0.3224. The results therefore suggest that accessibility of the discriminative pattern matters alongside domain familiarity. (2) Target granularity also affects performance: DodgerLoopWeekend achieves an AUC of 0.6953 when distinguishing weekdays from weekends, compared with 0.5056 for the seven-class DodgerLoopDay task. This contrast is consistent with stronger recognition of broad activity differences than fine distinctions within the same application domain. (3) Physiological signals remain difficult. ECG200, ECGFiveDays, and TwoLeadECG have AUC values of 0.4696, 0.4434, and 0.4629, while the EEG dataset FingerMovements reaches 0.4138. Limited domain-specific knowledge and insufficient representation of discriminative signal features are possible explanations. (4) Some scores retain useful ranking information despite weak final decisions. GunPoint achieves an AUC of 0.7215 with an ACC of 0.5200, and BasicMotions achieves an AUC of 0.7671 with an ACC of 0.3000. This gap motivates examining binary thresholds and the comparability of multiclass candidate scores.
Future-oriented tasks show a further contrast between event categories and numerical movement directions. (1) Event type prediction produces consistently higher AUC values in the reported experiments. Across the five EasyTPP datasets, AUC ranges from 0.6466 to 0.8241. Taobao achieves an ACC of 0.6343 and an AUC of 0.8241, while Taxi achieves 0.6740 and 0.7282. These results suggest that discrete event histories provide a useful input form for Jev, potentially because event categories expose predictive relationships more directly than raw numerical sequences. (2) Stock movement prediction remains close to chance-level discrimination. AAPL, BA, and CAT achieve AUC values of 0.4950, 0.5042, and 0.5129, respectively, with a mean ACC of 0.5034. Thus, converting continuous price forecasting into binary movement prediction makes the task compatible with the decision interface but yields little predictive discrimination in these experiments. Oil temperature direction prediction also shows modest discriminative ability, with a mean ACC of 0.5137 and a mean AUC of 0.5900 across the eight ETT settings (Table 5). Multivariate input achieves higher AUC on ETTm1, ETTm2, and ETTm2, while univariate input performs better on ETTh1. However, ACC remains close to 0.5 in most settings, suggesting a gap between probability ranking and final classification decisions. Together, the results identify discrete event prediction and selected explicit temporal patterns as the stronger settings, while numerical direction prediction and specialized signal classification remain challenging.
| Dataset | S ACC | S AUC | MS ACC | MS AUC |
| ETTh1 | 0.5126 | 0.6006 | 0.4895 | 0.5691 |
| ETTh2 | 0.5102 | 0.5816 | 0.5837 | 0.5935 |
| ETTm1 | 0.4898 | 0.5889 | 0.4909 | 0.6054 |
| ETTm2 | 0.5098 | 0.5559 | 0.5231 | 0.6253 |
| Mean | 0.5056 | 0.5818 | 0.5218 | 0.5983 |
Table 5. Oil temperature direction prediction results. Note: S denotes univariate input (oil temperature only), while MS denotes multivariate input (oil temperature and six electrical load variables).
The results suggest that future System 1 models should extend their recognition of explicit trends and simple shapes to finer temporal patterns and cross-channel relationships. Strong performance on SyntheticControl and SmoothSubspace contrasts with weaker results on physiological signals and most evaluated UEA datasets, motivating broader domain-specific training data and improved temporal and channel representations. These developments could strengthen the recognition of local waveforms and interactions between variables.
The higher AUC but lower ACC on GunPoint and BasicMotions also motivates better probability calibration, threshold selection, and alignment of multiclass scores. Meanwhile, the contrast between EasyTPP and stock movement prediction highlights the need for training that explicitly targets future decisions. Future models should improve the reliability of these decisions while retaining direct, structured outputs.
Conclusion
This study evaluates Jev 1.13.0 as a Structured System 1 model for time-series decisions through a unified evaluation framework. The results reveal uneven performance: Jev recognizes selected explicit temporal patterns and achieves relatively consistent discrimination in event type prediction, while performance on many numerical sequence classification tasks and stock movement prediction remains close to chance-level discrimination. Several datasets also show useful ranking information that does not translate into accurate final decisions. These findings highlight temporal representation, domain adaptation, and decision-score alignment as priorities for future Structured System 1 models. The framework supports extending this evaluation to additional models under consistent task definitions.
Complete UCR and UEA Results
Tables 7–9 present the complete UCR and UEA classification results.
| Dataset | ACC | AUC |
| BeetleFly | 0.4500 | 0.4200 |
| BirdChicken | 0.5500 | 0.5800 |
| Chinatown | 0.6793 | 0.6657 |
| Coffee | 0.5357 | 0.5128 |
| Computers | 0.5120 | 0.5022 |
| DistalPhalanxOutlineCorrect | 0.5688 | 0.5898 |
| DodgerLoopGame | 0.5942 | 0.6475 |
| DodgerLoopWeekend | 0.7464 | 0.6953 |
| Earthquakes | – | – |
| ECG200 | 0.5900 | 0.4696 |
| ECGFiveDays | 0.4901 | 0.4434 |
| FordA | 0.5114 | 0.5773 |
| FordB | 0.5173 | 0.5324 |
| FreezerRegularTrain | 0.5274 | 0.6013 |
| FreezerSmallTrain | 0.5246 | 0.5982 |
| GunPoint | 0.5200 | 0.7215 |
| GunPointAgeSpan | 0.4747 | 0.4514 |
| GunPointMaleVersusFemale | 0.4462 | 0.4655 |
| GunPointOldVersusYoung | 0.5460 | 0.5269 |
| Ham | 0.4952 | 0.4858 |
| HandOutlines | 0.6405 | 0.5805 |
| Herring | 0.5469 | 0.4165 |
| HouseTwenty | 0.6134 | 0.5048 |
| ItalyPowerDemand | 0.5598 | 0.6597 |
| Lightning2 | 0.5574 | 0.3956 |
| MiddlePhalanxOutlineCorrect | 0.5533 | 0.4993 |
| MoteStrain | 0.5391 | 0.6209 |
| PhalangesOutlinesCorrect | 0.6107 | 0.5219 |
| PowerCons | 0.5167 | 0.6019 |
| ProximalPhalanxOutlineCorrect | 0.6460 | 0.5277 |
| SemgHandGenderCh2 | 0.4683 | 0.5101 |
| ShapeletSim | 0.4778 | 0.3522 |
| SonyAIBORobotSurface1 | 0.5408 | 0.2929 |
| SonyAIBORobotSurface2 | 0.3442 | 0.2387 |
| Strawberry | 0.4378 | 0.5830 |
| ToeSegmentation1 | 0.6009 | 0.5998 |
| ToeSegmentation2 | 0.6308 | 0.6048 |
| TwoLeadECG | 0.5004 | 0.4629 |
| Wafer | 0.6525 | 0.8056 |
| Wine | 0.4815 | 0.5700 |
| WormsTwoClass | 0.4935 | 0.5492 |
| Yoga | 0.4947 | 0.5128 |
Table 7. Complete UCR binary results recorded in the source file. Missing scores are shown as –.
| Dataset | ACC | AUC |
| ACSF1 | 0.1000 | 0.4420 |
| Adiac | 0.0179 | 0.4969 |
| AllGestureWiimoteX | 0.1443 | 0.5621 |
| AllGestureWiimoteY | 0.1057 | 0.4572 |
| AllGestureWiimoteZ | 0.1171 | 0.5530 |
| ArrowHead | 0.3429 | 0.4816 |
| Beef | 0.2000 | 0.5597 |
| BME | 0.3933 | 0.7434 |
| Car | 0.2833 | 0.5857 |
| CBF | 0.3889 | 0.7019 |
| ChlorineConcentration | 0.2698 | 0.4654 |
| CinCECGTorso | 0.2536 | 0.4655 |
| CricketX | 0.0795 | 0.4812 |
| CricketY | 0.0718 | 0.4916 |
| CricketZ | 0.0821 | 0.4945 |
| Crop | 0.0475 | 0.5273 |
| DiatomSizeReduction | 0.2876 | 0.5240 |
| DistalPhalanxOutlineAgeGroup | 0.3813 | 0.4547 |
| DistalPhalanxTW | 0.1007 | 0.4074 |
| DodgerLoopDay | 0.2125 | 0.5056 |
| ECG5000 | 0.3811 | 0.5590 |
| ElectricDevices | 0.1323 | 0.5146 |
| EOGHorizontalSignal | 0.0718 | 0.5048 |
| EOGVerticalSignal | 0.0718 | 0.4936 |
| EthanolLevel | 0.2520 | 0.5199 |
| FaceAll | 0.0686 | 0.5063 |
| FaceFour | 0.2500 | 0.5375 |
| FacesUCR | 0.0683 | 0.5190 |
| FiftyWords | 0.0835 | 0.5048 |
| Fish | 0.1543 | 0.5235 |
| Fungi | 0.0645 | 0.5175 |
| GestureMidAirD1 | 0.0692 | 0.5705 |
| GestureMidAirD2 | 0.0846 | 0.5714 |
| GestureMidAirD3 | 0.0385 | 0.5762 |
| GesturePebbleZ1 | 0.1744 | 0.4911 |
| GesturePebbleZ2 | 0.1329 | 0.4943 |
| Haptics | 0.2143 | 0.5249 |
| InlineSkate | 0.0964 | 0.4832 |
| InsectEPGRegularTrain | 0.3373 | 0.5308 |
| InsectEPGSmallTrain | 0.3373 | 0.5304 |
| InsectWingbeatSound | 0.0975 | 0.5110 |
| LargeKitchenAppliances | 0.2907 | 0.3956 |
| Lightning7 | 0.1233 | 0.4915 |
| Mallat | 0.1190 | 0.5306 |
| Meat | 0.3333 | 0.5294 |
| MedicalImages | 0.1000 | 0.4920 |
| MelbournePedestrian | 0.1095 | 0.5190 |
| MiddlePhalanxOutlineAgeGroup | 0.4545 | 0.4114 |
| MiddlePhalanxTW | 0.0974 | 0.4648 |
| MixedShapesRegularTrain | 0.2256 | 0.5093 |
| MixedShapesSmallTrain | 0.2243 | 0.5029 |
| NonInvasiveFetalECGThorax1 | 0.0204 | 0.4941 |
| NonInvasiveFetalECGThorax2 | 0.0183 | 0.5045 |
| OliveOil | 0.2667 | 0.5510 |
| OSULeaf | 0.2149 | 0.5213 |
| Phoneme | 0.0211 | 0.5020 |
| PickupGestureWiimoteZ | 0.1200 | 0.5004 |
| PigAirwayPressure | 0.0192 | 0.4898 |
| PigArtPressure | 0.0192 | 0.4996 |
| PigCVP | 0.0192 | 0.4978 |
| PLAID | 0.0652 | 0.5016 |
| Plane | 0.1048 | 0.5274 |
| ProximalPhalanxOutlineAgeGroup | 0.3902 | 0.4332 |
| ProximalPhalanxTW | 0.0195 | 0.3344 |
| RefrigerationDevices | 0.3547 | 0.5161 |
| Rock | 0.4200 | 0.5535 |
| ScreenType | 0.3333 | 0.4908 |
| SemgHandMovementCh2 | 0.1578 | 0.4778 |
| SemgHandSubjectCh2 | 0.1822 | 0.5032 |
| ShakeGestureWiimoteZ | 0.1000 | 0.4560 |
| ShapesAll | 0.0167 | 0.5100 |
| SmallKitchenAppliances | 0.3280 | 0.4814 |
| SmoothSubspace | 0.9000 | 0.9815 |
| StarLightCurves | 0.2084 | 0.5856 |
| SwedishLeaf | 0.0800 | 0.5429 |
| Symbols | 0.1678 | 0.5365 |
| SyntheticControl | 0.8100 | 0.9978 |
| Trace | 0.2100 | 0.5328 |
| TwoPatterns | 0.1380 | 0.3224 |
| UMD | 0.4306 | 0.6121 |
| UWaveGestureLibraryAll | 0.1309 | 0.4791 |
| UWaveGestureLibraryX | 0.1259 | 0.4971 |
| UWaveGestureLibraryY | 0.1262 | 0.4968 |
| UWaveGestureLibraryZ | 0.1265 | 0.4859 |
| WordSynonyms | 0.0502 | 0.4875 |
| Worms | 0.2857 | 0.4766 |
Table 8. UCR results.
| Dataset | ACC | AUC |
| ArticularyWordRecognition | 0.0400 | 0.4802 |
| AtrialFibrillation | 0.3333 | 0.3567 |
| BasicMotions | 0.3000 | 0.7671 |
| CharacterTrajectories | 0.0599 | 0.5156 |
| Epilepsy | 0.2971 | 0.6622 |
| ERing | 0.1667 | 0.5620 |
| FingerMovements | 0.4400 | 0.4138 |
| Handwriting | 0.0412 | 0.4986 |
| JapaneseVowels | 0.1243 | 0.4935 |
| Libras | 0.0667 | 0.4950 |
| LSST | 0.0243 | 0.4898 |
| NATOPS | 0.1222 | 0.4552 |
| PenDigits | 0.1335 | 0.5998 |
| PhonemeSpectra | 0.0316 | 0.5160 |
| RacketSports | 0.2566 | 0.4613 |
| SpokenArabicDigits | 0.0991 | 0.5040 |
| UWaveGestureLibrary | 0.1250 | 0.5217 |
Table 9. UEA results
References
- [1]Mohammad Al Ridhawi, Mahtab Haj Ali, and Hussein Al Osman. Daily OHLCV data for 20 S&P 500 equities and the CBOE volatility index (1982–2025), 2026. URL https://doi.org/10.5281/zenodo.20192822. Dataset.
- [2]Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2403.07815.
- [3]Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Pablo Guerrero, Tony Hu, Junming Yin, Nick Erickson, Prateek Mutalik Desai, Hao Wang, Huzefa Rangwala, George Karypis, Yuyang Wang, and Michael Bohlke-Schneider. Chronos-2: From univariate to universal forecasting, 2025. URL https://arxiv.org/abs/2510.15821.
- [4]Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018, 2018. URL https://arxiv.org/abs/1811.00075.
- [5]Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001. doi: 10.1023/A:1010933404324.DOI
- [6]Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. Association for Computing Machinery, 2016. doi: 10.1145/2939672.2939785.DOI
- [7]Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273–297, 1995. doi: 10.1007/BF00994018.DOI
- [8]Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The UCR time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6):1293–1305, 2019. doi: 10.1109/JAS.2019.1911747.DOI
- [9]Tobias Deuber, Lorenz Sparrenberg, and Rafet Sifa. Evaluating and benchmarking the system one model Jev, 2026. URL https://arxiv.org/abs/2609.37647.
- [10]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/.DOI
- [11]Jonathan St. B. T. Evans and Keith E. Stanovich. Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science, 8(3):223–241, 2013. doi: 10.1177/1745691612460685.DOI
- [12]Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96, 2019. doi: 10.1038/s41597-019-0103-9.DOI
- [13]Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey I. Webb, Irwin King, and Shirui Pan. A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10466–10485, 2024. doi: 10.1109/TPAMI.2024.3443141.DOI
- [14]Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/680b2a8135b9c71278a09cafb605869e-Abstract-Conference.html.arxiv.org/abs/2310.01728
- [15]Ming Jin, Yaxuan Kong, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiao-Li Li, Vincent S. Tseng, Yu Zheng, Lei Chen, Hui Xiong, Shirui Pan, and Qingsong Wen. Large models for time series and spatio-temporal data: A survey and outlook. ACM Computing Surveys, 58(14), 2026. doi: 10.1145/3821637.DOI
- [16]Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In Proceedings of the European Conference of the PHM Society, 2016. doi: 10.36001/phme.2016.v3i1.1577.DOI
- [17]Haoxin Liu, Zhiyuan Zhao, Shiduo Li, and B. Aditya Prakash. Evaluating system 1 vs. 2 reasoning approaches for zero-shot time series forecasting: A benchmark and insights, 2025. URL https://arxiv.org/abs/2503.01895.
- [18]Jigar Patel, Sahil Shah, Priyank Thakkar, and Ketan Kotecha. Predicting stock and stock price index movement using trend deterministic data preparation and machine learning techniques. Expert Systems with Applications, 42(1):259–268, 2015. doi: 10.1016/j.eswa.2014.07.040.DOI
- [19]Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th International Symposium on Wearable Computers, pages 108–109. IEEE, 2012. doi: 10.1109/ISWC.2012.13.DOI
- [20]Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter, 2019. URL https://arxiv.org/abs/1910.01108.
- [21]TypeSafe AI. Introducing system one models and Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026. Accessed 2026-10-21.
- [22]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.arxiv.org/abs/1706.03762
- [23]Yumo Xu and Shay B. Cohen. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970–1979. Association for Computational Linguistics, 2018. doi: 10.18653/v1/P18-1183.DOI
- [24]Siqiao Xue, Xiaoming Shi, Zhixuan Chu, Yan Wang, Hongyan Hao, Fan Zhou, Caigao Jiang, Chen Pan, James Y. Zhang, Qingsong Wen, Jun Zhou, and Hongyuan Mei. EasyTPP: Towards open benchmarking temporal point processes. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper/2024/hash/e66233a208ef32f56df6312263239fa0-Abstract-Conference.html.arxiv.org/abs/2307.08097
- [25]Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1, 2024. URL https://arxiv.org/abs/2407.06023.
- [26]Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):11106–11115, 2021. doi: 10.1609/aaai.v35i12.17325.DOI

