betaFeedback

WorksComputer Science

Benchmarking System One Models for Time-Series Decision Making: A Case Study of Jev

Abstract

Time-series tasks require both accurate predictions and fast decisions. Advances in large models have introduced decision-oriented models that directly output answers and their probabilities, which we call Structured System 1 models. Their applicability to time-series tasks remains underexplored. Using Jev 1.13.0 as a case study, we evaluate time-series classification, event type prediction, oil temperature direction and stock movement prediction, converting continuous forecasting targets into discrete decisions to fit the model interface. We establish a unified evaluation framework using accuracy and AUC. Results show stronger performance on selected explicit temporal patterns and event type prediction, but near-chance discrimination on many numerical sequence classification tasks and stock movement prediction. Some datasets also reveal a gap between ranking quality and final decision accuracy. These findings motivate improvements in temporal representation, domain adaptation, and decision-score alignment. Standardized sample construction, model calls, and result recording support extending the evaluation to future models.

Introduction

Time series record how a system changes over time and support applications across diverse domains: wearable sensors enable human activity recognition [19], physiological measurements support patient assessment and deterioration prediction [12], operational signals enable equipment fault detection and diagnosis [16], and financial observations support market movement prediction [18]. These applications require time-series analysis to balance accuracy and processing speed. Traditional machine learning methods, including support vector machines, random forests, and XGBoost [7, 5, 6], are typically trained using task-specific features, and their predictive performance depends on feature representations and training data.

Recent advances in large language models and time-series foundation models have introduced new approaches to time-series analysis. Time-LLM maps temporal patches into the representation space of a frozen language model, Chronos quantizes time-series values for pretraining across domains, and Chronos-2 further supports zero-shot multivariate and covariate-informed forecasting [14, 2, 3]. Some studies use natural-language instructions to guide models in making time-series predictions [17].

Although complex temporal dependencies may require stronger modeling and reasoning capabilities, many time-series tasks can be naturally formulated as direct decision problems rather than continuous-valued forecasting. A unified, direct, and structured decision-making approach also offers potential advantages in computational efficiency and reduced generation overhead. These considerations motivate evaluating whether Structured System 1 can effectively interpret time-series information and produce reliable decisions. Inspired by dual-process theories in psychology, related studies use

Preprint. System 1 to describe fast, direct judgments and System 2 to describe deliberative reasoning involving multiple steps [11, 25], as shown in Figure 1(a).

Figure 1. (a) Schematic of System 1 and System 2, illustrating intuitive judgments and deliberative reasoning in dual-process theory. (b) Model taxonomy under our operational definitions: System 1 provides direct answers through structured outputs or text generation, while System 2 uses intermediate reasoning through test-time enhancement or built-in reasoning.

Existing studies differ in how they distinguish System 1 from System 2. Distilling System 2 into System 1 categorizes Chain-of-Thought (CoT) and other methods that generate intermediate reasoning steps as System 2 [25]. The time-series forecasting benchmark ReC4TS instead treats CoT, self-consistency, and self-correction as “System 1 reasoning enhancements,” while categorizing models that undergo dedicated reasoning post-training as System 2 [17].

This paper adopts an operational distinction based on the inference procedure. As shown in Figure 1(b), System 1 produces final answers without generating an intermediate reasoning sequence and is divided into Structured System 1 and Generative System 1 according to the output mechanism. The former directly computes decisions, while the latter generates answers through text decoding, with implications for response speed and output stability. System 2 uses additional inference-time computation for intermediate derivation, search, or reflection through Built-in Reasoning or Test-Time Enhanced approaches, which can be combined. Existing time-series research has focused primarily on System 2 and Generative System 1, with fewer systematic evaluations of Structured System 1.

Structured System 1 models score predefined alternatives, labels, or ordered levels conditioned on a state, a natural-language question, and candidate answers, returning decisions and probabilities through a unified interface. This output format supports time-series classification, anomaly detection, and discretized forecasting. A typical implementation uses an encoder to obtain contextual representations and a classification or scoring head to map them into the target output space, reducing sequential decoding and answer-parsing overhead. BERT combines a pretrained encoder with task-specific output layers for classification and natural language inference, while DistilBERT explores smaller and faster language representation models, providing a foundation for this approach [10, 20].

Jev is a System One model developed by TypeSafe AI, characterized by low latency and low API cost [21]. Independent research has evaluated its decision quality and invocation cost on text classification, natural language inference, reading comprehension, and content moderation [9], while its performance on time-series tasks awaits systematic evaluation. We use Jev as a case study, organizing temporal observations into a unified state-and-question representation and evaluating decision quality using accuracy (ACC) and the area under the receiver operating characteristic curve (AUC) to assess the applicability of Structured System 1 across time-series tasks.

Our contributions are threefold:

  • We define the task scope and decision mappings for time-series classification, anomaly detection, and adapted forecasting, mapping continuous future values to explicit decision targets.

  • We evaluate Jev 1.13.0 through binary and multiclass decision interfaces, reporting ACC and AUC and analyzing performance across datasets and task types.

  • We establish an extensible evaluation framework that standardizes sample construction, model calls, and result recording, enabling subsequent Structured System 1 models to be evaluated under the same task definitions and evaluation protocols.

Problem formulation

Time-series inputs and decision outputs

Following established time-series definitions [13, 15], let

X=(x1,…,xT)∈RT×D,xt∈RD,(1)\mathbf{X} = (\mathbf{x}_1,\ldots,\mathbf{x}_T) \in\mathbb{R}^{T \times D}, \qquad\mathbf{x}_t \in\mathbb{R}^{D}, \tag*{(1)}

where TT is the number of ordered observations, DD is the number of channels, and xt\mathbf{x}_t contains the values at time tt. A univariate series is the special case D=1D=1. For a decision at time tt, an observed window of length LL is

X(t)=Xt−L+1:t=(xt−L+1,…,xt)∈RL×D,L≤t≤T.(2)\mathbf{X}^{(t)} = \mathbf{X}_{t-L+1:t} = (\mathbf{x}_{t-L+1},\ldots,\mathbf{x}_t) \in\mathbb{R}^{L \times D}, \qquad L \le t \le T. \tag*{(2)}

The right endpoint is tt and values after tt cannot be used in the input to a future-oriented decision.

Some observations are events at irregular times. We write their history as

Ht={(τi,mi):τi≤t, i=1,…,nt},(3)\mathcal{H}_t = \{(\tau_i,m_i): \tau_i \le t,\ i=1,\ldots,n_t\}, \tag*{(3)}

where τi\tau_i is the event time, mim_i is its categorical mark, and ntn_t is the number of observed events by time tt. This representation retains the event times instead of imposing a regular sampling interval. We use I\mathcal{I} for the task-specific input, whether a complete series, a window, or an event history.

Let s=σ(I)s=\sigma(\mathcal{I}) be the state supplied to the model, with σ\sigma denoting a fixed representation procedure. Given a question qq and a candidate set Y={1,…,K}\mathcal{Y}=\{1,\ldots,K\}, a decision model returns

pθ(s,q,Y)=(pθ(k∣s,q,Y))k=1K,∑k=1Kpθ(k∣s,q,Y)=1,(4)p_\theta(s,q,\mathcal{Y}) = \left(p_\theta(k \mid s,q,\mathcal{Y})\right)_{k=1}^{K}, \qquad\sum_{k=1}^{K} p_\theta(k \mid s,q,\mathcal{Y})=1, \tag*{(4)}

with nonnegative components. Here θ\theta denotes model parameters and KK is the number of admissible decisions. The predicted answer is y^=arg⁡max⁡k∈Y pθ(k∣s,q,Y)\hat{y}=\underset{k\in\mathcal{Y}}{\arg\max}\ p_\theta(k\mid s,q,\mathcal{Y}). The three tasks below differ in what is observed, what the label denotes, and when it becomes known.

Time-series classification

For sample ii, let X(i)∈RTi×D\mathbf{X}^{(i)} \in\mathbb{R}^{T_i \times D} be an observed series of length TiT_i and let yi∈{1,…,K}y_i \in\{1,\ldots,K\} be its class. The goal is to estimate

pθ(yi=k∣σ(X(i)),q,Y),y^i=arg⁡max⁡k∈Y pθ(yi=k∣σ(X(i)),q,Y).(5)p_\theta\left(y_i=k \mid\sigma(\mathbf{X}^{(i)}),q,\mathcal{Y}\right), \qquad\hat{y}_i=\underset{k\in\mathcal{Y}}{\arg\max}\ p_\theta\left(y_i=k \mid\sigma(\mathbf{X}^{(i)}),q,\mathcal{Y}\right). \tag*{(5)}

The index ii identifies a sample, TiT_i its length, and DD the number of channels within its dataset. A segment cut from a continuous activity record can itself be a classification sample. For K=2K=2 the task is binary, for K>2K>2 it is multiclass. The label describes the observed segment rather than an unobserved future.

Time-series anomaly detection

Let at∈{0,1}a_t \in\{0,1\} indicate whether time tt is anomalous, with 1 denoting an anomaly. Using the historical window ending at tt, define the anomaly score

rt=pθ(at=1∣σ(X(t)),q,{0,1}),a^t=I[rt≥η],(6)r_t = p_\theta\left(a_t=1 \mid\sigma(\mathbf{X}^{(t)}),q,\{0,1\}\right), \qquad\hat{a}_t=\mathbb{I}[r_t \ge\eta], \tag*{(6)}

where η∈[0,1]\eta\in[0,1] is a prespecified decision threshold and I[⋅]\mathbb{I}[\cdot] is the indicator function. If anomalies are annotated as intervals [uj,vj][u_j,v_j], the corresponding point label is at=I[t∈⋃j[uj,vj]]a_t=\mathbb{I}[t\in\bigcup_j[u_j,v_j]]. The window supplies evidence, whereas its right endpoint is the object being labeled. This distinction prevents the label “the window contains an anomaly” from silently replacing the label “time tt is anomalous.” The continuous score rtr_t can also be used to compute threshold-independent ranking measures.

Decision-oriented time-series forecasting

Conventional forecasting estimates future continuous observations. Let HH be the forecast horizon and DyD_y the number of forecast target channels. With yt+h∈RDy\mathbf{y}_{t+h} \in\mathbb{R}^{D_y}, the future target is

Y(t)=(yt+1,…,yt+H)∈RH×Dy,Y^(t)=fθ(X(t)),(7)\mathbf{Y}^{(t)} = (\mathbf{y}_{t+1}, \ldots, \mathbf{y}_{t+H}) \in\mathbb{R}^{H \times D_y}, \qquad\widehat{\mathbf{Y}}^{(t)} = f_\theta(\mathbf{X}^{(t)}), \tag*{(7)}

where 1≤h≤H1 \le h \le H denotes a future step and fθf_\theta is a numeric forecasting model. For example, squared-error training minimizes a sum of ∥Y(t)−fθ(X(t))∥F2\lVert\mathbf{Y}^{(t)} - f_\theta(\mathbf{X}^{(t)}) \rVert_F^2 over training examples, with ∥⋅∥F\lVert\cdot\rVert_F denoting the Frobenius norm.

Our reformulation fixes a label map ϕ\phi before evaluating the model:

zt=ϕ(Y(t);λ)∈{1,…,K},z^t=arg⁡max⁡k∈Y  pθ(zt=k∣σ(X(t)),q,Y).(8)z_t = \phi(\mathbf{Y}^{(t)};\lambda) \in\{1,\ldots,K\}, \qquad\widehat{z}_t = \underset{k \in\mathcal{Y}}{\arg\max}\; p_\theta\left(z_t = k \mid\sigma(\mathbf{X}^{(t)}), q, \mathbf{Y}\right). \tag*{(8)}

Here λ\lambda collects task parameters, such as a horizon or threshold. The future values generate the ground-truth label ztz_t, but are never included in the input state. The model directly answers a question about an event in the future.

For a concrete example, let Pt>0P_t > 0 be a stock’s closing price at time tt, and define the hh-step return, for h≤Hh \le H, by

Rt,h=Pt+h−PtPt.(9)R_{t,h} = \frac{P_{t+h} - P_t}{P_t}. \tag*{(9)}

The binary upward-movement target is

zt=I[Rt,h>0]=I[Pt+h>Pt].(10)z_t = \mathbb{I}[R_{t,h} > 0] = \mathbb{I}[P_{t+h} > P_t]. \tag*{(10)}

The equivalence follows because dividing by the positive PtP_t does not change the sign of Pt+h−PtP_{t+h} - P_t. The requested quantity is thus Pr⁡(Pt+h>Pt∣X(t))\Pr(P_{t+h} > P_t \mid\mathbf{X}^{(t)}), rather than a point estimate of Pt+hP_{t+h}. A three-class version, with a fixed tolerance δ>0\delta> 0, uses

zt={down,Rt,h<−δ,flat,−δ≤Rt,h≤δ,up,Rt,h>δ.(11)z_t = \begin{cases} \text{down}, & R_{t,h} < -\delta,\\ \text{flat}, & -\delta\le R_{t,h} \le\delta,\\ \text{up}, & R_{t,h} > \delta. \end{cases} \tag*{(11)}

The threshold δ\delta determines the width of the flat class and must remain fixed across systems being compared. Stock movement prediction has previously been posed as a directional task [23], the example here illustrates the general conversion of a continuous forecast target into an explicit decision event.

The reformulation changes the statistical object being estimated. For a scalar future price U=Pt+hU = P_{t+h} and a squared-error point forecast cc, differentiating its conditional loss gives

∂∂cE[(U−c)2∣X(t)]=2(c−E[U∣X(t)]),c∗=E[U∣X(t)].(12)\frac{\partial}{\partial c}\mathbb{E}[(U-c)^2 \mid\mathbf{X}^{(t)}] = 2(c-\mathbb{E}[U \mid\mathbf{X}^{(t)}]), \qquad c^* = \mathbb{E}[U \mid\mathbf{X}^{(t)}]. \tag*{(12)}

In contrast, the binary decision depends on Pr⁡(U>Pt∣X(t))\Pr(U > P_t \mid\mathbf{X}^{(t)}). A conditional mean and a probability mass above a threshold summarize different aspects of the future distribution. The same historical window can therefore lead to different optimal numerical and decision outputs.

Mapping tasks to typed answers

Equation (4) gives a common output form. Sequence classification uses the class of the observed sample, anomaly detection uses the status of a specified endpoint, and decision-oriented forecasting uses a future-derived class. Binary questions are evaluated through Jev’s Noul primitive, which returns the probability that a statement is true. Choice selects from a fixed candidate set and returns option probabilities, supporting both binary and multiclass decisions. The label construction and the evaluation unit remain task-specific even though the external decision format is shared.

Method

Jev is a commercial model whose full internal architecture is not publicly documented [21]. Figure 2 illustrates one possible architecture for Structured System 1 decision making using a BERT-based classifier [10]. We use this example to explain the main computational stages and evaluate Jev through its public decision interface.

Figure 2. One possible architecture for a Structured System 1 model, illustrated using BERT. Embedding, encoding, and classification map the input to class probabilities and a predicted class.

Input Embedding

Given an observed time series I\mathcal{I}, a representation procedure constructs the input state s=σ(I)s = \sigma(\mathcal{I}). The state records observations chronologically and can include channel names and timestamps. Together with the task question qq, it is serialized into a token sequence:

u=Tokenize⁡([CLS] s [SEP] q [SEP]).(13)\mathbf{u} = \operatorname{Tokenize}([\mathrm{CLS}]\ s\ [\mathrm{SEP}]\ q\ [\mathrm{SEP}]). \tag*{(13)}

Here, [CLS] is the classification token and [SEP] separates the state and question. Numerical observations are represented through the tokens produced by the tokenizer.

BERT combines token, position, and segment embeddings to construct the initial representation of token uju_j:

hj(0)=etok(uj)+epos(j)+eseg(aj),(14)\mathbf{h}_j^{(0)} = \mathbf{e}_{\mathrm{tok}}(u_j) + \mathbf{e}_{\mathrm{pos}}(j) + \mathbf{e}_{\mathrm{seg}}(a_j), \tag*{(14)}

where aja_j identifies whether the token belongs to the state or question. For NN tokens and hidden dimension dd, these vectors form H(0)∈RN×d\mathbf{H}^{(0)} \in\mathbb{R}^{N \times d}. Token embeddings represent the input content, position embeddings distinguish token locations, and segment embeddings distinguish the two input components [10].

Contextual Encoder

The embedding sequence passes through a stack of bidirectional Transformer encoder layers:

H(ℓ)=Encoder⁡ℓ(H(ℓ−1)),ℓ=1,…,Lenc,(15)\mathbf{H}^{(\ell)} = \operatorname{Encoder}_{\ell}\left(\mathbf{H}^{(\ell-1)}\right), \qquad\ell= 1,\ldots,L_{\mathrm{enc}}, \tag*{(15)}

where LencL_{\mathrm{enc}} is the number of encoder layers. Each layer uses multi-head self-attention and a feedforward network, together with residual connections and layer normalization [22].

For one attention head, the input representations are projected into queries, keys, and values:

Q=HWQ,K=HWK,V=HWV.(16)Q = \mathbf{H}W_Q,\qquad K = \mathbf{H}W_K,\qquad V = \mathbf{H}W_V. \tag*{(16)}

The attention output is

Attention⁡(Q,K,V)=softmax⁡(QKTdk)V,(17)\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^{\mathsf{T}}}{\sqrt{d_k}}\right)V, \tag*{(17)}

where dkd_k is the key dimension. This operation allows each token to incorporate information from other tokens in the observed input. In the time-series setting, it provides a mechanism for relating observations at different positions and conditioning their interpretation on the task question.

The final classification-token representation,

hCLS=HCLS(Lenc)∈Rd,(18)\mathbf{h}_{\mathrm{CLS}} = \mathbf{H}_{\mathrm{CLS}}^{(L_{\mathrm{enc}})} \in\mathbb{R}^{d}, \tag*{(18)}

provides a contextual representation for the decision head. Bidirectional encoding operates over the supplied observations, which contain only the history available for the task.

Classification and Decision Probabilities

For a fixed answer set Y={1,…,K}\mathcal{Y} = \{1,\ldots,K\}, a linear classification head maps the contextual representation to class logits:

ℓ=WhCLS+b∈RK,(19)\ell= W\mathbf{h}_{\mathrm{CLS}} + \mathbf{b} \in\mathbb{R}^{K}, \tag*{(19)}

where W∈RK×dW \in\mathbb{R}^{K \times d} and b∈RK\mathbf{b} \in\mathbb{R}^{K} are learned parameters. A softmax converts these logits into a probability distribution:

pθ(y=k∣s,q,Y)=exp⁡(ℓk)∑j=1Kexp⁡(ℓj),y^=arg⁡max⁡k∈Ypθ(y=k∣s,q,Y).(20)p_{\theta}(y = k \mid s, q, \mathcal{Y}) = \frac{\exp(\ell_k)}{\sum_{j=1}^{K} \exp(\ell_j)}, \qquad\hat{y} = \arg\max_{k \in\mathcal{Y}} p_{\theta}(y = k \mid s, q, \mathcal{Y}). \tag*{(20)}

For binary decisions, a single logit v=wThCLS+bv = \mathbf{w}^{\mathsf{T}}\mathbf{h}_{\mathrm{CLS}} + b can instead be converted into the positive-class probability:

pθ(y=1∣s,q)=11+e−v,pθ(y=0∣s,q)=1−pθ(y=1∣s,q).(21)p_{\theta}(y = 1 \mid s, q) = \frac{1}{1 + e^{-v}}, \qquad p_{\theta}(y = 0 \mid s, q) = 1 - p_{\theta}(y = 1 \mid s, q). \tag*{(21)}

The fixed-label head illustrates standard BERT classification. To accommodate changing candidate sets, an alternative is to encode each candidate together with the state and question, apply a shared scoring head, and normalize the resulting scores across candidates. This makes the output depend explicitly on the supplied alternatives.

In both cases, the encoder representations are mapped directly to decision probabilities without generating an answer token by token. The predicted class supports ACC evaluation, while the probabilities support AUC evaluation. In our Jev experiments, these outputs are obtained through Noul for binary questions and Choice for multiclass questions.

Experiments

Datasets And Evaluation Metrics

This study selects datasets suitable for evaluating System 1 models, covering time series classification, event type prediction, time series anomaly detection, and forecasting. These datasets include univariate and multivariate numerical sequences, as well as irregular event sequences consisting of timestamps and event types. We evaluate Jev on these datasets using accuracy (ACC) and the area under the receiver operating characteristic curve (AUC).

(1) Time series classification: UCR and UEA archives. The UCR Archive is a representative collection of datasets for univariate time series classification. Its 2018 release contains 128 datasets, covering sensor recordings, physiological signals such as electrocardiograms, motion data, and sequences derived from image outlines [8]. Each sample consists of a univariate sequence and its class label. The number of samples, sequence lengths, and numbers of classes vary substantially across datasets. The task is therefore univariate time series classification: the model reads an entire sequence and predicts its class, under either a binary or multiclass setting. This study uses the archive to examine Jev’s ability to recognize temporal patterns in a single channel.

The UEA Archive focuses on multivariate time series classification. Its 2018 release contains 30 datasets covering activity recognition, motion recognition, physiological signals, and other settings involving multiple observation channels [4]. Whereas each time step in UCR contains a single observation, each UEA sample consists of multiple channels evolving over time and is associated with a sequence-level class label. The model must identify temporal changes within individual channels and exploit relationships between channels. For example, activity recognition may require combining recordings from several motion sensors to distinguish actions with similar local patterns. UCR and UEA thus provide sequence classification evaluations under univariate and multivariate input settings, respectively.

(2) Event type prediction: EasyTPP. EasyTPP is a benchmarking framework for temporal point processes that provides event sequence datasets, model implementations, and standardized evaluation procedures [24]. The five real-world datasets used in its original paper are Amazon, Retweet, Taxi, Taobao, and StackOverflow. They contain user reviews, social reposts, taxi pick-up and drop-off events, e-commerce clicks, and user badge awards, respectively. Each sequence consists of events ordered by occurrence time, with each event containing a timestamp and a discrete event type. The intervals between events are generally irregular.

Dataset groupNumberMean ACCMean AUCMedian AUC
UCR binary410.54110.53410.5277
UCR multiclass860.18740.51870.5047
UEA170.15660.51720.4986
EasyTPP50.51270.70280.6684
ETT80.51370.59000.5912
Stocks30.50340.50400.5042

Table 1. Overall results, averaged equally across datasets with available scores.

(3) Direction prediction: ETT and stock market data. The Electricity Transformer Temperature (ETT) dataset contains transformer monitoring records used in the Informer study [26]. Its ETT-small collection includes ETTh1, ETTh2, ETTm1, and ETTm2, with hourly or 15-minute observations of six electrical load variables and oil temperature (OT). The stock market data comprise daily records for AAPL, BA, and CAT released by Al Ridhawi et al. [1]. These datasets cover two forecasting settings: temperature changes in electrical equipment and price movements in financial markets.

This study reformulates continuous-valued forecasting on both datasets as binary direction prediction: the model reads a historical observation window and predicts whether the target value at a specified future horizon exceeds its current value, yielding “increase” and “not increase.” For ETT, the target is OT at a 24-hour horizon, corresponding to 24 hourly steps or 96 15-minute steps. We evaluate univariate (S) and multivariate (MS) input settings, using OT alone or OT together with the six electrical load variables, respectively. For the stock data, the target is the closing price at a fixed number of trading steps ahead. Together, these datasets examine Jev’s ability to use temporal observations to anticipate directional changes, providing information relevant to transformer operational monitoring and directional investment decisions.

Implementation Details

Each sample is represented by an input state, a task question, candidate answers, and a ground-truth label. Observations are arranged chronologically using consistent input settings within each experiment.

We evaluate Jev 1.13.0 with a fixed question template for each task. Binary tasks use Noul to estimate the probability of the positive event, while multiclass tasks use Choice to return a selected answer and candidate probabilities. Predictions and probabilities are recorded alongside ground-truth labels for evaluation.

Experimental Results and Analysis

Table 1 summarizes Jev’s performance across the evaluated dataset groups. Dataset-level results supporting the analysis are presented in Table 2–Table 6. (1) Numerical sequence classification shows limited overall discrimination: the mean AUC is 0.5341 for binary UCR tasks, 0.5187 for multiclass

SettingDatasetACCAUC
BinaryChinatown0.67930.6657
DodgerLoopWeekend0.74640.6953
DodgerLoopGame0.59420.6475
ItalyPowerDemand0.55980.6597
GunPoint0.52000.7215
Wafer0.65250.8056
ECG2000.59000.4696
ECGFiveDays0.49010.4434
TwoLeadECG0.50040.4629
MulticlassSyntheticControl0.81000.9978
SmoothSubspace0.90000.9815
BME0.39330.7434
CBF0.38890.7019
DodgerLoopDay0.21250.5056
TwoPatterns0.13800.3224

Table 2. Selected UCR results used in the analysis.

StockACCAUC
AAPL0.50370.4950
BA0.50000.5042
CAT0.50640.5129
Mean0.50340.5040

Table 6. Binary stock movement prediction results.

DatasetACCAUC
BasicMotions0.30000.7671
Epilepsy0.29710.6622
PenDigits0.13350.5998
FingerMovements0.44000.4138
AtrialFibrillation0.33330.3567

Table 3. Selected UEA results used in the analysis.

DatasetACCAUC
Amazon0.27950.6684
Retweet0.57180.6466
Stockflow0.40370.6470
Taobao0.63430.8241
Taxi0.67400.7282
Mean0.51270.7028

Table 4. Event type prediction results on EasyTPP datasets.

UCR tasks, and 0.5172 for UEA tasks. Among the 86 multiclass UCR datasets, 63 have AUC values between 0.45 and 0.55, with a median of 0.5047. (2) Strong performance is concentrated in a small subset. SyntheticControl achieves an ACC of 0.8100 and an AUC of 0.9978, while SmoothSubspace achieves 0.9000 and 0.9815. Excluding these two datasets reduces the mean multiclass UCR AUC to 0.5075. These results show that success on selected temporal patterns has not translated into consistent performance across the archive.

The dataset-level results reveal differences associated with temporal structure, target granularity, and signal type. (1) Several stronger results involve recognizable trends or shapes. SyntheticControl contains classes defined by cycles, trends, and level shifts, while SmoothSubspace distinguishes the position of a continuous segment [8]. BME and CBF also achieve AUC values of 0.7434 and 0.7019. Chinatown and ItalyPowerDemand reach 0.6657 and 0.6597, suggesting some ability to recognize patterns in everyday activity and demand. However, Wafer reaches 0.8056 despite its specialized industrial setting, whereas the synthetic TwoPatterns dataset reaches only 0.3224. The results therefore suggest that accessibility of the discriminative pattern matters alongside domain familiarity. (2) Target granularity also affects performance: DodgerLoopWeekend achieves an AUC of 0.6953 when distinguishing weekdays from weekends, compared with 0.5056 for the seven-class DodgerLoopDay task. This contrast is consistent with stronger recognition of broad activity differences than fine distinctions within the same application domain. (3) Physiological signals remain difficult. ECG200, ECGFiveDays, and TwoLeadECG have AUC values of 0.4696, 0.4434, and 0.4629, while the EEG dataset FingerMovements reaches 0.4138. Limited domain-specific knowledge and insufficient representation of discriminative signal features are possible explanations. (4) Some scores retain useful ranking information despite weak final decisions. GunPoint achieves an AUC of 0.7215 with an ACC of 0.5200, and BasicMotions achieves an AUC of 0.7671 with an ACC of 0.3000. This gap motivates examining binary thresholds and the comparability of multiclass candidate scores.

Future-oriented tasks show a further contrast between event categories and numerical movement directions. (1) Event type prediction produces consistently higher AUC values in the reported experiments. Across the five EasyTPP datasets, AUC ranges from 0.6466 to 0.8241. Taobao achieves an ACC of 0.6343 and an AUC of 0.8241, while Taxi achieves 0.6740 and 0.7282. These results suggest that discrete event histories provide a useful input form for Jev, potentially because event categories expose predictive relationships more directly than raw numerical sequences. (2) Stock movement prediction remains close to chance-level discrimination. AAPL, BA, and CAT achieve AUC values of 0.4950, 0.5042, and 0.5129, respectively, with a mean ACC of 0.5034. Thus, converting continuous price forecasting into binary movement prediction makes the task compatible with the decision interface but yields little predictive discrimination in these experiments. Oil temperature direction prediction also shows modest discriminative ability, with a mean ACC of 0.5137 and a mean AUC of 0.5900 across the eight ETT settings (Table 5). Multivariate input achieves higher AUC on ETTm1, ETTm2, and ETTm2, while univariate input performs better on ETTh1. However, ACC remains close to 0.5 in most settings, suggesting a gap between probability ranking and final classification decisions. Together, the results identify discrete event prediction and selected explicit temporal patterns as the stronger settings, while numerical direction prediction and specialized signal classification remain challenging.

DatasetS ACCS AUCMS ACCMS AUC
ETTh10.51260.60060.48950.5691
ETTh20.51020.58160.58370.5935
ETTm10.48980.58890.49090.6054
ETTm20.50980.55590.52310.6253
Mean0.50560.58180.52180.5983

Table 5. Oil temperature direction prediction results. Note: S denotes univariate input (oil temperature only), while MS denotes multivariate input (oil temperature and six electrical load variables).

The results suggest that future System 1 models should extend their recognition of explicit trends and simple shapes to finer temporal patterns and cross-channel relationships. Strong performance on SyntheticControl and SmoothSubspace contrasts with weaker results on physiological signals and most evaluated UEA datasets, motivating broader domain-specific training data and improved temporal and channel representations. These developments could strengthen the recognition of local waveforms and interactions between variables.

The higher AUC but lower ACC on GunPoint and BasicMotions also motivates better probability calibration, threshold selection, and alignment of multiclass scores. Meanwhile, the contrast between EasyTPP and stock movement prediction highlights the need for training that explicitly targets future decisions. Future models should improve the reliability of these decisions while retaining direct, structured outputs.

Conclusion

This study evaluates Jev 1.13.0 as a Structured System 1 model for time-series decisions through a unified evaluation framework. The results reveal uneven performance: Jev recognizes selected explicit temporal patterns and achieves relatively consistent discrimination in event type prediction, while performance on many numerical sequence classification tasks and stock movement prediction remains close to chance-level discrimination. Several datasets also show useful ranking information that does not translate into accurate final decisions. These findings highlight temporal representation, domain adaptation, and decision-score alignment as priorities for future Structured System 1 models. The framework supports extending this evaluation to additional models under consistent task definitions.

Complete UCR and UEA Results

Tables 7–9 present the complete UCR and UEA classification results.

DatasetACCAUC
BeetleFly0.45000.4200
BirdChicken0.55000.5800
Chinatown0.67930.6657
Coffee0.53570.5128
Computers0.51200.5022
DistalPhalanxOutlineCorrect0.56880.5898
DodgerLoopGame0.59420.6475
DodgerLoopWeekend0.74640.6953
Earthquakes––
ECG2000.59000.4696
ECGFiveDays0.49010.4434
FordA0.51140.5773
FordB0.51730.5324
FreezerRegularTrain0.52740.6013
FreezerSmallTrain0.52460.5982
GunPoint0.52000.7215
GunPointAgeSpan0.47470.4514
GunPointMaleVersusFemale0.44620.4655
GunPointOldVersusYoung0.54600.5269
Ham0.49520.4858
HandOutlines0.64050.5805
Herring0.54690.4165
HouseTwenty0.61340.5048
ItalyPowerDemand0.55980.6597
Lightning20.55740.3956
MiddlePhalanxOutlineCorrect0.55330.4993
MoteStrain0.53910.6209
PhalangesOutlinesCorrect0.61070.5219
PowerCons0.51670.6019
ProximalPhalanxOutlineCorrect0.64600.5277
SemgHandGenderCh20.46830.5101
ShapeletSim0.47780.3522
SonyAIBORobotSurface10.54080.2929
SonyAIBORobotSurface20.34420.2387
Strawberry0.43780.5830
ToeSegmentation10.60090.5998
ToeSegmentation20.63080.6048
TwoLeadECG0.50040.4629
Wafer0.65250.8056
Wine0.48150.5700
WormsTwoClass0.49350.5492
Yoga0.49470.5128

Table 7. Complete UCR binary results recorded in the source file. Missing scores are shown as –.

DatasetACCAUC
ACSF10.10000.4420
Adiac0.01790.4969
AllGestureWiimoteX0.14430.5621
AllGestureWiimoteY0.10570.4572
AllGestureWiimoteZ0.11710.5530
ArrowHead0.34290.4816
Beef0.20000.5597
BME0.39330.7434
Car0.28330.5857
CBF0.38890.7019
ChlorineConcentration0.26980.4654
CinCECGTorso0.25360.4655
CricketX0.07950.4812
CricketY0.07180.4916
CricketZ0.08210.4945
Crop0.04750.5273
DiatomSizeReduction0.28760.5240
DistalPhalanxOutlineAgeGroup0.38130.4547
DistalPhalanxTW0.10070.4074
DodgerLoopDay0.21250.5056
ECG50000.38110.5590
ElectricDevices0.13230.5146
EOGHorizontalSignal0.07180.5048
EOGVerticalSignal0.07180.4936
EthanolLevel0.25200.5199
FaceAll0.06860.5063
FaceFour0.25000.5375
FacesUCR0.06830.5190
FiftyWords0.08350.5048
Fish0.15430.5235
Fungi0.06450.5175
GestureMidAirD10.06920.5705
GestureMidAirD20.08460.5714
GestureMidAirD30.03850.5762
GesturePebbleZ10.17440.4911
GesturePebbleZ20.13290.4943
Haptics0.21430.5249
InlineSkate0.09640.4832
InsectEPGRegularTrain0.33730.5308
InsectEPGSmallTrain0.33730.5304
InsectWingbeatSound0.09750.5110
LargeKitchenAppliances0.29070.3956
Lightning70.12330.4915
Mallat0.11900.5306
Meat0.33330.5294
MedicalImages0.10000.4920
MelbournePedestrian0.10950.5190
MiddlePhalanxOutlineAgeGroup0.45450.4114
MiddlePhalanxTW0.09740.4648
MixedShapesRegularTrain0.22560.5093
MixedShapesSmallTrain0.22430.5029
NonInvasiveFetalECGThorax10.02040.4941
NonInvasiveFetalECGThorax20.01830.5045
OliveOil0.26670.5510
OSULeaf0.21490.5213
Phoneme0.02110.5020
PickupGestureWiimoteZ0.12000.5004
PigAirwayPressure0.01920.4898
PigArtPressure0.01920.4996
PigCVP0.01920.4978
PLAID0.06520.5016
Plane0.10480.5274
ProximalPhalanxOutlineAgeGroup0.39020.4332
ProximalPhalanxTW0.01950.3344
RefrigerationDevices0.35470.5161
Rock0.42000.5535
ScreenType0.33330.4908
SemgHandMovementCh20.15780.4778
SemgHandSubjectCh20.18220.5032
ShakeGestureWiimoteZ0.10000.4560
ShapesAll0.01670.5100
SmallKitchenAppliances0.32800.4814
SmoothSubspace0.90000.9815
StarLightCurves0.20840.5856
SwedishLeaf0.08000.5429
Symbols0.16780.5365
SyntheticControl0.81000.9978
Trace0.21000.5328
TwoPatterns0.13800.3224
UMD0.43060.6121
UWaveGestureLibraryAll0.13090.4791
UWaveGestureLibraryX0.12590.4971
UWaveGestureLibraryY0.12620.4968
UWaveGestureLibraryZ0.12650.4859
WordSynonyms0.05020.4875
Worms0.28570.4766

Table 8. UCR results.

DatasetACCAUC
ArticularyWordRecognition0.04000.4802
AtrialFibrillation0.33330.3567
BasicMotions0.30000.7671
CharacterTrajectories0.05990.5156
Epilepsy0.29710.6622
ERing0.16670.5620
FingerMovements0.44000.4138
Handwriting0.04120.4986
JapaneseVowels0.12430.4935
Libras0.06670.4950
LSST0.02430.4898
NATOPS0.12220.4552
PenDigits0.13350.5998
PhonemeSpectra0.03160.5160
RacketSports0.25660.4613
SpokenArabicDigits0.09910.5040
UWaveGestureLibrary0.12500.5217

Table 9. UEA results

References

  1. [1]Mohammad Al Ridhawi, Mahtab Haj Ali, and Hussein Al Osman. Daily OHLCV data for 20 S&P 500 equities and the CBOE volatility index (1982–2025), 2026. URL https://doi.org/10.5281/zenodo.20192822. Dataset.
  2. [2]Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2403.07815.
  3. [3]Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Pablo Guerrero, Tony Hu, Junming Yin, Nick Erickson, Prateek Mutalik Desai, Hao Wang, Huzefa Rangwala, George Karypis, Yuyang Wang, and Michael Bohlke-Schneider. Chronos-2: From univariate to universal forecasting, 2025. URL https://arxiv.org/abs/2510.15821.
  4. [4]Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018, 2018. URL https://arxiv.org/abs/1811.00075.
  5. [5]Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001. doi: 10.1023/A:1010933404324.DOI
  6. [6]Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. Association for Computing Machinery, 2016. doi: 10.1145/2939672.2939785.DOI
  7. [7]Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273–297, 1995. doi: 10.1007/BF00994018.DOI
  8. [8]Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The UCR time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6):1293–1305, 2019. doi: 10.1109/JAS.2019.1911747.DOI
  9. [9]Tobias Deuber, Lorenz Sparrenberg, and Rafet Sifa. Evaluating and benchmarking the system one model Jev, 2026. URL https://arxiv.org/abs/2609.37647.
  10. [10]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/.DOI
  11. [11]Jonathan St. B. T. Evans and Keith E. Stanovich. Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science, 8(3):223–241, 2013. doi: 10.1177/1745691612460685.DOI
  12. [12]Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96, 2019. doi: 10.1038/s41597-019-0103-9.DOI
  13. [13]Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey I. Webb, Irwin King, and Shirui Pan. A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10466–10485, 2024. doi: 10.1109/TPAMI.2024.3443141.DOI
  14. [14]Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/680b2a8135b9c71278a09cafb605869e-Abstract-Conference.html.arxiv.org/abs/2310.01728
  15. [15]Ming Jin, Yaxuan Kong, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiao-Li Li, Vincent S. Tseng, Yu Zheng, Lei Chen, Hui Xiong, Shirui Pan, and Qingsong Wen. Large models for time series and spatio-temporal data: A survey and outlook. ACM Computing Surveys, 58(14), 2026. doi: 10.1145/3821637.DOI
  16. [16]Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In Proceedings of the European Conference of the PHM Society, 2016. doi: 10.36001/phme.2016.v3i1.1577.DOI
  17. [17]Haoxin Liu, Zhiyuan Zhao, Shiduo Li, and B. Aditya Prakash. Evaluating system 1 vs. 2 reasoning approaches for zero-shot time series forecasting: A benchmark and insights, 2025. URL https://arxiv.org/abs/2503.01895.
  18. [18]Jigar Patel, Sahil Shah, Priyank Thakkar, and Ketan Kotecha. Predicting stock and stock price index movement using trend deterministic data preparation and machine learning techniques. Expert Systems with Applications, 42(1):259–268, 2015. doi: 10.1016/j.eswa.2014.07.040.DOI
  19. [19]Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th International Symposium on Wearable Computers, pages 108–109. IEEE, 2012. doi: 10.1109/ISWC.2012.13.DOI
  20. [20]Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter, 2019. URL https://arxiv.org/abs/1910.01108.
  21. [21]TypeSafe AI. Introducing system one models and Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026. Accessed 2026-10-21.
  22. [22]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.arxiv.org/abs/1706.03762
  23. [23]Yumo Xu and Shay B. Cohen. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970–1979. Association for Computational Linguistics, 2018. doi: 10.18653/v1/P18-1183.DOI
  24. [24]Siqiao Xue, Xiaoming Shi, Zhixuan Chu, Yan Wang, Hongyan Hao, Fan Zhou, Caigao Jiang, Chen Pan, James Y. Zhang, Qingsong Wen, Jun Zhou, and Hongyuan Mei. EasyTPP: Towards open benchmarking temporal point processes. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper/2024/hash/e66233a208ef32f56df6312263239fa0-Abstract-Conference.html.arxiv.org/abs/2307.08097
  25. [25]Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1, 2024. URL https://arxiv.org/abs/2407.06023.
  26. [26]Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):11106–11115, 2021. doi: 10.1609/aaai.v35i12.17325.DOI

Paper details

vlnc:W_ffybfrfx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents