betaFeedback

WorksComputer Science

DES-JEPA: Learning a Shared Vocabulary for High-Dimensional Discrete Event Sequences

Abstract

Temporal foundation models such as Chronos and TimesFM generalize across domains through a shared numerical observation space. In contrast, discrete event sequences (DESs), including operational event logs or patient clinical-event trajectories, have domain-specific event vocabularies whose labels lack a consistent correspondence across domains. Beyond this lack of consistency, the number of distinct events in real-world settings is high enough to challenge both classical temporal point process modeling and current in-context foundation inference models such as FIM-PP. Critically, current foundation models for DESs are limited to a small, fixed number of event types, restricting their applicability to realistic, high-dimensional event streams. To address this, we propose two complementary modules: (i) DES-JEPA, a joint-embedding predictive architecture that learns a shared discrete vocabulary from synthetic stochastic processes with unaligned event vocabularies; and (ii) DES-PFN, a discrete event sequence prior-fitted network that reuses the learned vocabulary for in-context next-event prediction. Without task-specific fine-tuning, this framework achieves competitive zero-shot performance on eight unseen real-world datasets and scales up to 700 distinct event types.

Introduction and Related Work

Discrete event sequences (DESs) arise ubiquitously across domains, from vehicle diagnostic logs [1] to electronic health records (EHRs) [2, 3, 4]. Learning across these domains requires handling event vocabularies whose labels have no consistent meaning across datasets: the same event type may denote a machine error in one system and a clinical observation in another. Concurrent work on temporal foundation models (TFMs) treats a sequence as a continuous-valued signal and pretrains a shared numerical value that lives on a comparable scale [5, 6]. However, categorical event identifiers provide no equivalent basis for such an aligned observation space.

Prior work [7, 8, 9, 10] leverages temporal point processes (TPPs) for modeling event types and their time of occurrence [11]. Conventional methods therefore rely on fitting expensive statistical models (e.g., Hawkes processes [7]) that are known to be impractical at hundreds or thousands of distinct event types. Importantly, they must be fitted from scratch for every new batch of data. Berghaus et al. [12] address this with a foundation inference model FIM-PP, enabling in-context learning of TPPs without fine-tuning. However, FIM-PP supports at most 22 event types, restricting its direct application to datasets with larger event vocabularies and substantially limiting its scope as a foundation model.

Pipeline overview showing DES-JEPA, DES-PFN, and forecasted query

Figure 1. Pipeline overview. (a): an event sequence is tokenized into a shared codebook Z\mathcal{Z} via DES-JEPA. DES-PFN (b) learns in-context to predict the next code of a query q<tq_{<t}. Right (c): a sequence of interest is forecasted, the resulting code is then decoded into the event types.

Therefore, extending current methodologies to more realistic real-world domains with hundreds of discrete event types remains an important open challenge.

We hypothesize that the missing component is a representation that transfers across processes, not a better fit to any single one. Therefore, the correct invariant to learn across anchor-free data-generating processes (DGPs) is therefore not the symbol itself, but alphabets of shared patterns that recur across unaligned processes. We gather this representation in a learned codebook, which is reusable across downstream tasks such as zero-shot next-event prediction or long-horizon forecasting [13].

To address this gap, we introduce a joint-embedding predictive architecture for discrete event sequences (DES-JEPA), a JEPA-based method [14, 15, 16] that learns a shared discrete vocabulary, i.e., codebook. It consists of two transformer-based encoders: (i) a teacher encoder which quantizes future windows into latent codes, and (ii) a student encoder which maps historical observations to a latent representation. Through a prior-fitted network (PFN)-style meta-pretraining (TabPFN, TabICLv2 [17, 18, 19, 20]) using synthetic data, the model learns to capture recurring dynamics across stochastic processes with unaligned vocabularies. Via the frozen teacher encoder and codebook embeddings, we tokenize any sequence of interest (query). Then, a second model (DES-PFN) predicts the next-event on this query, without fine-tuning, conditioned on a reference set of sequences (context). Together, they support zero-shot next-event prediction through a shared latent vocabulary, extending amortized in-context inference to discrete representations of temporal dynamics. Our contributions are:

  • We introduce DES-JEPA, combining predictive latent learning, vector quantization, and meta-pretraining to learn a shared codebook from any discrete event sequences.

  • We introduce DES-PFN, which uses the shared codebook computed by the teacher encoder of DES-JEPA, to autoregressively predict unseen event types without task-specific fine-tuning. We evaluate it on zero-shot next-event prediction on various unseen real-world datasets ranging from 3 to 700 event types, placing it as the best performing model for event sequences.

Methodology

Problem Statement. Let M\mathcal{M} denote a meta-distribution over a DGP, for instance a high-order Markov chain [21]. Each draw m∼Mm \sim\mathcal{M} instantiates a process {Xt}t≥0\{X_t\}_{t\geq0} over a finite, mm-specific alphabet Xm\mathcal{X}_m, which changes identity with every draw mm. Thus, no embedding table indexed by a local event type xi∈Xmx_i \in\mathcal{X}_m is shareable. Given any process, we aim to learn a shared representation that is invariant to event types xix_i that enables downstream tasks such as the modeling of P(xt∣x<t)P(x_t|x_{<t}) (Section 2.2).

DES-JEPA: A Shared Discrete Codebook

Following JEPA [14, 22], DES-JEPA consists of two encoding paths: (i) an encoder (student) fθf_{\theta} maps past observations to a representation h=fθ(x<t)h = f_{\theta}(x_{<t}), which are then mapped via a predictor gθg_{\theta} to logits z^=gθ(h)\hat{z} = g_{\theta}(h) over the KK codes of Z\mathcal{Z}, and (ii) a bidirectional encoder (teacher) fˉθ\bar{f}_{\theta}, a shallow prefix of fθf_{\theta} which is only updated by an exponential moving average (EMA) [15, 16], encodes the next raw events xt:t+w=(xt,…,xt+w)x_{t:t+w} = (x_t, \ldots, x_{t+w}) into e=fˉθ(xt:t+w)∈Rde = \bar{f}_{\theta}(x_{t:t+w}) \in\mathbb{R}^{d}, quantized against Z={c1,…,cK}\mathcal{Z} = \{c_1,\ldots,c_K\} by nearest-neighbor lookup as,

z=k∗(xt:t+w)=arg min⁡k∈{1,…,K} ∥e−ck∥22,where w≥1.(1)z = k^*(x_{t:t+w}) = \underset{k\in\{1,\ldots,K\}}{\operatorname{arg\,min}}\ \lVert e-c_k\rVert_2^2,\qquad\text{where } w\geq1. \tag*{(1)}

We keep w=2w=2 fixed. The student never observes xt:t+wx_{t:t+w} and predicts autoregressively z^\hat{z} purely from x<tx_{<t}, while the teacher is evaluated under stop-gradient sg⁡[⋅]\operatorname{sg}[\cdot] and receives no gradient at all. This prevents both encoders from collapsing to a constant map [15]. For a single trajectory x<tx_{<t} drawn from m∼Mm\sim\mathcal{M}, we train fθ,gθf_\theta,g_\theta by minimizing,

LJEPA=Em∼ME(x<t,xt:t+w)∼m[CE⁡Ξ(gθ(h),z)+∥sg⁡[e]−ck∥22+β∥e−sg⁡[ck]∥22],(2)\mathcal{L}_{\mathrm{JEPA}}=\mathbb{E}_{m\sim\mathcal{M}}\mathbb{E}_{(x_{<t},x_{t:t+w})\sim m}\left[\operatorname{CE}_{\Xi}\left(g_\theta(h),z\right)+\left\lVert\operatorname{sg}[e]-c_k\right\rVert_2^2+\beta\left\lVert e-\operatorname{sg}[c_k]\right\rVert_2^2\right], \tag*{(2)}

at every position tt. The main training signal is the cross-entropy loss, predicting the next latent code z^\hat{z} from history hh. A standard VQ-VAE codebook commitment terms [23] (β=0.25\beta=0.25) pulls ckc_k and ee toward each other. Since fθˉf_{\bar{\theta}} is trained only via EMA updates, the commitment term never actually back-propagates through fθˉf_{\bar{\theta}}. During inference, the future tokens zt:t+wz_{t:t+w} are replaced by the embedding of the token ztz_t to prevent any information leakage.

During training, sampling m∼Mm\sim\mathcal{M} realizes amortized Bayesian PFN-style meta-training [17, 18, 19]: the codebook acts as a finite statistic for mm’s identity, rather than representing mm directly (Appendix § 7). Naive VQ training is well known to collapse onto a small subset of indices. Thus, we actively monitor and enforce codebook utilization via a perplexity regularization and revive dead codes with a usage-EMA criterion. For additional details, please refer to Appendix § 8.

DES-PFN: In-Context Autoregressive Next-Event Prediction

Through the creation of codebook Z\mathcal{Z}, we created a common encoding of DESs. This encoding is leveraged to pretrain a model on next-event prediction tasks in order to perform downstream zero-shot next-event prediction on unseen data. To do so, we leverage in-context learning of event sequences [12], where for a sequence of interest q=((x0,t0),⋯ ,(xt,tt))q=((x_0,t_0),\cdots,(x_t,t_t)) (query) given a context C={S0,⋯ ,SC}\mathcal{C}=\{S_0,\cdots,S_C\}, both drawn from the same DGP mm, one performs the next-event prediction.

We freeze the codebook Z\mathcal{Z} and use it to tokenize each element in the raw sequence qq to form a sequence of event codes qz=(z1,…,zL)q_z=(z_1,\ldots,z_L) via the teacher path of Eq. (1). We train DES-PFN on tokenized queries qzq_z and context CzC_z generated from the same synthetic DGPs meta-distributions (M\mathcal{M}) used in the training of DES-JEPA. Given these inputs, DES-PFN predicts the next token q^z,t\hat{q}_{z,t} as,

q^z,t=arg max⁡pϕ(qz,t∣qz,<t,Cz),wherepϕ=softmax⁡(hϕ(qz,<t,Cz)).(3)\hat{q}_{z,t}=\operatorname*{arg\,max} p_\phi(q_{z,t}\mid q_{z,<t},C_z),\qquad\text{where}\quad p_\phi=\operatorname{softmax}(h_\phi(q_{z,<t},C_z)). \tag*{(3)}

At inference, the frozen DES-JEPA teacher maps the context C\mathcal{C} and query history to discrete codes. DES-PFN then forecasts autoregressively in this shared code space, conditioning on the encoded inputs and previously predicted codes. This representation enables zero-shot transfer across DGPs with disjoint event vocabularies, without updating either model. In the experiments, DES-JEPA and DES-PFN are pretrained on six DGPs with a maximum codebook capacity of K=768K=768 and contain 60 million and 40 million parameters, respectively. Architectural details are provided in Appendix § 6.

Decoding Codes to Event Types. DES-PFN forecasts a tokenized code z^=q^z,t\hat{z}=\hat{q}_{z,t}, whereas the actually desired event prediction is required in its raw alphabet Xm\mathcal{X}_m. Since event labels are unaligned across processes and quantization is a many-to-one projection and therefore non-invertible, we estimate a context-specific token from the tokenized pairs {(xi,zi)}i=1C\{(x_i,z_i)\}_{i=1}^{C} in C\mathcal{C}. We use a Nadaraya-Watson estimator [24, 25] for candidate vv,

p^(x=v∣z^)=∑i=1CI[xi=v] κτ(ez^,ezi)∑i=1Cκτ(ez^,ezi),(4)\hat{p}(x=v\mid\hat{z})=\frac{\sum_{i=1}^{C}\mathbb{I}[x_i=v]\,\kappa_\tau(e_{\hat{z}},e_{z_i})}{\sum_{i=1}^{C}\kappa_\tau(e_{\hat{z}},e_{z_i})}, \tag*{(4)}

where κτ\kappa_\tau is a nonnegative cosine-based kernel with temperature τ\tau calibrated per context and v∈Xmv\in\mathcal{X}_m a candidate event type. The predicted code z^\hat{z} queries each context occurrence’s code ziz_i, which casts a similarity-weighted vote for its identity xix_i, allowing identities associated with nearby codes to be recovered without an exact match to z^\hat{z} in C\mathcal{C} (Figure 2).

Kernel-decode inverse mapping and encode/decode round trip on StackOverflow and Vehicle DTC sequences

Figure 2. Top: Kernel-decode inverse mapping on real StackOverflow and Vehicle DTC contexts, generated directly from the trained checkpoint (no illustrative numbers). Bottom row: the encode/decode round trip on ONE real sequence over cumulative time: raw ids, the codes DES-JEPA tokenizes them into, and the raw ids recovered by decoding those codes back. The color coding matches the top row, so a mismatch is directly visible as a color change.

Experiments

Baselines, Datasets & Experimental Setup. We evaluate DES-PFN against FIM-PP [12] and three gradient-trained neural TPP baselines, RMTPP [26], NHP [27], and SAHP [28], for next-event prediction tasks. We benchmark the model performance on eight real event-type datasets: Twitter retweet cascades, StackOverflow badge awards, Taobao and Amazon user actions [29], HDFS and BGL system logs [30, 31, 32], MIMIC-IV hospital lab-test sequences [3] and a vehicle event logs [1]. Every zero-/few-shot method, i.e., DES-PFN, FIM-PP and each neural baseline’s few-shot variant (refit from scratch on only the in-context set), sees the identical ncontext=128n_{\mathrm{context}}=128 example sequences. DES-PFN and FIM-PP receive no gradient updates. We report the validation top-1 accuracy for each fully-trained neural baseline variant trained for 5 epochs, as a reference upper bound.

Results. Table 1 reports DES-PFN’s zero-shot (no-gradient-update) next-event prediction accuracy across eight real datasets spanning three to seven hundred event types. Fully-trained specialists gradient-trained on each dataset’s entire training split still outperform DES-JEPA on the four lower-cardinality easytpp datasets. However, this specialism does not generalize well, since each needs a real held-out test split and per-dataset retraining from scratch to be accurate. In low cardinality, FIM-PP, which shares DES-PFN’s zero-shot, in-context inference setting has competitive performance beating DES-PFN in three out of five datasest. Importantly, its fixed 22-event type architecture precludes evaluation on BGL, MIMIC-IV, and Vehicle. Using the single frozen codebook, DES-PFN supports larger event vocabularies and is the only method with reported results on these datasets; all gradient-trained baselines either lack a usable data split or exceed the prescribed training-time budget. DES-PFN’s weakest result is on MIMIC-IV: we reason that the shared blood-draw times tampes across 95% of consecutive lab events may leave DES-PFN little time information to exploit.

Table 1: Real-world next-event-type accuracy (Top-1, mean±std averaged over 5 runs) for ncontext=128n_{\mathrm{context}}=128, nquery=20n_{\mathrm{query}}=20 and L=32L=32. Baselines are reported in two regimes: few-shot (refit from scratch on the same 128 context sequences) and full (fully-trained). For HDFS/[30, 31]/MIMIC-IV, since no real train/test split is available, the ‘full’ column instead reports a §-marked local train/test split (trained once on every earlier sequence actually available, evaluated on the same held-out query set); ‘few-shot’ stays “–” for these three (not run). FIM-PP is capped at ∣X∣≤22|\mathcal{X}| \le22 marks (“–”). Bold/underline mark the best/second-best score per row. Vehicle is reported at two vocabulary caps (∣X∣=500|\mathcal{X}|=500 and ∣X∣=700|\mathcal{X}|=700), † marks a method that genuinely exceeded the 600s time budget.

Dataset∣X∣|\mathcal{X}|RMTPP [26] few-shotRMTPP [26] fullNHP [27] few-shotNHP [27] fullSAHP [28] few-shotSAHP [28] fullFIM-PP [12]DES-PFN (ours)
retweet [29]30.464±0.0210.464\pm0.0210.5330.5330.235±0.0110.235\pm0.0110.5910.5910.531±0.0200.531\pm0.0200.4710.4710.462±0.0220.462\pm0.0220.520±0.0230.520\pm0.023
stackoverflow [29]220.009±0.0030.009\pm0.0030.4110.4110.007±0.0020.007\pm0.0020.4200.4200.413±0.0350.413\pm0.0350.4200.4200.441±0.0320.441\pm0.0320.356±0.0300.356\pm0.030
taobao [29]170.227±0.0490.227\pm0.0490.5590.5590.490±0.0490.490\pm0.0490.5220.5220.433±0.0410.433\pm0.0410.4650.4650.516±0.0400.516\pm0.0400.420±0.0570.420\pm0.057
amazon [29]160.151±0.0280.151\pm0.0280.2990.2990.089±0.0060.089\pm0.0060.3190.3190.193±0.0310.193\pm0.0310.2540.2540.207±0.0410.207\pm0.0410.241±0.0350.241\pm0.035
HDFS [30, 31]14–0.074§0.074^{\S}–0.063§0.063^{\S}–0.103§0.103^{\S}0.266±0.0180.266\pm0.0180.200±0.0280.200\pm0.028
BGL [30, 32]120–0.000§0.000^{\S}–0.000§0.000^{\S}–0.063§0.063^{\S}–0.136±0.0200.136\pm0.020
MIMIC-IV [3]441–0.000§0.000^{\S}–0.004§0.004^{\S}–0.002§0.002^{\S}–0.029±0.0100.029\pm0.010
Vehicle [1]500–†–†–†–0.162±0.0140.162\pm0.014
Vehicle [1]700–†–†–†–0.094±0.0090.094\pm0.009

Table 1.

Conclusion

We introduced a framework to model high-dimensional event sequences and enable zero-shot next-event prediction. The first model, DES-JEPA pretrains a single, shared codebook over a meta-distribution of synthetic DGPs, giving arbitrary, anchor-free event sequences that share no vocabulary or numeric scale, a common representation. By applying this frozen codebook, DES-PFN is able to perform zero-shot next-event prediction on eight real-world dataset spanning across three orders of magnitude in cardinality, from K=3K=3 to K=700K=700. This approach remains competitive at low cardinality, and remains the only reliable method producing a non-trivial number once cardinality grows to hundreds of event types. Two limitations motivate future work: (i) the codebook capacity is fixed during pretraining, and currently lacks rate-distortion experiments (ii) evaluation is restricted to one-step-ahead prediction. Long-horizon autoregressive rollouts remain untested, particularly regarding error accumulation and the ability to represent multiple plausible futures beyond a single point forecast.

Limitations.

Two limitations point to natural extensions. The codebook size is fixed at pretraining time rather than adapted to a downstream dataset’s true cardinality. Moreover, every result in this paper is one-step-ahead: DES-PFN has not been evaluated on genuine long-horizon, multi-step rollout, where a single point forecast poorly represents the many plausible futures a real event stream can take and small early errors can compound over many steps. We see these last two limitations as connected: the natural next step is to replace the single-code predictor with a multi-head decoder trained under a winner-takes-all, multiple-choice-learning objective [33], letting DES-JEPA and DES-PFN output several plausible future codes per query instead of one point forecast, precisely the mechanism long-horizon rollout needs to stay calibrated as uncertainty compounds step by step.

Architecture Detail

Both DES-JEPA and DES-PFN were pretrained on a single NVIDIA RTX 2080 Ti, approximately one day each. All experimental results were measured at inference time on an Apple M2 Pro CPU, with no GPU acceleration.

DES-JEPA

Embeddings

Every event is embedded from purely statistics of its own occurrence history: self-information/surprisal, running (global and local-window) frequency, whether the type has been seen before, and recency and dispersion of its own inter-arrival gaps, together with event-count (“step”) analogues of the timing features that need no real timestamp. None of these is computed from a learned lookup table keyed on the raw event id, since a raw id carries no stable meaning across draws of m∼Mm \sim\mathcal{M} (§ 2): therefore event identity is represented purely through these statistics. We attach no additional per-event covariate beyond type and time (no “marks” in the temporal-point-process sense) to any data-generating process.

Figure 3 shows the pairwise correlation of these features on a representative synthetic batch: highly correlated pairs (e.g. causal_freq/local_freq, mean_dt_type/mean_step_gap_type) are deliberately kept rather than pruned, since their real-valued/step-valued counterparts remain the only signal available whenever timestamps are missing or unreliable.

JEPAEventEmbedding feature correlation matrix

Figure 3. Pairwise Pearson correlation across the 14 causal statistical features feeding the DES-JEPA event embedding, computed on a mixed-DGP synthetic batch.

Transformer Block

Student Encoder. A causal Transformer over the whole past, with sliding-window attention (window w=32w=32: token tt attends only to the 32 preceding tokens) and RoPE [34] position encoding.

Teacher Encoder. A shallow, bidirectional prefix of the student stack (2 of the student’s 6 layers in the checkpoint used throughout this paper), updated only by an EMA of the student’s own weights (τ=0.999\tau=0.999) (§ 2.1); it encodes the next W=2W=2 raw events as one coherent unit rather than causally within that short window.

Vector Quantizer. The teacher’s continuous output is matched to the nearest of KK codebook vectors (Eq. (1)) and trained with the combined cross-entropy/VQ-VAE objective of Eq. (2). Appendix D details the perplexity regularizer and dead-code revival mechanism that keep this codebook from collapsing into a dozen of codes.

DES-PFN

Embeddings

DES-PFN’s event-code embedding table is the frozen DES-JEPA codebook itself (§ 2.2), followed by a learned linear projection whenever the two models’ hidden sizes differ. Inter-arrival time uses a Chronos-style [5] discretized embedding: log⁡(1+Δt/scale)\log(1 + \Delta t/\mathrm{scale}) is binned into a fixed vocabulary and looked up, turning next-time prediction into classification rather than continuous regression. An optional duplicate-count embedding additionally marks immediate event repeats.

Transformer Block

DES-PFN alternates two attention axes, at every layer, over the (sequence, time, hidden) grid formed by the reference context and the query.

Causal Attention. A standard causal Transformer block over the temporal axis, applied independently to every sequence (context and query alike), with RoPE and a sliding-window band (32 in the checkpoint used throughout this paper).

Col-Wise Attention. Following the column-then-row, induced-attention design recently used by tabular foundation models such as TabICLv2 [20] and the inducing-point mechanism of Set Transformer [35], cross-sequence attention pools information through a small, fixed set of ninducing=32n_{\mathrm{inducing}}=32 learnable vectors rather than full O(n2)O(n^2) attention across reference sequences.

Pretraining

Prior-Fitted Network and Synthetic Data Generation

Both stages are pretrained exclusively on synthetic data drawn from a meta-distribution M\mathcal{M} over six data-generating process (DGP) families, each implemented in synthetic_data/. No real-world data is used at either pretraining stage; every real-world result in § 3 is zero-shot transfer.

Every family is instantiated fresh per task (all parameters below redrawn) and draws its own active event-type count ∣X∣\lvert\mathcal{X} \rvert (the curriculum’s nactiven_{\mathrm{active}}, denoted this way to avoid clashing with the codebook size KK) uniformly from a shared range, detailed at the end of this section.

Markov (rr-th order chain). Order r∼Unif⁡{1,…,5}r \sim\operatorname{Unif}\{1,\ldots,5\} is drawn once per task – not fixed at 1: the next type follows

P(Xt=k∣Xt−r:t−1)=softmax⁡k(bk+∑ℓ=1re−ℓW[Xt−ℓ,k]+fNL(Xt−r:t−1)k),(5)P(X_t = k \mid X_{t-r:t-1}) = \operatorname{softmax}_k\left(b_k + \sum_{\ell=1}^{r} e^{-\ell}W[X_{t-\ell}, k] + f_{\mathrm{NL}}(X_{t-r:t-1})_k\right), \tag*{(5)}

with bias b∈R∣X∣b \in\mathbb{R}^{\lvert\mathcal{X} \rvert}, sparse mixed-sign interaction matrix W∈R∣X∣×∣X∣W \in\mathbb{R}^{\lvert\mathcal{X} \rvert\times\lvert\mathcal{X} \rvert} (density 1−sparsity1-\mathrm{sparsity}, sparsity∼U(0.3,0.8)\mathrm{sparsity} \sim U(0.3, 0.8); Wj,k>0W_{j,k}>0 excitatory, Wj,k<0W_{j,k}<0 inhibitory), and e−ℓe^{-\ell} an exponential recency decay over the rr lags. With probability 0.5 per task, a two-layer MLP term fNLf_{\mathrm{NL}} (random fixed embedding, one hidden layer, a randomly chosen activation) adds non-additive lag interactions; otherwise fNL≡0f_{\mathrm{NL}}\equiv0. Holding times are Weibull with a history-modulated scale: Δt∣Xt=k∼Weibull⁡(shapek,scalet)\Delta t \mid X_t=k \sim\operatorname{Weibull}(\mathrm{shape}_k,\mathrm{scale}_t), scalet=softplus⁡(sk+∑ℓ=1re−ℓ/2w[Xt−ℓ]+ϵ)\mathrm{scale}_t = \operatorname{softplus}(s_k+\sum_{\ell=1}^{r}e^{-\ell/2}w[X_{t-\ell}]+\epsilon), shapek=softplus⁡(gk)+0.1\mathrm{shape}_k = \operatorname{softplus}(g_k)+0.1, where sks_k, ww, gkg_k are fixed-at-construction (not trained) per-task parameters; ww depends only on the preceding types, not on the candidate kk.

Hawkes (exact Ogata [7] thinning). A multivariate self-exciting process with conditional intensity

λk(t)=μk+∑j=1∣X∣∑i:tij<tAj,kκ(t−tij),(6)\lambda_k(t) = \mu_k + \sum_{j=1}^{\lvert\mathcal{X} \rvert}\sum_{i:t_i^j<t} A_{j,k}\kappa(t-t_i^j), \tag*{(6)}

baseline μk∼LogUniform⁡(0.05,0.5)\mu_k \sim\operatorname{LogUniform}(0.05, 0.5); excitation A∈R≥0∣X∣×∣X∣A \in\mathbb{R}_{\geq0}^{\lvert\mathcal{X} \rvert\times\lvert\mathcal{X} \rvert} built sparse (each row excites 1–3 other types, sparsity∼U(0.2,0.5)\mathrm{sparsity}\sim U(0.2, 0.5), weights ∼Exp⁡(1)\sim\operatorname{Exp}(1)) plus a diagonal Ak,k=(∑jAk,j)ukA_{k,k} = (\sum_j A_{k,j})u_k, uk∼U(0.2,0.5)u_k \sim U(0.2, 0.5), globally rescaled so the spectral radius ρ(A)∼U(0.7,0.9)\rho(A) \sim U(0.7, 0.9) (stationarity, ρ(A)<1\rho(A)<1). The kernel κ\kappa is, with probability 0.5 each: exponential κ(Δ)=βe−βΔ\kappa(\Delta) = \beta e^{-\beta\Delta}, β∼U(1,5)\beta\sim U(1,5) (short-range, Markovian); or power-law/Omori–Utsu κ(Δ)=αcα(Δ+c)−(1+α)\kappa(\Delta) = \alpha c^\alpha(\Delta+c)^{-(1+\alpha)}, α∼U(0.3,1.2)\alpha\sim U(0.3,1.2), c∼U(0.05,0.5)c \sim U(0.05,0.5) (long-range, non-Markovian).

Neural Autoregressive Models A randomly-initialized, never-trained LSTM: ht=LSTM⁡(e(Xt),ht−1)h_t = \operatorname{LSTM}(e(X_t),h_{t-1}), Xt+1∼Categorical⁡(softmax⁡(Wtokht))X_{t+1} \sim\operatorname{Categorical}(\operatorname{softmax}(W_{\mathrm{tok}}h_t)), Δtt+1∼Weibull⁡(softplus⁡(wshapeht),softplus⁡(wscaleht))\Delta t_{t+1} \sim\operatorname{Weibull}(\operatorname{softplus}(w_{\mathrm{shape}}h_t),\operatorname{softplus}(w_{\mathrm{scale}}h_t)), where token and timing heads read from the SAME hth_t (fully joint dynamics) and ee is a random embedding table. Hidden size ∼U{16,…,64}\sim U\{16,\ldots,64\}, depth ∼U{1,2,3}\sim U\{1,2,3\} layers, embedding dimension ∼U{8,…,32}\sim U\{8,\ldots,32\}, all weights i.i.d. N(0,σ2)\mathcal{N}(0,\sigma^2), σ∼U(0.5,2.0)\sigma\sim U(0.5,2.0) – freshly redrawn every task.

Hidden Markov Model. S∼U{3,…,7}S \sim U\{3,\ldots,7\} latent states, zt+1∣zt∼Categorical⁡(Tzt,:)z_{t+1} \mid z_t \sim\operatorname{Categorical}(T_{z_t,:}); TT is sticky, with the escape probability (not the sticky value itself) drawn log-uniformly, 1−pstick∼LogUniform⁡(0.01,0.3)1-p_{\mathrm{stick}} \sim\operatorname{LogUniform}(0.01,0.3) (i.e. pstick∈[0.7,0.99]p_{\mathrm{stick}} \in[0.7,0.99]) – avoiding oversampling short dwell times, since dwell =1/(1−pstick)=1/(1-p_{\mathrm{stick}}) is highly nonlinear in pstickp_{\mathrm{stick}}. Emission Xt∣zt∼Categorical⁡(Ezt,:)X_t \mid z_t \sim\operatorname{Categorical}(E_{z_t,:}), Ek,:∼Dirichlet⁡(α1∣X∣)E_{k,:} \sim\operatorname{Dirichlet}(\alpha\mathbf{1}_{\lvert\mathcal{X} \rvert}), α∼U(0.2,0.5)\alpha\sim U(0.2,0.5) (small α\alpha – each state emits a small, peaked subset of types); timing Δtt∣zt∼Weibull⁡(shapezt,scalezt)\Delta t_t \mid z_t \sim\operatorname{Weibull}(\mathrm{shape}_{z_t},\mathrm{scale}_{z_t}), shapek∼U(0.8,2.5)\mathrm{shape}_k \sim U(0.8,2.5), scalek∼LogUniform⁡(0.3,3)\mathrm{scale}_k \sim\operatorname{LogUniform}(0.3,3) – the same latent state drives both what happens and when.

Burst. An i.i.d. renewal timing process, Δtt∼Weibull⁡(shape,scale)\Delta t_t \sim\operatorname{Weibull}(\mathrm{shape},\mathrm{scale}), shape∼U(0.15,0.7)<1\mathrm{shape} \sim U(0.15,0.7)<1 (decreasing hazard, genuinely bursty), scale∼U(0.2,1.0)\mathrm{scale} \sim U(0.2,1.0), both fixed for the whole task and independent of type/history – combined with a first-order sticky-categorical type chain: Xt=Xt−1X_t=X_{t-1} with probability pstick∼U(0,0.9)p_{\mathrm{stick}} \sim U(0,0.9), else Xt∼Categorical⁡(p)X_t \sim\operatorname{Categorical}(p) resampled, p∼Dirichlet⁡(α1∣X∣)p \sim\operatorname{Dirichlet}(\alpha\mathbf{1}_{\lvert\mathcal{X} \rvert}), α∼U(0.3,1.5)\alpha\sim U(0.3,1.5). Models verbatim repeat-heavy streams (keep-alives, retries, repeated clicks).

Template. A shared directed graph next: {1,…,∣X∣}−{1,…,∣X∣}\{1,\ldots,\lvert\mathcal{X}\rvert\} - \{1,\ldots,\lvert\mathcal{X}\rvert\} (at most one outgoing edge per node) is built from short chains, length ∼U{3,…,8}\sim U\{3,\ldots,8\}; a latent walk deterministically follows zt+1=next⁡(zt)z_{t+1}=\operatorname{next}(z_t) whenever defined, otherwise restarts at a start node chosen per-template weights ∼Dirichlet⁡(α1)\sim\operatorname{Dirichlet}(\alpha\mathbf{1}), α∼U(3,8)\alpha\sim U(3,8). The emitted, noisy type mixes a point mass on the latent node with a uniform background,

P(Xt=v∣zt)=(1−ϵ)I[v=zt]+ϵ/∣X∣,ϵ∼U(0,0.1),(7)P(X_t = v \mid z_t) = (1-\epsilon)\mathbb{I}[v=z_t] + \epsilon/\lvert\mathcal{X}\rvert,\qquad\epsilon\sim U(0,0.1), \tag*{(7)}

so every modeled edge (j,next⁡(j))(j,\operatorname{next}(j)) has expected causal strength 1−ϵ1-\epsilon – this is what gives event-type pairs a near-deterministic causal coupling that Hawkes-style self-excitation (which only boosts future rate, never identity) cannot express. Timing is Δt∼Weibull⁡(1,st)\Delta t \sim\operatorname{Weibull}(1,s_t) (i.e. Exponential), with scale st=sslow∼U(0.5,3)s_t=s_{\mathrm{slow}}\sim U(0.5,3) right after a template restart and st=sfast∼U(0.05,0.3)s_t=s_{\mathrm{fast}}\sim U(0.05,0.3) while continuing along an established edge.

Sampling Parameters During Pretraining. DES-JEPA (Stage 1) samples a single trajectory per task, from all six DGPs with TEMPLATE and HAWKES up-weighted to 30% and 15% probability respectively (the remaining four share the rest evenly, 13.75% each), a fixed sequence length of 128, and event-type cardinality nactiven_{\mathrm{active}} drawn log-uniformly from [3,1024][3,1024] per task. DES-PFN (Stage 2) instead samples both a reference context CC and a query sequence per task, following a TabICLv2-style [20] curriculum that couples context size and sequence length under a fixed per-batch token budget (ncontext×length⁡≈const⁡n_{\mathrm{context}}\times\operatorname{length}\approx\operatorname{const}): ncontext∈[16,256]n_{\mathrm{context}}\in[16,256] and sequence length ℓ∈[16,128]\ell\in[16,128] vary anticorrelated by construction, so a batch of many short reference sequences and a batch of few long ones cost roughly the same compute; nactive∈[3,1024]n_{\mathrm{active}}\in[3,1024] as in Stage 1.

Tokenizing an observed sequence at inference (the last w−1w-1 positions). During training, the teacher only ever supervises positions with a genuine, fully-observed future window: for a length-LL training sequence, positions 0,…,L−W−10,\ldots,L-W-1 each have a real xt:t+Wx_{t:t+W} target, and the remaining WW tail positions are simply dropped from the loss. At inference, however, DES-PFN needs one code per position for the entire query, including its most recent, currently-last observed event, but that position’s own future (xt+1)(x_{t+1}) is exactly what is being forecast and does not exist yet. The tokenization made by DES-JEPA resolves this by computing genuine windows for every position that has one (positions 0,…,L−W0,\ldots,L-W) and, for the remaining W−1W-1 tail position(s) only, substituting a degenerate window built by repeating the last observed embedding WW times (emb[:, -1:].expand(...)) in place of the unavailable future. With W=2W=2 (the value used throughout this paper) this affects only a single position per tokenized sequence: the code assigned to the most recently observed event is computed from this repeated-embedding fallback rather than a genuine future window, for every context sequence and for the query alike. This is a real, deliberate train/inference asymmetry (never exercised during training, since the tail positions are excluded from the loss there) rather than a bug, but it does mean the single most recent position’s code is systematically less informative than every earlier position’s.

Additional Analysis

Codebook Perplexity. A universal codebook shared across heterogeneous DGPs is only useful if its capacity is actually exploited; naive VQ training is well known to collapse onto a small subset of indices (e.g., DES-JEPA using 10 codes at most instead of the KK available ones). We track the empirical code-usage distribution over a batch of BB target windows, p^k=1B∑i=1B1[k∗(yi)=k]\hat{p}_k=\frac{1}{B}\sum_{i=1}^{B}\mathbb{1}[k^*(y_i)=k], and define

Definition 1 (Codebook Perplexity)

PPL⁡(Z)=exp⁡(−∑k=1Kp^klog⁡p^k).(8)\operatorname{PPL}(\mathcal{Z})=\exp\left(-\sum_{k=1}^{K}\hat{p}_k\log\hat{p}_k\right). \tag*{(8)}

PPL⁡(Z)∈[1,K]\operatorname{PPL}(\mathcal{Z})\in[1,K], with PPL⁡=K\operatorname{PPL}=K under uniform code usage and PPL⁡→1\operatorname{PPL}\to1 under index collapse onto a single code. We monitor PPL per-DGP-family and aggregated across M\mathcal{M}: a universal codebook whose perplexity remains high within every family but whose per-family code supports remain distinguishable across families is precisely the signature of a shared dictionary that is neither collapsed nor merely memorizing family identity. Aggregated across the full training mixture, K=768K=768 codes give a final perplexity of ≈500\approx500. Figure 4 breaks this down per DGP family, measured directly from the trained checkpoint on 800 fresh synthetic tasks per family (nactive∈[3,300]n_{\mathrm{active}}\in[3,300]): perplexity is high and comparable across all six families (601–716 of the 768 available codes actually used, no family collapsing onto a handful or hogging most of the codebook), supporting the "shared, non-collapsed dictionary" reading above. The decode-accuracy panel (true code →\to raw id via the same per-context majority-vote decoder used for real-world data, § 2.2) is uniformly modest (2.5–12.7%) across every family: with up to 300 active raw ids sharing only 768 codes total across six families. Exact code identity alone cannot always pin down a unique raw id (the same codebook-capacity/collision cost documented for real-world high-cardinality datasets in the experiment section. An high perplexity number indicates the codebook is being used broadly and non-degenerately, not that every individual code is uniquely decodable.

Per-DGP codebook usage: no single family dominates or collapses the shared codebook

Figure 4. Per-DGP codebook Perplexity (left) and round-trip true-code-to-raw-id decode accuracy (right), measured directly from the trained checkpoint used throughout this paper (no retraining), 800 fresh synthetic tasks per DGP family, ncontext=32n_{\mathrm{context}}=32.

Dead-code revival. A perplexity regularizer only discourages collapse in aggregate; it does not repair codes that have already drifted out of use. Following the EMA-cluster-size restarts of VQ-VAE-2 [36] and its scaled-up variant in Jukebox [37], we track an exponential moving average of each code’s usage share,

pˉk(t)←μpˉk(t−1)+(1−μ)p^k(t),pˉk(0)=1K,(9)\bar{p}_k^{(t)} \leftarrow\mu\bar{p}_k^{(t-1)} + (1-\mu)\hat{p}_k^{(t)}, \qquad\bar{p}_k^{(0)} = \frac{1}{K}, \tag*{(9)}

and flag code kk as underused whenever pˉk(t)\bar{p}_k^{(t)} falls below a fraction ϑ\vartheta of its fair share 1/K1/K. A per-code counter dkd_k accumulates consecutive underused steps and resets to zero the moment kk recovers. The code kk becomes reset-eligible once dk≥Tpatd_k \ge T_{\mathrm{pat}}. Unlike a hard, per-step presence test, pˉk(t)\bar{p}_k^{(t)} is barely perturbed by a single rare assignment, so a code that only survives on stray, noise-level picks is still correctly identified as dead. All simultaneously reset-eligible codes are re-seeded in one batched event, from encoder outputs zz of the current batch, once the reset-eligible fraction exceeds a threshold ρ\rho,

1K∑k=1K1[dk≥Tpat]≥ρ.(10)\frac{1}{K}\sum_{k=1}^{K}\mathbb{1}[d_k \ge T_{\mathrm{pat}}] \ge\rho. \tag*{(10)}

Relative to the per-code restarts of prior work, our two departures are: (i) a batched trigger that rescues a whole backlog of dead codes in a single event rather than continuously reviving them one at a time, and (ii) a usage-share-relative threshold, rather than an absolute cluster-size count, which keeps the criterion scale-invariant as B×LB \times L grows far beyond KK.

Related Work

Transfer across temporal processes. Continuous TFMs such as Chronos, TimesFM, and UniTok transfer forecasts through a shared numerical observation space [5, 6, 38]. PFNs, Foundation inference models (FIMs), MetaTPP and LaT-PFN instead amortize inference across tasks or stochastic processes, but transfer continuous forecasts, intensity functions, rate matrices, or task representations rather than a common discrete vocabulary [17, 12, 39, 40]. Models for DESs, including CoLES, MOTOR, and PinFM, learn reusable representations within domains whose event schemas remain stable [41, 42, 43].

Event-sequence forecasting beyond a single stochastic-process family. A separate line of work adapts a single event-sequence model to changing data without our cross-alphabet, zero-shot goal. LAMP couples a TPP with a large language model performing few-shot abductive reasoning over retrieved historical events to improve mark prediction [44]; HYPRO reweights an autoregressive TPP’s proposals with a learned energy function for long-horizon, multi-step rollout [45]; PromptTPP retrieves a continuous-time prompt pool to continually adapt a TPP to streaming, non-stationary event data [46]. These target complementary problems: reasoning-augmented prediction, long-horizon calibration, and continual adaptation, respectively, within one dataset’s fixed vocabulary, rather than a codebook shared, without any fine-tuning, across datasets whose event identities are mutually meaningless; we do not compare against them directly for this reason, but HYPRO in particular addresses the long-horizon rollout setting we flag as future work (Appendix § 4).

Vocabulary symmetry and predictive discretization. EquiTabPFN and lexinvariant language models address target-label and vocabulary permutations, respectively, but neither learns a shared predictive codebook across stochastic-process families [47, 48]. Vector-quantized and predictive self-supervised methods learn discrete or latent representations whose alignment is inherited from a common visual or acoustic space [23, 49, 15, 22]; predictive-state, causal-state, and information-bottleneck approaches instead motivate grouping histories by their future behavior [50, 51, 52]. DES-JEPA connects these directions by learning a shared discrete prediction space across processes whose native event alphabets are unaligned.

References

  1. [1]Hugo Math, Rainer Lienhart, and Robin Schön. Harnessing event sensory data for error pattern prediction in vehicles: A language model approach. Proceedings of the AAAI Conference on Artificial Intelligence, 39(18):19423–19431, 4 2025.
  2. [2]Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digital Medicine, 4:86, 2021.DOI
  3. [3]Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. Mimic-iv, a freely accessible electronic health record dataset. Scientific Data, 10(1):1, 2023.DOI
  4. [4]Alex Labach, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. Duett: Dual event time transformer for electronic health records. In Kaivalya Deshpande, Madalina Fiterau, Shalmali Joshi, Zachary Lipton, Rajesh Ranganath, Íñigo Urteaga, and Serene Yeung, editors, Proceedings of the 8th Machine Learning for Healthcare Conference, volume 219 of Proceedings of Machine Learning Research, pages 403–422. PMLR, 8 2023.arxiv.org/abs/2304.13017
  5. [5]Abdul Fatir Ansari, Lorenzo Stella, Caner Türkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, et al. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2403.07815
  6. [6]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10148–10167, 2024.arxiv.org/abs/2310.10688
  7. [7]Yoshihiko Ogata et al. Statistical studies of the occurrence of small earthquakes: I. A point process model for earthquake sequences. Annals of the Institute of Statistical Mathematics, 30(1):379–395, 1978.
  8. [8]Hongyuan Mei and Jason Eisner. The neural Hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, volume 30, 2017.
  9. [9]Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer Hawkes process. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11692–11702. PMLR, 7 2020.arxiv.org/abs/2002.09291
  10. [10]Emmanuel Bacry, Martin Bompaire, Stéphane Gaffas, and Jean-Francois Muzy. Sparse and low-rank multivariate hawkes processes. Journal of Machine Learning Research, 21(50):1–32, 2020.
  11. [11]Patrick J. Laub, Young Lee, Philip K. Pollett, and Thomas Taimre. Hawkes models and their applications. Annual Review of Statistics and Its Application, 12(Volume 12, 2025):233–258, 2025.arxiv.org/abs/2405.10527
  12. [12]David Berghaus, Patrick Seifner, Kostadin Cvejkoski, Cesar Ojeda, and Ramses J Sanchez. In-context learning of temporal point processes with foundation inference models. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2509.24762
  13. [13]Xiao Shou. Unified flow matching for long horizon event forecasting, 2025.arxiv.org/abs/2508.04843
  14. [14]Yann LeCun. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. 2022.
  15. [15]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael G. Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, pages 15619–15629, 2023.arxiv.org/abs/2301.08243
  16. [16]Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-jepa 2: Self-supervised video models enable understanding, prediction and planning, 2025.arxiv.org/abs/2506.09985
  17. [17]Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. International Conference on Learning Representations, 2022.arxiv.org/abs/2112.10510
  18. [18]Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. International Conference on Learning Representations, 2023.arxiv.org/abs/2207.01848
  19. [19]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637:319–326, 2025.DOI
  20. [20]Jingang QU, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2602.11139
  21. [21]D. Koller and N. Friedman. Probabilistic Graphical Models: Principles and Techniques. Adaptive computation and machine learning. MIT Press, 2009.
  22. [22]Junyeob Baek, Hosung Lee, Christopher Hoang, Mengye Ren, and Sungjin Ahn. Discrete JEPA: Learning discrete token representations without reconstruction. In ICML 2025 Workshop on Tokenization, volume 267 of Proceedings of Machine Learning Research, 2025.arxiv.org/abs/2506.14373
  23. [23]Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1711.00937
  24. [24]E. Nadaraya. On estimating regression. Theory of Probability and Its Applications, 9:141–142, 1964.DOI
  25. [25]Geoffrey S. Watson. Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, 26(4):359–372, 1964.
  26. [26]Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1555–1564, New York, NY, USA, 2016. Association for Computing Machinery.
  27. [27]Hongyuan Mei and Jason Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, volume 30, 2017.
  28. [28]Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive Hawkes process. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11183–11193. PMLR, 7 2020.
  29. [29]Siqiao Xue, Xiaoming Shi, Zhixuan Chu, Yan Wang, Hongyan Hao, Fan Zhou, Caigao Jiang, Chen Pan, James Y. Zhang, Qingsong Wen, Jun Zhou, and Hongyuan Mei. Easytpp: Towards open benchmarking temporal point processes. In International Conference on Learning Representations, 2024.arxiv.org/abs/2307.08097
  30. [30]Shilin He, Jieming Zhu, Pinjia He, and Michael R. Lyu. Loghub: A large collection of system log datasets towards automated log analytics. arXiv preprint arXiv:2008.06448, 2020.arxiv.org/abs/2008.06448
  31. [31]Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I. Jordan. Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, pages 117–132, 2009.
  32. [32]Adam Oliner and Jon Stearley. What supercomputers say: A study of five system logs. In 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN ’07), pages 575–584. IEEE, 2007.DOI
  33. [33]Adrien Cortes, Remi Rehm, and Victor Letzelter. Winner-takes-all for multivariate probabilistic time series forecasting. In Forty-second International Conference on Machine Learning, 2025.arxiv.org/abs/2506.05515
  34. [34]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomput., 568(C), 2024.arxiv.org/abs/2104.09864
  35. [35]Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the 36th International Conference on Machine Learning, pages 3744–3753, 2019.arxiv.org/abs/1810.00825
  36. [36]Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1906.00446
  37. [37]Prafullla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.arxiv.org/abs/2005.00341
  38. [38]Yunhao Zhang, Ruiying Qi, Jiale Zheng, Jianfeng Zhang, Lujia Pan, and Junchi Yan. Time series as language: A universal tokeniser for general-purpose time series foundation models. arXiv preprint arXiv:2606.09861, 2026.arxiv.org/abs/2606.09861
  39. [39]Wonho Bae, Mohamed Osama Ahmed, Frederick Tung, and Gabriel L. Oliveira. Meta temporal point processes. In International Conference on Learning Representations, 2023.arxiv.org/abs/2301.12023
  40. [40]Stijn Verdenius, Andrea Zerio, and Roy L. M. Wang. LaT-PFN: A joint embedding predictive architecture for in-context time-series forecasting. arXiv preprint arXiv:2405.10093, 2024.arxiv.org/abs/2405.10093
  41. [41]Dmitrii Babaev, Ivan Kireev, Nikita Ovsov, Mariya Ivanova, Gleb Gusev, Ivan Nazarov, and Alexander Tuzhilin. CoLES: Contrastive learning for event sequences with self-supervision. In Proceedings of the 2022 International Conference on Management of Data, 2022.DOI
  42. [42]Ethan Steinberg, Jason Alan Fries, Yizhe Xu, and Nigam Shah. MOTOR: A time-to-event foundation model for structured medical records. In International Conference on Learning Representations, 2024.arxiv.org/abs/2301.03150
  43. [43]Xiangyi Chen, Kousik Rajesh, Matthew Lawhon, Zelun Wang, Hanyu Li, Haomiao Li, Saurabh Vishwas Joshi, Pong Eksombatchai, Jaewon Yang, Yi-Ping Hsu, Jia Jing Xu, and Charles Rosenberg. PinFM: Foundation model for user activity sequences at a billion-scale visual discovery platform. In Proceedings of the Nineteenth ACM Conference on Recommender Systems, pages 381–390, 2025.
  44. [44]Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Zhang, Jun Zhou, Chenhao Tan, and Hongyuan Mei. Language models can improve event prediction by few-shot abductive reasoning. In Advances in Neural Information Processing Systems, volume 36, 2023.
  45. [45]Siqiao Xue, Xiaoming Shi, James Zhang, and Hongyuan Mei. HYPRO: A hybridly normalized probabilistic model for long-horizon prediction of event sequences. In Advances in Neural Information Processing Systems, volume 35, 2022.
  46. [46]Siqiao Xue, Yan Wang, Zhixuan Chu, Xiaoming Shi, Caigao Jiang, Hongyan Hao, Gangwei Jiang, Xiaoyun Feng, James Zhang, and Jun Zhou. Prompt-augmented temporal point process for streaming event sequence. In Advances in Neural Information Processing Systems, volume 36, 2023.
  47. [47]Michael Arbel, David Salinas, and Frank Hutter. EquiTabPFN: A target-permutation equivariant prior fitted network. arXiv preprint arXiv:2502.06684, 2025.arxiv.org/abs/2502.06684
  48. [48]Qian Huang, Eric Zelikman, Sarah Li Chen, Yuhuai Wu, Gregory Valiant, and Percy Liang. Lexinvariant language models. arXiv preprint arXiv:2305.16349, 2023.arxiv.org/abs/2305.16349
  49. [49]Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  50. [50]Satinder Singh, Michael R. James, and Matthew R. Rudary. Predictive state representations: A new theory for modeling dynamical systems. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pages 512–519, 2004.
  51. [51]Cosma Rohilla Shalizi and James P. Crutchfield. Computational mechanics: Pattern and prediction, structure and simplicity. Journal of Statistical Physics, 104:817–879, 2001.DOI
  52. [52]Felix Creutzig, Amir Globerson, and Naftali Tishby. Past-future information bottleneck in dynamical systems. Physical Review E, 79(4):041925, 2009.pubmed.ncbi.nlm.nih.gov/19518274

Paper details

vlnc:W_wktjqk3u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents