WorksComputer Science
DES-JEPA: Learning a Shared Vocabulary for High-Dimensional Discrete Event Sequences
Abstract
Temporal foundation models such as Chronos and TimesFM generalize across domains through a shared numerical observation space. In contrast, discrete event sequences (DESs), including operational event logs or patient clinical-event trajectories, have domain-specific event vocabularies whose labels lack a consistent correspondence across domains. Beyond this lack of consistency, the number of distinct events in real-world settings is high enough to challenge both classical temporal point process modeling and current in-context foundation inference models such as FIM-PP. Critically, current foundation models for DESs are limited to a small, fixed number of event types, restricting their applicability to realistic, high-dimensional event streams. To address this, we propose two complementary modules: (i) DES-JEPA, a joint-embedding predictive architecture that learns a shared discrete vocabulary from synthetic stochastic processes with unaligned event vocabularies; and (ii) DES-PFN, a discrete event sequence prior-fitted network that reuses the learned vocabulary for in-context next-event prediction. Without task-specific fine-tuning, this framework achieves competitive zero-shot performance on eight unseen real-world datasets and scales up to 700 distinct event types.
Introduction and Related Work
Discrete event sequences (DESs) arise ubiquitously across domains, from vehicle diagnostic logs [1] to electronic health records (EHRs) [2, 3, 4]. Learning across these domains requires handling event vocabularies whose labels have no consistent meaning across datasets: the same event type may denote a machine error in one system and a clinical observation in another. Concurrent work on temporal foundation models (TFMs) treats a sequence as a continuous-valued signal and pretrains a shared numerical value that lives on a comparable scale [5, 6]. However, categorical event identifiers provide no equivalent basis for such an aligned observation space.
Prior work [7, 8, 9, 10] leverages temporal point processes (TPPs) for modeling event types and their time of occurrence [11]. Conventional methods therefore rely on fitting expensive statistical models (e.g., Hawkes processes [7]) that are known to be impractical at hundreds or thousands of distinct event types. Importantly, they must be fitted from scratch for every new batch of data. Berghaus et al. [12] address this with a foundation inference model FIM-PP, enabling in-context learning of TPPs without fine-tuning. However, FIM-PP supports at most 22 event types, restricting its direct application to datasets with larger event vocabularies and substantially limiting its scope as a foundation model.

Figure 1. Pipeline overview. (a): an event sequence is tokenized into a shared codebook via DES-JEPA. DES-PFN (b) learns in-context to predict the next code of a query . Right (c): a sequence of interest is forecasted, the resulting code is then decoded into the event types.
Therefore, extending current methodologies to more realistic real-world domains with hundreds of discrete event types remains an important open challenge.
We hypothesize that the missing component is a representation that transfers across processes, not a better fit to any single one. Therefore, the correct invariant to learn across anchor-free data-generating processes (DGPs) is therefore not the symbol itself, but alphabets of shared patterns that recur across unaligned processes. We gather this representation in a learned codebook, which is reusable across downstream tasks such as zero-shot next-event prediction or long-horizon forecasting [13].
To address this gap, we introduce a joint-embedding predictive architecture for discrete event sequences (DES-JEPA), a JEPA-based method [14, 15, 16] that learns a shared discrete vocabulary, i.e., codebook. It consists of two transformer-based encoders: (i) a teacher encoder which quantizes future windows into latent codes, and (ii) a student encoder which maps historical observations to a latent representation. Through a prior-fitted network (PFN)-style meta-pretraining (TabPFN, TabICLv2 [17, 18, 19, 20]) using synthetic data, the model learns to capture recurring dynamics across stochastic processes with unaligned vocabularies. Via the frozen teacher encoder and codebook embeddings, we tokenize any sequence of interest (query). Then, a second model (DES-PFN) predicts the next-event on this query, without fine-tuning, conditioned on a reference set of sequences (context). Together, they support zero-shot next-event prediction through a shared latent vocabulary, extending amortized in-context inference to discrete representations of temporal dynamics. Our contributions are:
We introduce DES-JEPA, combining predictive latent learning, vector quantization, and meta-pretraining to learn a shared codebook from any discrete event sequences.
We introduce DES-PFN, which uses the shared codebook computed by the teacher encoder of DES-JEPA, to autoregressively predict unseen event types without task-specific fine-tuning. We evaluate it on zero-shot next-event prediction on various unseen real-world datasets ranging from 3 to 700 event types, placing it as the best performing model for event sequences.
Methodology
Problem Statement. Let denote a meta-distribution over a DGP, for instance a high-order Markov chain [21]. Each draw instantiates a process over a finite, -specific alphabet , which changes identity with every draw . Thus, no embedding table indexed by a local event type is shareable. Given any process, we aim to learn a shared representation that is invariant to event types that enables downstream tasks such as the modeling of (Section 2.2).
DES-JEPA: A Shared Discrete Codebook
Following JEPA [14, 22], DES-JEPA consists of two encoding paths: (i) an encoder (student) maps past observations to a representation , which are then mapped via a predictor to logits over the codes of , and (ii) a bidirectional encoder (teacher) , a shallow prefix of which is only updated by an exponential moving average (EMA) [15, 16], encodes the next raw events into , quantized against by nearest-neighbor lookup as,
We keep fixed. The student never observes and predicts autoregressively purely from , while the teacher is evaluated under stop-gradient and receives no gradient at all. This prevents both encoders from collapsing to a constant map [15]. For a single trajectory drawn from , we train by minimizing,
at every position . The main training signal is the cross-entropy loss, predicting the next latent code from history . A standard VQ-VAE codebook commitment terms [23] () pulls and toward each other. Since is trained only via EMA updates, the commitment term never actually back-propagates through . During inference, the future tokens are replaced by the embedding of the token to prevent any information leakage.
During training, sampling realizes amortized Bayesian PFN-style meta-training [17, 18, 19]: the codebook acts as a finite statistic for ’s identity, rather than representing directly (Appendix § 7). Naive VQ training is well known to collapse onto a small subset of indices. Thus, we actively monitor and enforce codebook utilization via a perplexity regularization and revive dead codes with a usage-EMA criterion. For additional details, please refer to Appendix § 8.
DES-PFN: In-Context Autoregressive Next-Event Prediction
Through the creation of codebook , we created a common encoding of DESs. This encoding is leveraged to pretrain a model on next-event prediction tasks in order to perform downstream zero-shot next-event prediction on unseen data. To do so, we leverage in-context learning of event sequences [12], where for a sequence of interest (query) given a context , both drawn from the same DGP , one performs the next-event prediction.
We freeze the codebook and use it to tokenize each element in the raw sequence to form a sequence of event codes via the teacher path of Eq. (1). We train DES-PFN on tokenized queries and context generated from the same synthetic DGPs meta-distributions () used in the training of DES-JEPA. Given these inputs, DES-PFN predicts the next token as,
At inference, the frozen DES-JEPA teacher maps the context and query history to discrete codes. DES-PFN then forecasts autoregressively in this shared code space, conditioning on the encoded inputs and previously predicted codes. This representation enables zero-shot transfer across DGPs with disjoint event vocabularies, without updating either model. In the experiments, DES-JEPA and DES-PFN are pretrained on six DGPs with a maximum codebook capacity of and contain 60 million and 40 million parameters, respectively. Architectural details are provided in Appendix § 6.
Decoding Codes to Event Types. DES-PFN forecasts a tokenized code , whereas the actually desired event prediction is required in its raw alphabet . Since event labels are unaligned across processes and quantization is a many-to-one projection and therefore non-invertible, we estimate a context-specific token from the tokenized pairs in . We use a Nadaraya-Watson estimator [24, 25] for candidate ,
where is a nonnegative cosine-based kernel with temperature calibrated per context and a candidate event type. The predicted code queries each context occurrence’s code , which casts a similarity-weighted vote for its identity , allowing identities associated with nearby codes to be recovered without an exact match to in (Figure 2).

Figure 2. Top: Kernel-decode inverse mapping on real StackOverflow and Vehicle DTC contexts, generated directly from the trained checkpoint (no illustrative numbers). Bottom row: the encode/decode round trip on ONE real sequence over cumulative time: raw ids, the codes DES-JEPA tokenizes them into, and the raw ids recovered by decoding those codes back. The color coding matches the top row, so a mismatch is directly visible as a color change.
Experiments
Baselines, Datasets & Experimental Setup. We evaluate DES-PFN against FIM-PP [12] and three gradient-trained neural TPP baselines, RMTPP [26], NHP [27], and SAHP [28], for next-event prediction tasks. We benchmark the model performance on eight real event-type datasets: Twitter retweet cascades, StackOverflow badge awards, Taobao and Amazon user actions [29], HDFS and BGL system logs [30, 31, 32], MIMIC-IV hospital lab-test sequences [3] and a vehicle event logs [1]. Every zero-/few-shot method, i.e., DES-PFN, FIM-PP and each neural baseline’s few-shot variant (refit from scratch on only the in-context set), sees the identical example sequences. DES-PFN and FIM-PP receive no gradient updates. We report the validation top-1 accuracy for each fully-trained neural baseline variant trained for 5 epochs, as a reference upper bound.
Results. Table 1 reports DES-PFN’s zero-shot (no-gradient-update) next-event prediction accuracy across eight real datasets spanning three to seven hundred event types. Fully-trained specialists gradient-trained on each dataset’s entire training split still outperform DES-JEPA on the four lower-cardinality easytpp datasets. However, this specialism does not generalize well, since each needs a real held-out test split and per-dataset retraining from scratch to be accurate. In low cardinality, FIM-PP, which shares DES-PFN’s zero-shot, in-context inference setting has competitive performance beating DES-PFN in three out of five datasest. Importantly, its fixed 22-event type architecture precludes evaluation on BGL, MIMIC-IV, and Vehicle. Using the single frozen codebook, DES-PFN supports larger event vocabularies and is the only method with reported results on these datasets; all gradient-trained baselines either lack a usable data split or exceed the prescribed training-time budget. DES-PFN’s weakest result is on MIMIC-IV: we reason that the shared blood-draw times tampes across 95% of consecutive lab events may leave DES-PFN little time information to exploit.
Table 1: Real-world next-event-type accuracy (Top-1, mean±std averaged over 5 runs) for , and . Baselines are reported in two regimes: few-shot (refit from scratch on the same 128 context sequences) and full (fully-trained). For HDFS/[30, 31]/MIMIC-IV, since no real train/test split is available, the ‘full’ column instead reports a §-marked local train/test split (trained once on every earlier sequence actually available, evaluated on the same held-out query set); ‘few-shot’ stays “–” for these three (not run). FIM-PP is capped at marks (“–”). Bold/underline mark the best/second-best score per row. Vehicle is reported at two vocabulary caps ( and ), † marks a method that genuinely exceeded the 600s time budget.
Table 1.
Conclusion
We introduced a framework to model high-dimensional event sequences and enable zero-shot next-event prediction. The first model, DES-JEPA pretrains a single, shared codebook over a meta-distribution of synthetic DGPs, giving arbitrary, anchor-free event sequences that share no vocabulary or numeric scale, a common representation. By applying this frozen codebook, DES-PFN is able to perform zero-shot next-event prediction on eight real-world dataset spanning across three orders of magnitude in cardinality, from to . This approach remains competitive at low cardinality, and remains the only reliable method producing a non-trivial number once cardinality grows to hundreds of event types. Two limitations motivate future work: (i) the codebook capacity is fixed during pretraining, and currently lacks rate-distortion experiments (ii) evaluation is restricted to one-step-ahead prediction. Long-horizon autoregressive rollouts remain untested, particularly regarding error accumulation and the ability to represent multiple plausible futures beyond a single point forecast.
Limitations.
Two limitations point to natural extensions. The codebook size is fixed at pretraining time rather than adapted to a downstream dataset’s true cardinality. Moreover, every result in this paper is one-step-ahead: DES-PFN has not been evaluated on genuine long-horizon, multi-step rollout, where a single point forecast poorly represents the many plausible futures a real event stream can take and small early errors can compound over many steps. We see these last two limitations as connected: the natural next step is to replace the single-code predictor with a multi-head decoder trained under a winner-takes-all, multiple-choice-learning objective [33], letting DES-JEPA and DES-PFN output several plausible future codes per query instead of one point forecast, precisely the mechanism long-horizon rollout needs to stay calibrated as uncertainty compounds step by step.
Architecture Detail
Both DES-JEPA and DES-PFN were pretrained on a single NVIDIA RTX 2080 Ti, approximately one day each. All experimental results were measured at inference time on an Apple M2 Pro CPU, with no GPU acceleration.
DES-JEPA
Embeddings
Every event is embedded from purely statistics of its own occurrence history: self-information/surprisal, running (global and local-window) frequency, whether the type has been seen before, and recency and dispersion of its own inter-arrival gaps, together with event-count (“step”) analogues of the timing features that need no real timestamp. None of these is computed from a learned lookup table keyed on the raw event id, since a raw id carries no stable meaning across draws of (§ 2): therefore event identity is represented purely through these statistics. We attach no additional per-event covariate beyond type and time (no “marks” in the temporal-point-process sense) to any data-generating process.
Figure 3 shows the pairwise correlation of these features on a representative synthetic batch: highly correlated pairs (e.g. causal_freq/local_freq, mean_dt_type/mean_step_gap_type) are deliberately kept rather than pruned, since their real-valued/step-valued counterparts remain the only signal available whenever timestamps are missing or unreliable.

Figure 3. Pairwise Pearson correlation across the 14 causal statistical features feeding the DES-JEPA event embedding, computed on a mixed-DGP synthetic batch.
Transformer Block
Student Encoder. A causal Transformer over the whole past, with sliding-window attention (window : token attends only to the 32 preceding tokens) and RoPE [34] position encoding.
Teacher Encoder. A shallow, bidirectional prefix of the student stack (2 of the student’s 6 layers in the checkpoint used throughout this paper), updated only by an EMA of the student’s own weights () (§ 2.1); it encodes the next raw events as one coherent unit rather than causally within that short window.
Vector Quantizer. The teacher’s continuous output is matched to the nearest of codebook vectors (Eq. (1)) and trained with the combined cross-entropy/VQ-VAE objective of Eq. (2). Appendix D details the perplexity regularizer and dead-code revival mechanism that keep this codebook from collapsing into a dozen of codes.
DES-PFN
Embeddings
DES-PFN’s event-code embedding table is the frozen DES-JEPA codebook itself (§ 2.2), followed by a learned linear projection whenever the two models’ hidden sizes differ. Inter-arrival time uses a Chronos-style [5] discretized embedding: is binned into a fixed vocabulary and looked up, turning next-time prediction into classification rather than continuous regression. An optional duplicate-count embedding additionally marks immediate event repeats.
Transformer Block
DES-PFN alternates two attention axes, at every layer, over the (sequence, time, hidden) grid formed by the reference context and the query.
Causal Attention. A standard causal Transformer block over the temporal axis, applied independently to every sequence (context and query alike), with RoPE and a sliding-window band (32 in the checkpoint used throughout this paper).
Col-Wise Attention. Following the column-then-row, induced-attention design recently used by tabular foundation models such as TabICLv2 [20] and the inducing-point mechanism of Set Transformer [35], cross-sequence attention pools information through a small, fixed set of learnable vectors rather than full attention across reference sequences.
Pretraining
Prior-Fitted Network and Synthetic Data Generation
Both stages are pretrained exclusively on synthetic data drawn from a meta-distribution over six data-generating process (DGP) families, each implemented in synthetic_data/. No real-world data is used at either pretraining stage; every real-world result in § 3 is zero-shot transfer.
Every family is instantiated fresh per task (all parameters below redrawn) and draws its own active event-type count (the curriculum’s , denoted this way to avoid clashing with the codebook size ) uniformly from a shared range, detailed at the end of this section.
Markov (-th order chain). Order is drawn once per task – not fixed at 1: the next type follows
with bias , sparse mixed-sign interaction matrix (density , ; excitatory, inhibitory), and an exponential recency decay over the lags. With probability 0.5 per task, a two-layer MLP term (random fixed embedding, one hidden layer, a randomly chosen activation) adds non-additive lag interactions; otherwise . Holding times are Weibull with a history-modulated scale: , , , where , , are fixed-at-construction (not trained) per-task parameters; depends only on the preceding types, not on the candidate .
Hawkes (exact Ogata [7] thinning). A multivariate self-exciting process with conditional intensity
baseline ; excitation built sparse (each row excites 1–3 other types, , weights ) plus a diagonal , , globally rescaled so the spectral radius (stationarity, ). The kernel is, with probability 0.5 each: exponential , (short-range, Markovian); or power-law/Omori–Utsu , , (long-range, non-Markovian).
Neural Autoregressive Models A randomly-initialized, never-trained LSTM: , , , where token and timing heads read from the SAME (fully joint dynamics) and is a random embedding table. Hidden size , depth layers, embedding dimension , all weights i.i.d. , – freshly redrawn every task.
Hidden Markov Model. latent states, ; is sticky, with the escape probability (not the sticky value itself) drawn log-uniformly, (i.e. ) – avoiding oversampling short dwell times, since dwell is highly nonlinear in . Emission , , (small – each state emits a small, peaked subset of types); timing , , – the same latent state drives both what happens and when.
Burst. An i.i.d. renewal timing process, , (decreasing hazard, genuinely bursty), , both fixed for the whole task and independent of type/history – combined with a first-order sticky-categorical type chain: with probability , else resampled, , . Models verbatim repeat-heavy streams (keep-alives, retries, repeated clicks).
Template. A shared directed graph next: (at most one outgoing edge per node) is built from short chains, length ; a latent walk deterministically follows whenever defined, otherwise restarts at a start node chosen per-template weights , . The emitted, noisy type mixes a point mass on the latent node with a uniform background,
so every modeled edge has expected causal strength – this is what gives event-type pairs a near-deterministic causal coupling that Hawkes-style self-excitation (which only boosts future rate, never identity) cannot express. Timing is (i.e. Exponential), with scale right after a template restart and while continuing along an established edge.
Sampling Parameters During Pretraining. DES-JEPA (Stage 1) samples a single trajectory per task, from all six DGPs with TEMPLATE and HAWKES up-weighted to 30% and 15% probability respectively (the remaining four share the rest evenly, 13.75% each), a fixed sequence length of 128, and event-type cardinality drawn log-uniformly from per task. DES-PFN (Stage 2) instead samples both a reference context and a query sequence per task, following a TabICLv2-style [20] curriculum that couples context size and sequence length under a fixed per-batch token budget (): and sequence length vary anticorrelated by construction, so a batch of many short reference sequences and a batch of few long ones cost roughly the same compute; as in Stage 1.
Tokenizing an observed sequence at inference (the last positions). During training, the teacher only ever supervises positions with a genuine, fully-observed future window: for a length- training sequence, positions each have a real target, and the remaining tail positions are simply dropped from the loss. At inference, however, DES-PFN needs one code per position for the entire query, including its most recent, currently-last observed event, but that position’s own future is exactly what is being forecast and does not exist yet. The tokenization made by DES-JEPA resolves this by computing genuine windows for every position that has one (positions ) and, for the remaining tail position(s) only, substituting a degenerate window built by repeating the last observed embedding times (emb[:, -1:].expand(...)) in place of the unavailable future. With (the value used throughout this paper) this affects only a single position per tokenized sequence: the code assigned to the most recently observed event is computed from this repeated-embedding fallback rather than a genuine future window, for every context sequence and for the query alike. This is a real, deliberate train/inference asymmetry (never exercised during training, since the tail positions are excluded from the loss there) rather than a bug, but it does mean the single most recent position’s code is systematically less informative than every earlier position’s.
Additional Analysis
Codebook Perplexity. A universal codebook shared across heterogeneous DGPs is only useful if its capacity is actually exploited; naive VQ training is well known to collapse onto a small subset of indices (e.g., DES-JEPA using 10 codes at most instead of the available ones). We track the empirical code-usage distribution over a batch of target windows, , and define
Definition 1 (Codebook Perplexity)
, with under uniform code usage and under index collapse onto a single code. We monitor PPL per-DGP-family and aggregated across : a universal codebook whose perplexity remains high within every family but whose per-family code supports remain distinguishable across families is precisely the signature of a shared dictionary that is neither collapsed nor merely memorizing family identity. Aggregated across the full training mixture, codes give a final perplexity of . Figure 4 breaks this down per DGP family, measured directly from the trained checkpoint on 800 fresh synthetic tasks per family (): perplexity is high and comparable across all six families (601–716 of the 768 available codes actually used, no family collapsing onto a handful or hogging most of the codebook), supporting the "shared, non-collapsed dictionary" reading above. The decode-accuracy panel (true code raw id via the same per-context majority-vote decoder used for real-world data, § 2.2) is uniformly modest (2.5–12.7%) across every family: with up to 300 active raw ids sharing only 768 codes total across six families. Exact code identity alone cannot always pin down a unique raw id (the same codebook-capacity/collision cost documented for real-world high-cardinality datasets in the experiment section. An high perplexity number indicates the codebook is being used broadly and non-degenerately, not that every individual code is uniquely decodable.

Figure 4. Per-DGP codebook Perplexity (left) and round-trip true-code-to-raw-id decode accuracy (right), measured directly from the trained checkpoint used throughout this paper (no retraining), 800 fresh synthetic tasks per DGP family, .
Dead-code revival. A perplexity regularizer only discourages collapse in aggregate; it does not repair codes that have already drifted out of use. Following the EMA-cluster-size restarts of VQ-VAE-2 [36] and its scaled-up variant in Jukebox [37], we track an exponential moving average of each code’s usage share,
and flag code as underused whenever falls below a fraction of its fair share . A per-code counter accumulates consecutive underused steps and resets to zero the moment recovers. The code becomes reset-eligible once . Unlike a hard, per-step presence test, is barely perturbed by a single rare assignment, so a code that only survives on stray, noise-level picks is still correctly identified as dead. All simultaneously reset-eligible codes are re-seeded in one batched event, from encoder outputs of the current batch, once the reset-eligible fraction exceeds a threshold ,
Relative to the per-code restarts of prior work, our two departures are: (i) a batched trigger that rescues a whole backlog of dead codes in a single event rather than continuously reviving them one at a time, and (ii) a usage-share-relative threshold, rather than an absolute cluster-size count, which keeps the criterion scale-invariant as grows far beyond .
Related Work
Transfer across temporal processes. Continuous TFMs such as Chronos, TimesFM, and UniTok transfer forecasts through a shared numerical observation space [5, 6, 38]. PFNs, Foundation inference models (FIMs), MetaTPP and LaT-PFN instead amortize inference across tasks or stochastic processes, but transfer continuous forecasts, intensity functions, rate matrices, or task representations rather than a common discrete vocabulary [17, 12, 39, 40]. Models for DESs, including CoLES, MOTOR, and PinFM, learn reusable representations within domains whose event schemas remain stable [41, 42, 43].
Event-sequence forecasting beyond a single stochastic-process family. A separate line of work adapts a single event-sequence model to changing data without our cross-alphabet, zero-shot goal. LAMP couples a TPP with a large language model performing few-shot abductive reasoning over retrieved historical events to improve mark prediction [44]; HYPRO reweights an autoregressive TPP’s proposals with a learned energy function for long-horizon, multi-step rollout [45]; PromptTPP retrieves a continuous-time prompt pool to continually adapt a TPP to streaming, non-stationary event data [46]. These target complementary problems: reasoning-augmented prediction, long-horizon calibration, and continual adaptation, respectively, within one dataset’s fixed vocabulary, rather than a codebook shared, without any fine-tuning, across datasets whose event identities are mutually meaningless; we do not compare against them directly for this reason, but HYPRO in particular addresses the long-horizon rollout setting we flag as future work (Appendix § 4).
Vocabulary symmetry and predictive discretization. EquiTabPFN and lexinvariant language models address target-label and vocabulary permutations, respectively, but neither learns a shared predictive codebook across stochastic-process families [47, 48]. Vector-quantized and predictive self-supervised methods learn discrete or latent representations whose alignment is inherited from a common visual or acoustic space [23, 49, 15, 22]; predictive-state, causal-state, and information-bottleneck approaches instead motivate grouping histories by their future behavior [50, 51, 52]. DES-JEPA connects these directions by learning a shared discrete prediction space across processes whose native event alphabets are unaligned.
References
- [1]Hugo Math, Rainer Lienhart, and Robin Schön. Harnessing event sensory data for error pattern prediction in vehicles: A language model approach. Proceedings of the AAAI Conference on Artificial Intelligence, 39(18):19423–19431, 4 2025.
- [2]Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digital Medicine, 4:86, 2021.DOI
- [3]Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. Mimic-iv, a freely accessible electronic health record dataset. Scientific Data, 10(1):1, 2023.DOI
- [4]Alex Labach, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. Duett: Dual event time transformer for electronic health records. In Kaivalya Deshpande, Madalina Fiterau, Shalmali Joshi, Zachary Lipton, Rajesh Ranganath, Íñigo Urteaga, and Serene Yeung, editors, Proceedings of the 8th Machine Learning for Healthcare Conference, volume 219 of Proceedings of Machine Learning Research, pages 403–422. PMLR, 8 2023.arxiv.org/abs/2304.13017
- [5]Abdul Fatir Ansari, Lorenzo Stella, Caner Türkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, et al. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2403.07815
- [6]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10148–10167, 2024.arxiv.org/abs/2310.10688
- [7]Yoshihiko Ogata et al. Statistical studies of the occurrence of small earthquakes: I. A point process model for earthquake sequences. Annals of the Institute of Statistical Mathematics, 30(1):379–395, 1978.
- [8]Hongyuan Mei and Jason Eisner. The neural Hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, volume 30, 2017.
- [9]Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer Hawkes process. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11692–11702. PMLR, 7 2020.arxiv.org/abs/2002.09291
- [10]Emmanuel Bacry, Martin Bompaire, Stéphane Gaffas, and Jean-Francois Muzy. Sparse and low-rank multivariate hawkes processes. Journal of Machine Learning Research, 21(50):1–32, 2020.
- [11]Patrick J. Laub, Young Lee, Philip K. Pollett, and Thomas Taimre. Hawkes models and their applications. Annual Review of Statistics and Its Application, 12(Volume 12, 2025):233–258, 2025.arxiv.org/abs/2405.10527
- [12]David Berghaus, Patrick Seifner, Kostadin Cvejkoski, Cesar Ojeda, and Ramses J Sanchez. In-context learning of temporal point processes with foundation inference models. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2509.24762
- [13]Xiao Shou. Unified flow matching for long horizon event forecasting, 2025.arxiv.org/abs/2508.04843
- [14]Yann LeCun. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. 2022.
- [15]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael G. Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, pages 15619–15629, 2023.arxiv.org/abs/2301.08243
- [16]Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-jepa 2: Self-supervised video models enable understanding, prediction and planning, 2025.arxiv.org/abs/2506.09985
- [17]Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. International Conference on Learning Representations, 2022.arxiv.org/abs/2112.10510
- [18]Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. International Conference on Learning Representations, 2023.arxiv.org/abs/2207.01848
- [19]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637:319–326, 2025.DOI
- [20]Jingang QU, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2602.11139
- [21]D. Koller and N. Friedman. Probabilistic Graphical Models: Principles and Techniques. Adaptive computation and machine learning. MIT Press, 2009.
- [22]Junyeob Baek, Hosung Lee, Christopher Hoang, Mengye Ren, and Sungjin Ahn. Discrete JEPA: Learning discrete token representations without reconstruction. In ICML 2025 Workshop on Tokenization, volume 267 of Proceedings of Machine Learning Research, 2025.arxiv.org/abs/2506.14373
- [23]Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1711.00937
- [24]E. Nadaraya. On estimating regression. Theory of Probability and Its Applications, 9:141–142, 1964.DOI
- [25]Geoffrey S. Watson. Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, 26(4):359–372, 1964.
- [26]Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1555–1564, New York, NY, USA, 2016. Association for Computing Machinery.
- [27]Hongyuan Mei and Jason Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, volume 30, 2017.
- [28]Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive Hawkes process. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11183–11193. PMLR, 7 2020.
- [29]Siqiao Xue, Xiaoming Shi, Zhixuan Chu, Yan Wang, Hongyan Hao, Fan Zhou, Caigao Jiang, Chen Pan, James Y. Zhang, Qingsong Wen, Jun Zhou, and Hongyuan Mei. Easytpp: Towards open benchmarking temporal point processes. In International Conference on Learning Representations, 2024.arxiv.org/abs/2307.08097
- [30]Shilin He, Jieming Zhu, Pinjia He, and Michael R. Lyu. Loghub: A large collection of system log datasets towards automated log analytics. arXiv preprint arXiv:2008.06448, 2020.arxiv.org/abs/2008.06448
- [31]Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I. Jordan. Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, pages 117–132, 2009.
- [32]Adam Oliner and Jon Stearley. What supercomputers say: A study of five system logs. In 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN ’07), pages 575–584. IEEE, 2007.DOI
- [33]Adrien Cortes, Remi Rehm, and Victor Letzelter. Winner-takes-all for multivariate probabilistic time series forecasting. In Forty-second International Conference on Machine Learning, 2025.arxiv.org/abs/2506.05515
- [34]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomput., 568(C), 2024.arxiv.org/abs/2104.09864
- [35]Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the 36th International Conference on Machine Learning, pages 3744–3753, 2019.arxiv.org/abs/1810.00825
- [36]Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1906.00446
- [37]Prafullla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.arxiv.org/abs/2005.00341
- [38]Yunhao Zhang, Ruiying Qi, Jiale Zheng, Jianfeng Zhang, Lujia Pan, and Junchi Yan. Time series as language: A universal tokeniser for general-purpose time series foundation models. arXiv preprint arXiv:2606.09861, 2026.arxiv.org/abs/2606.09861
- [39]Wonho Bae, Mohamed Osama Ahmed, Frederick Tung, and Gabriel L. Oliveira. Meta temporal point processes. In International Conference on Learning Representations, 2023.arxiv.org/abs/2301.12023
- [40]Stijn Verdenius, Andrea Zerio, and Roy L. M. Wang. LaT-PFN: A joint embedding predictive architecture for in-context time-series forecasting. arXiv preprint arXiv:2405.10093, 2024.arxiv.org/abs/2405.10093
- [41]Dmitrii Babaev, Ivan Kireev, Nikita Ovsov, Mariya Ivanova, Gleb Gusev, Ivan Nazarov, and Alexander Tuzhilin. CoLES: Contrastive learning for event sequences with self-supervision. In Proceedings of the 2022 International Conference on Management of Data, 2022.DOI
- [42]Ethan Steinberg, Jason Alan Fries, Yizhe Xu, and Nigam Shah. MOTOR: A time-to-event foundation model for structured medical records. In International Conference on Learning Representations, 2024.arxiv.org/abs/2301.03150
- [43]Xiangyi Chen, Kousik Rajesh, Matthew Lawhon, Zelun Wang, Hanyu Li, Haomiao Li, Saurabh Vishwas Joshi, Pong Eksombatchai, Jaewon Yang, Yi-Ping Hsu, Jia Jing Xu, and Charles Rosenberg. PinFM: Foundation model for user activity sequences at a billion-scale visual discovery platform. In Proceedings of the Nineteenth ACM Conference on Recommender Systems, pages 381–390, 2025.
- [44]Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Zhang, Jun Zhou, Chenhao Tan, and Hongyuan Mei. Language models can improve event prediction by few-shot abductive reasoning. In Advances in Neural Information Processing Systems, volume 36, 2023.
- [45]Siqiao Xue, Xiaoming Shi, James Zhang, and Hongyuan Mei. HYPRO: A hybridly normalized probabilistic model for long-horizon prediction of event sequences. In Advances in Neural Information Processing Systems, volume 35, 2022.
- [46]Siqiao Xue, Yan Wang, Zhixuan Chu, Xiaoming Shi, Caigao Jiang, Hongyan Hao, Gangwei Jiang, Xiaoyun Feng, James Zhang, and Jun Zhou. Prompt-augmented temporal point process for streaming event sequence. In Advances in Neural Information Processing Systems, volume 36, 2023.
- [47]Michael Arbel, David Salinas, and Frank Hutter. EquiTabPFN: A target-permutation equivariant prior fitted network. arXiv preprint arXiv:2502.06684, 2025.arxiv.org/abs/2502.06684
- [48]Qian Huang, Eric Zelikman, Sarah Li Chen, Yuhuai Wu, Gregory Valiant, and Percy Liang. Lexinvariant language models. arXiv preprint arXiv:2305.16349, 2023.arxiv.org/abs/2305.16349
- [49]Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
- [50]Satinder Singh, Michael R. James, and Matthew R. Rudary. Predictive state representations: A new theory for modeling dynamical systems. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pages 512–519, 2004.
- [51]Cosma Rohilla Shalizi and James P. Crutchfield. Computational mechanics: Pattern and prediction, structure and simplicity. Journal of Statistical Physics, 104:817–879, 2001.DOI
- [52]Felix Creutzig, Amir Globerson, and Naftali Tishby. Past-future information bottleneck in dynamical systems. Physical Review E, 79(4):041925, 2009.pubmed.ncbi.nlm.nih.gov/19518274