Model learns shared vocabulary to predict complex event sequences
DES-JEPA: Learning a Shared Vocabulary for High-Dimensional Discrete Event Sequences
Artificial Intelligence
Summary
Many real-world events, like patient health records or machine logs, use different names for similar things, making it hard to analyze them together. The authors created a new approach that finds a shared set of event categories from different sources without needing labels to match. Their model can then predict what happens next in these event sequences even when it hasn’t seen that specific data before. This helps handle many types of events at once and works well on varied real datasets.
What this means in practice
- •For hospital data teams: Predict upcoming clinical events in patient records even when event labels vary between different hospitals.
- •For industrial operations teams: Analyze and anticipate machine or system event sequences with many distinct event types from diverse sources without manual labeling.
Authors
Hugo Math, Maximilian Forstenhäusler, Christos Anagnostopoulos
Abstract
Temporal foundation models such as Chronos and TimesFM generalize across domains through a shared numerical observation space. In contrast, discrete event sequences (DESs), including operational event logs or patient clinical-event trajectories, have domain-specific event vocabularies whose labels lack a consistent correspondence across domains. Beyond this lack of consistency, the number of distinct events in real-world settings is high enough to challenge both classical temporal point process modeling and current in-context foundation inference models such as FIM-PP. Critically, current foundation models for DESs are limited to a small, fixed number of event types, restricting their applicability to realistic, high-dimensional event streams. To address this, we propose two complementary modules: (i) DES-JEPA, a joint-embedding predictive architecture that learns a shared discrete vocabulary from synthetic stochastic processes with unaligned event vocabularies; and (ii) DES-PFN, a discrete event sequence prior-fitted network that reuses the learned vocabulary for in-context next-event prediction. Without task-specific fine-tuning, this framework achieves competitive zero-shot performance on eight unseen real-world datasets and scales up to 700 distinct event types.