Hybrid attention model improves irregular multivariate time series forecasting
A Hybrid Attention Model Learning Unified Time-aware Patch Representation for Irregular Multivariate Time Series Forecasting
Machine Learning
Summary
Forecasting data that happens at uneven times and from multiple sources is tricky because traditional methods often guess missing values or ignore the true timing. The authors created a model that understands the exact timing within small groups of data points and adjusts how it pays attention to these groups across variables. They also built a large dataset that keeps the natural irregularities in data for better training. Their approach performs well in tests, even when the data is irregular or regular.
What this means in practice
- •For financial analysts: Improve forecasting accuracy on irregularly timed financial indicators with asynchronous data from multiple sources using the hybrid attention model.
- •For healthcare data teams: Support forecasting patient health metrics collected intermittently and asynchronously from multiple sensors without imputing missing values.
Authors
Zhihao Lin, Li Lin, Qi Zhang, Kaiwen Xia, Shuai Wang, Jialin Qiao
Abstract
Time series foundation models (TSFMs) have recently delivered impressive zero-shot performance across diverse forecasting tasks. However, real-world decision-making frequently relies on \emph{irregular multivariate time series} (IMTS), where inconsistent inter-observation intervals and asynchronous sampling across variables coexist with informative missingness. Existing TSFMs handle such inputs either through imputation that injects spurious values or through index-based positional encodings that ignore continuous time. There is still a gap in the foundation model that follows the original IMTS patterns. In this paper, we propose a hybrid attention model that learns a unified time-aware patch representation for IMTS forecasting. We first design a \emph{time-aware patch encoding} that maps a variable number of intra-patch timestamps into a fixed-size embedding, producing a uniform format for irregular patches without resorting to imputation. We then introduce a \emph{time bias attention} mechanism that calibrates inter-patch temporal misalignment and asynchronous cross-channel dependencies as auxiliary attention offset. Finally, on top of a decoder-only Transformer backbone, we adopt a \emph{hybrid causal mask} that preserves a bidirectional full view over the historical context while keeping the forecast horizon strictly autoregressive. To support large-scale pretraining under irregular settings, we also curate VersaTSA, an archive of $30$B observations that retains the native sampling sparsity of its sources. Experiments on three IMTS benchmarks and a standard regular-MTS benchmark show that our model achieves state-of-the-art zero-shot performance on IMTS and remains competitive when transferred to regular forecasting.