Transformers improve long-term forecasting with adaptive time-aware attention
Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting
Machine LearningArtificial Intelligence
Summary
Forecasting events far into the future using data that changes over time is difficult because patterns can shift. The authors show that current methods using fixed ways to mark time don't handle these changing patterns well. They propose a new attention method that dynamically adapts how time is represented based on the input data. This approach improves forecasting accuracy without making training more complicated or slow.
What this means in practice
- •For data engineering teams: Improve predictive maintenance models by adapting to changing sensor data patterns over long time periods using Bayesian Time-aware Attention.
- •For financial forecasting teams: Enhance long-term market trend forecasting by capturing shifting temporal dynamics with adaptive positional biases.
Authors
Xinjie Lan, Yunbo Mei, Leheng Zhou, Hao Qi
Abstract
Transformers have demonstrated strong performance in long-term time series forecasting (LTSF), yet modeling non-stationary temporal dependencies remains challenging. In this paper, we interpret self-attention as a Bayesian posterior over attended positions, combining a content-based prior induced by scaled dot-product similarity with a temporal-evidence likelihood induced by positional encoding. This formulation provides a probabilistic interpretation of how positional encoding shapes the temporal dependencies captured by self-attention. Building on this perspective, we derive a population-level evidence lower bound (ELBO) that characterizes the role of positional encoding in forecasting weakly stationary time series. Our analysis further highlights a limitation of static positional encoding: when temporal dependence structures vary across input windows, a fixed temporal bias must accommodate heterogeneous dynamics and can therefore be suboptimal for non-stationary forecasting. Motivated by this analysis, we propose Bayesian Time-aware Attention (BTA), which employs a mixture-of-experts (MoE) router to adaptively combine temporal kernels and generate input-conditioned, head-specific positional biases. Extensive experiments on standard LTSF benchmarks demonstrate that BTA consistently improves forecasting performance across multiple Transformer backbones and achieves competitive overall performance with modest computational overhead and stable training.