RankFormer improves multivariate time series forecasting with better transformers
RankFormer: Rethinking Time Series Transformers through Representation Capacity Analysis and Progressive Decoding
Machine Learning
Summary
Forecasting data that changes over time and has many connected parts is important for systems like the internet and sensors. The authors found that current transformer models, which are good at handling sequences, do not fully use their ability to understand complex patterns. They created RankFormer, which uses a new way to analyze and build the model so it can better capture important information over time and across different parts of the data. Their tests show it competes well with the best forecasting methods today.
What this means in practice
- •For iot system engineers: Improve prediction of sensor data trends by using RankFormer’s better interactions across multiple data channels for long-term forecasting.
- •For web infrastructure teams: Enhance traffic and usage forecasting with RankFormer to better anticipate complex multivariate patterns from large-scale service data.
Authors
Peiwang Tang, Cheng HE, Xijie Liang, Defu Lian
Abstract
Multivariate time series forecasting plays a critical role in Web and cyber-physical systems, where large-scale services and IoT infrastructures generate complex temporal data. Transformer-based models have achieved strong empirical performance in this domain, yet the principles governing the expressive capacity of their learned representations remain insufficiently understood. Moreover, many existing models rely on simplified decoding strategies, including channel-independent modeling and direct projection heads, which limit their ability to capture complex multivariate dynamics. To address these challenges, we propose RankFormer, a novel encoder-decoder architecture for multivariate time series forecasting. We establish a rank-based analysis framework to investigate how tokenization, embedding, and attention designs affect the representation capacity of time series Transformers. Our analysis reveals that patchification increases the potential representation capacity by aggregating multiple temporal observations into each token, while subsequent encoding operations determine how much of this capacity is realized. Guided by these insights, we develop a rank-enhancing encoder with Hourglass Attention to improve rank stability during information interaction. Furthermore, we introduce the Interactive Progressive Decoder (IPD), which improves future trajectory modeling through global-guided cross-channel interaction and progressive refinement. Extensive experiments on real-world long-term forecasting benchmarks demonstrate that RankFormer achieves competitive performance against state-of-the-art methods. Code repository: https://github.com/TangPeiwang/Rankformer.