A new benchmark standardizes evaluation of irregular time series forecasting
BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting
Machine LearningArtificial Intelligence
Summary
Irregular time series forecasting means predicting data collected at uneven time intervals, which is tricky to test fairly. The authors created BITS, a new system that brings together many datasets and methods under the same testing rules. BITS measures performance in multiple ways, not just by error, so it can better show how different methods work depending on the type of data gaps or irregularities. They found no single method works best for all cases, and evaluation method choice changes which model looks best.
What this means in practice
- •For data engineering teams: Evaluate irregular time series forecasting models consistently across diverse real-world datasets to improve model selection decisions.
- •For machine learning engineers: Use a unified platform to test and compare various time series forecasting methods under the same settings and metrics.
Authors
Kangjia Yan, Linfeng Wang, Tianen Shen, Xiangfei Qiu, Ruitong Zhang, Hao Miao, Jilin Hu, Chenjuan Guo, Bin Yang, Christian S. Jensen
Abstract
Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing evaluations are often conducted on a limited set of datasets with inconsistent experimental protocols and predominantly error-based metrics, rendering it difficult to compare and assess methods fairly and comprehensively across diverse settings. To eliminate these limitations and accelerate progress, we propose BITS, a standardized, reproducible, and extensible benchmark for advancing research on irregular time series forecasting. BITS covers eleven datasets from nine domains with diverse irregularity characteristics, and it characterizes the datasets according to their missing rate, missing pattern complexity, sampling irregularity, and skewness. Further, it offers a unified pipeline for data preprocessing, model integration and evaluation, and reporting. It accommodates regular and irregular time series forecasting methods, including time series foundation models, under consistent settings, incorporating both error-based and non-error-based evaluation metrics. Findings include that method performance varies substantially across irregularity characteristics, with no single modeling strategy consistently dominating. We also find that using error-based or non-error-based metrics can yield different model rankings, highlighting the need for multi-dimensional evaluation. The code can be found at https://anonymous.4open.science/r/BITS-8F2E/.