Time-series models struggle to rank sparse future events accurately
When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events
Machine Learning
Summary
When predicting rare events in time data, knowing when events happen is often more important than their exact values. The authors found that popular time-series models do not do much better than simple guesses in spotting these event times. However, adding small event-focused adjustments to these models helps them identify when events occur more accurately than the base models alone or simpler methods. This suggests better evaluation methods and fine-tuning are needed to improve forecasting of sparse events.
What this means in practice
- •For supply chain analysts: Improve prediction of intermittent demand periods by using event-aware enhancements on pretrained forecasting models.
- •For energy grid operators: Better identify when rare energy usage spikes will happen by applying fine-tuned event ranking on time-series forecasts.
Authors
Daniel Schoess, Florian von Wangenheim
Abstract
Pretrained time-series foundation models (TSFMs) are evaluated as forecasters of future values, yet for sparse series many decisions depend only on which future periods contain activity. Standard benchmarks do not assess this. On five sparse datasets, we rank positions within forecast windows that contain both events and zeros. The released point forecasts of 12 TSFMs improve chance-corrected average precision over training-free references by at most 0.031, and in chance-corrected AUC the median TSFM falls below them on every dataset. With event supervision, linear probes of six frozen backbones improve on their backbone's point forecast in 29 of 30 backbone--dataset pairs. Averaging the predicted quantiles instead of taking their median improves the ranking of most TSFMs that forecast the median, and on two datasets the strongest such outputs rival the probes. The probes' advantage over raw-context learners depends on the dataset, and under the same probe, pretrained features outperform randomly initialized ones for five of six backbones. For sparse-event ranking, released point forecasts thus add little over simple references, whereas lightweight event heads on frozen TSFMs rank events better than these forecasts, and the best of them exceed gradient-boosted trees trained on the raw context on three of the five datasets. More broadly, assessing pretrained forecasters on tasks beyond value forecasting requires reporting their outputs, supervised probes of their representations, and raw-context and randomized controls side by side, since each supports a different conclusion.