Severity measures fail to predict frozen time-series forecaster errors
Severe but Harmless: Stress-Testing Shift Severity for Frozen Time-Series Forecasters
Machine LearningArtificial Intelligence
Summary
Time-series models that handle different tasks without retraining sometimes face changes called distribution shifts in data. The authors found that common ways to measure how big these changes are do not actually predict when the model’s predictions will get worse. In fact, big measured shifts don’t always cause big errors, and small measured shifts can cause big errors. They explain this mismatch through a technical analysis based on model behavior when changes happen.
What this means in practice
- •For data scientists: Identify that existing severity metrics may mislead about model robustness in time-series forecasting when using foundation models without retraining.
- •For traffic management teams: Evaluate traffic forecasting systems more accurately by recognizing that traditional shift severity scores might not indicate forecast reliability.
Authors
Korede R. Bishi, Dominic Okonkwo, Adetayo Okunoye
Abstract
Zero-shot time-series foundation models (TSFMs) have become attractive alternatives to task-specific models because they handle diverse downstream tasks, including anomaly detection and forecasting, without per-series training. However, time-series data undergo distribution shifts, and the dissimilarity between two regimes is commonly scored with statistical measures such as Wasserstein-1, where a higher score is read as a more severe shift. In systematic experiments using three established severity measures on nine regime shifts from three standard traffic datasets, we show that these measures do not reliably predict degradation for instance-normalized TSFMs. The regimes rated most severe are not the regimes that degrade the models most, and degradation reaches 29% in the lowest-scoring regime. In most cases, the audit shows no positive relationship between measured severity and model degradation. We provide an equivariance-based stress-testing analysis that explains why severity readings can become decoupled from model degradation.