Time series forecasting needs scenario based stress testing for failures
Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing
Artificial Intelligence
Summary
Current ways of checking how well time series forecasting works focus mainly on accuracy with simple noise changes, which misses many real-world problem types. The authors point out that real issues often involve complex events that change patterns or cause errors in collected data. They suggest tests that include detailed scenarios describing these event types, how they affect data, and how hard they are to predict. This approach helps understand when and why forecasting models fail, making it easier to improve reliability in practical uses.
What this means in practice
- •For transportation planners: Improve forecasting reliability by testing models under specific failure scenarios drawn from realistic transport events.
- •For energy grid operators: Validate forecasting tools against scenarios involving sensor faults or abnormal demand to better prepare for operational challenges.
A position paper. It proposes an approach and reports no results.
Authors
Yuyang Zhao, Lian Xu, Hao Xue
Abstract
Time series forecasting (TSF) increasingly drives decisions in transportation, energy, finance, healthcare, and infrastructure, yet current evaluation remains overly narrow: standard benchmarks reward low held-out error, while robustness studies typically reduce failure to Gaussian noise, random masking, or bounded adversarial perturbations. This obscures the real failure modes of deployed forecasting systems. Input-side anomalies are not merely noisier inputs: they often reflect structured events that alter temporal dynamics, break cross-variable dependencies, induce regime shifts, or propagate from faulty sensors to downstream decisions. These semantic, causal, and system-level failures cannot be faithfully captured by i.i.d. perturbations alone. The rise of TSF foundation models makes this evaluation gap more urgent, as unauditable pretraining corpora make held-out generalization increasingly unreliable. We therefore advocate scenario-grounded stress testing. Each test instance should include historical inputs and future targets, together with a semantic scenario, an explicit failure operator, and a measurable difficulty level. This shift makes evaluation interpretable, attributable, and deployment-relevant and friendly, enabling the community to ask not only which model is accurate, but under what conditions it fails and why.