Stress tests reveal hidden causes of time-series forecasting errors
Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecasting
Machine Learning
Summary
Forecasting models often look good on average, but this average error hides whether bad predictions come from unpredictable future changes or from models missing important information. The authors created special tests using a perfect reference (an oracle) that breaks down errors into those caused by the changing environment versus those caused by forecast shortcomings. They tested many forecasting methods and found different causes of errors depending on situations and prediction times. Their approach helps better understand why forecasting models fail and to improve their stability under changing conditions.
What this means in practice
- •For forecasting teams: Diagnose specific causes of forecasting errors to improve model robustness under changing environmental conditions.
- •For financial analysts: Identify when market changes cause forecast degradation vs model limitations to better manage forecasting risks.
Authors
Xu Lin, Runheng Zuo, Shengxuan Xu, Qitai Tan, Hongyu Lin, Xiao-Ping Zhang
Abstract
Average squared error cannot reveal whether forecasting performance degrades because the future becomes less predictable or because forecasts move farther from the conditional mean. We introduce paired, mechanism-controlled stress tests that decompose changes in expected squared error at each lead time into environmental risk and forecast-oracle distance, using an origin-conditioned predictive oracle unavailable to the evaluated models. Three end-to-end controls have known attribution. Specifically, the null, environmental-only, and information-gap controls verify that the pipeline assigns changes to the correct component. We then apply the benchmark to 24 deployable forecasters. Under frequent switching, 14 methods have higher realized MSE but lower oracle distance; under outlier-variance feedback, 19 have higher MSE but lower scale-standardized MSE. Short- and long-lead stress-response rankings have Spearman correlation 0.624, revealing substantial horizon-dependent reordering. We then study multivariate relation shifts. Across six models and three coupling severities, oracle distance accounts for only 0.7-3.9% of the decomposed expected-risk increase, and environmental-risk majority persists in an eight-channel system and a matched-difficulty audit of Ring, Block, and Hub relations. Finally, prespecified contrasts on independent data-generating process (DGP) realizations show that several visually compelling discovery profiles, including trend accumulation and the hypothesized switching reversal, do not replicate. The benchmark thus combines component-wise diagnosis with a held-out stability audit. It complements real-data out-of-distribution evaluation, which measures performance under realistic shifts when exact oracle attribution is unavailable.