Regime diagnosis reveals hidden biases in industrial time-series forecasting
Beyond Model Ranking: Regime Diagnosis for Distributional-Statistical Misspecification in Industrial Time-Series Forecasting
Machine Learning
Summary
Time-series forecasting models often work well in tests but show consistent errors when used in real-world industrial settings. The authors found that these errors come not just from changes over time or data shifts, but also because usual model losses assume certain data behaviors that don't hold in all parts of the data. They introduce a new method to diagnose which parts of the data cause these biases and separate problems from the model itself versus those from how it’s trained. Their work helps improve forecasting by recognizing and addressing diverse data regimes instead of treating all data the same.
What this means in practice
- •For retail data teams: Detect and correct systematic bias in demand forecasts by diagnosing biases linked to different sales regimes like zero demand or high variability.
- •For industrial forecasting engineers: Improve model training to address bias from pooling heterogeneous demand patterns using regime-aware evaluation and correction techniques.
Authors
Pengyu Nie, Chenglang Xu, Yaoshi Chen, Chaogan Ren, Wei Hu, Chao Yang, Jiangong Zhang
Abstract
Time-series forecasting models achieve strong benchmark performance but exhibit severe systematic bias in industrial deployments. This train--deploy gap is conventionally attributed to temporal-structural errors or distribution shifts. We characterize a complementary source that these explanations overlook: canonical losses embed fixed statistical priors, while industrial demand mixes benign and pathological regimes---zero-inflation, skewness, high variability---in which these priors are systematically violated. The induced bias persists even under perfect temporal modeling, remains in a distributional-shape component that normalization cannot remove, and creates an aggregation trade-off invisible to aggregate metrics. We turn these observations into an evaluation toolkit centered on the Regime-wise Relative Bias Vector (RBV): a metric-agnostic, regime-decomposed diagnostic that audits how pooled training allocates systematic mismatch across pathological subpopulations. A controlled attribution analysis decomposes RBV into a model-independent intrinsic floor, set by each loss's estimand, and an excess component attributable to training, tracing observed bias to the loss rather than the model. A large-scale study---13 loss objectives, 3 seeds, 60,000+ series spanning RetailShiftBench and M5, with random-split controls---shows that regime-aware diagnosis separates optimization-type from bias-type failure, and that regime-aware training resolves the pooling-induced bias that capacity scaling cannot, for mean-type losses. A formal structural observation, that risk under evaluation-distribution contamination is affine in the pathology mixture weight, grounds these findings. Our work complements model ranking with mechanism-grounded, regime-oriented evaluation.