Summary
Forecasting machines and processes over time helps industries plan and respond to changes. The authors created a new public test called IND-Eval to better evaluate forecasting tools on real industrial data, focusing on more detailed and multivariable cases. They found that some forecasting models work better overall but vary by specific data types, showing the need to test models on data representative of the target industry. They also improved simulated training data by adding specific oscillator-based patterns, which helped in one domain but slightly hurt performance on another. Balancing different simulation methods helped achieve better accuracy while keeping flexibility.
What this means in practice
- •For industrial forecasting teams: Improve model selection by using benchmark tests tailored to the industrial domain and specific data types.
- •For simulation engineers: Design simulators that combine general and domain-specific patterns to create training data that better matches real industrial processes.
Authors
Vebjørn Haugland, Pavel Gulin, David Foss, Peet Cremer, Erlend Aune, Simone Casolo, Andreas Solbrå, Milad Mobarhan
Abstract
We introduce IND-Eval, a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks. Relative to established benchmarks such as GIFT-Eval, it emphasizes finer sampling resolutions and a greater prevalence of multivariate forecasting. Evaluating Chronos-2, TiRex-2, and TimesFM-3, we find that TimesFM-3 achieves the lowest source-balanced aggregate nMASE and nCRPS and leads seven of nine sources under each metric; source-specific exceptions reinforce the need for domain-representative evaluation for model selection. These findings also motivate improving domain alignment during pretraining, where real industrial data are often too scarce to close domain-specific gaps directly; we therefore investigate whether a domain's statistical structure can guide simulator design. In a case study on two offshore well-monitoring regimes, we improve a general simulator's coverage using an oscillator-augmented simulation pipeline. This lowers forecasting error in the target domain but reduces performance on the subset of GIFT-Eval with natively univariate targets. A 50/50 mixture of the general and augmented simulation recipes achieves the lowest error among the three training recipes for all four event-metric pairs in the case study while recovering most of the lost GIFT-Eval performance. Together, these results illustrate how domain-representative evaluation can guide model selection and the composition of simulated data while revealing a trade-off between specialization and generality.