Benchmark tests time series forecasts for future what if scenarios
WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting
Artificial IntelligenceMachine Learning
Summary
Forecasting how things like power grids or traffic respond to future events is tricky, especially when those events have never happened before. The authors created WhatIf-Bench, a set of tests using simulated data from real systems to check how well forecasting models predict outcomes when covariates change in new ways. They found that while models predict familiar future patterns fairly well, their accuracy drops when facing new combinations of changes. This benchmark highlights that generalizing to new future scenarios remains a key challenge.
What this means in practice
- •For energy grid operators: Assess how future changes in external factors like weather or demand affect power grid performance under novel conditions.
- •For transport planning teams: Evaluate traffic forecasting models’ ability to predict outcomes of new traffic interventions or policy changes before deployment.
Tested on simulated data.
Authors
Arjun Ashok, Mihir Parmar, Palash Goyal, Bhavana Dalvi Mishra, Chun-Liang Li, Irina Rish, Alexandre Drouin, Jinsung Yoon, Tomas Pfister
Abstract
Recent time series foundation models (TSFMs) can consume covariates in-context, extending zero-shot forecasting to problems driven by external factors. This enables scenario analysis: forecasting how a target responds to events that set covariates to chosen future values. Whether they can do this reliably is an open question. We introduce WhatIf-Bench, a benchmark measuring how well forecasters use covariates to forecast a target's response to future events. WhatIf-Bench is semi-synthetic: time series come from mechanistic simulators of data centers, power grids, road traffic, and water networks, with controlled covariate-target relationships grounded in each system. Each task presents a history of events intervening on covariates, then asks to forecast a future event that either replays seen values (Replay) or sets them to unseen ones (Novel). Varying the number of covariates, historical events, and event length yields 37,665 tasks. Evaluating TSFMs, frontier LLMs, and simple regression baselines, we find the LLMs reach the highest skill (improvement over a naive baseline), ahead of the best TSFMs. Every model forecasts well on Replay futures (48 to 85% skill) but drops to -7 to 50% on Novel futures, with the regression baselines falling the most. Skill on Novel futures also falls as events set more covariates jointly. More historical events raise skill on Novel futures across the board, but large differences remain. WhatIf-Bench reveals generalization to Novel futures as the core challenge, and offers a testbed toward arbitrary scenario analysis.