Time series models with similar accuracy produce different forecasts
Temporal Predictive Multiplicity: Equally Accurate Time Series Models Yield Different Forecast Trajectories
Machine LearningArtificial Intelligence
Summary
Sometimes, different time series models can predict equally well but give very different future outcomes when you look at their full predictions over time. This means just checking accuracy at each time step doesn’t show you how forecasts might really disagree as a whole. The authors studied these differences using many models and datasets, showing that even equally good models can suggest very different future trends. This finding is important because decisions based on these forecasts could vary depending on which model is chosen.
What this means in practice
- •For financial analysts: Evaluate investment risks by comparing entire forecast trajectories from multiple models, not just per-time-point accuracy.
- •For supply chain planners: Use trajectory-level comparisons of demand forecasts to better understand variability and plan stock levels more robustly.
Authors
Emanuele Albini, Francesca Toni, Saumitra Mishra, Francesco Leofante
Abstract
Models with near-identical predictive performance can yield substantially different predictions, a phenomenon known as predictive multiplicity. Prior work has mostly studied this at the level of individual scalar outputs. In time-series forecasting, however, predictions across horizons jointly define a trajectory, and horizon-wise comparisons can hide important differences in predictive behavior. To address this problem, we introduce temporal predictive multiplicity, a framework that characterizes disagreement over complete forecast trajectories among models with near-identical predictive performance. We show that constraining predictive performance alone can still admit a broad range of different trajectories. We further show that constraining multiplicity at individual horizons partially reduces, but does not eliminate, trajectory-level multiplicity. Experiments with 19 neural forecasting architectures on 11 datasets confirm that near-optimal models can exhibit substantial variability in the forecast trajectories they produce, and trajectory-level disagreement is largely unrelated to horizon-wise disagreement. Our framework, therefore, exposes a gap in existing multiplicity studies: models with indistinguishable predictive performance imply fundamentally different temporal trajectories, with consequential downstream effects.