Evaluating Generative Time-Series Models on Data with Point Masses
2026-08-10 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors found that many time-series models are tested on data that mostly contains one repeated value, like 'no event'. They show that standard testing methods can give misleading results because the evaluation windows don’t match the actual data’s pattern. They also introduce a test to isolate how much timing influences model scores. When comparing models fairly, a certain type called autoregressive hurdle outperforms others, but results vary a lot depending on how the data occurrence is measured. Different ways of measuring occurrences can even change which model looks best.
generative time-series modelsrolling-origin protocoloccurrence statisticsCRPS (Continuous Ranked Probability Score)autoregressive modelsconditional flow modelstemporal couplingbenchmarkingevaluation metricshurdle models
Authors
Jian Xu
Abstract
Many of the series that generative time-series models are benchmarked on place a large probability mass on a single value --- it does not rain, no ride is requested, no part is ordered. We report what happens when such data is evaluated carefully. First, the standard rolling-origin protocol can score a model on a window whose atom structure bears no resemblance to the dataset: on one benchmark the dataset is $42\%$ zeros and the evaluation windows are $13\%$, on another $47\%$ against $5\%$. This is not a cosmetic problem --- it reversed one of our own conclusions, turning the strongest occurrence model in our study into what looked like a cautionary tale. Second, we give a control in which CRPS is invariant \emph{by construction} while the temporal coupling is destroyed, which measures exactly how much that coupling contributes to a chosen statistic. Third, benchmarking seven models on a matched protocol over five seeds, an autoregressive hurdle beats a conditional flow on five of six datasets, by up to a factor of $153$, while the flow's own occurrence statistics vary by up to $62\%$ across training seeds and every baseline is deterministic. Finally, the model ordering is not the same under five different occurrence statistics, and the two that do not share a construction agree with each other least.