Time series models train faster using new synthetic data loss method
Distillation of Synthetic Data for Time Series Foundation Models
Machine Learning
Summary
Training computer models to predict data over time usually involves comparing their guesses to exact future values. The authors suggest instead comparing model outputs to the full range of possible future values based on known data patterns. This new training approach called synthetic data distillation helps models learn faster and more efficiently. They tested this on many model sizes and saw quicker improvement in accuracy.
What this means in practice
- •For machine learning engineers: Reduce training time for time series prediction models using improved loss functions based on synthetic data.
- •For financial model developers: Speed up development of forecasting models by training on synthetic financial time series with reduced iterations.
Authors
Niloy Biswas, Noureddine El Karoui
Abstract
Time series foundation models (TSFMs) are increasingly pre-trained on synthetically generated time series trajectories, where the data generating process is known. Current pre-training recipes are based on loss objectives which compare TSFM outputs to realized future values of each trajectory. We instead propose loss objectives which compare TSFM outputs to the conditional forecast distribution of each trajectory, a procedure we call synthetic data distillation (SDD). SDD corresponds to a Rao-Blackwellization of the training objective, in that it leaves the expectation of stochastic gradients unchanged while provably reducing the covariance of the stochastic gradient under the Loewner partial ordering. We empirically validate SDD on a TSFM model family of sizes from $4$M to $2.5$B parameters, and observe faster convergence of validation loss at every model size: on Gaussian Process data, SDD attains or improves upon the Status Quo loss whilst requiring $10\%-40\%$ less training iterations.