Loss scores guide data selection to improve time series model training
Loss-Guided Pretraining Data Selection for Time-Series Foundation Models
Machine LearningArtificial Intelligence
Summary
Training large time series models often involves using all available data without checking if each piece helps learning. The authors propose scoring parts of the data by how hard they are to predict using a simple model, then selecting a balanced subset that is more informative for training. This method helps models learn better and faster, even outperforming training on all the data. They also find that a small reference model can pick good data for bigger models if their difficulty rankings match.
What this means in practice
- •For data science teams: Select a small, diverse set of time series windows that improve model accuracy and reduce training costs compared to random sampling.
- •For financial forecasting teams: Improve prediction model training by prioritizing time series data segments that offer more learning signal, boosting model quality with less data.
Authors
Yike Li, Shaoxu Song, Jianmin Wang
Abstract
Time series foundation models (TSFMs) are pretrained on heterogeneous collections containing billions of observations, yet their training windows are typically sampled without estimating whether they provide useful learning signal. We introduce a static data-selection framework that scores each window with a reference forecaster and retains an intermediate interval within every source dataset. Specifically, we connect forecasting loss to optimization difficulty by showing that normalized squared loss controls the per-sample gradient norm under a local Jacobian condition. We then define a reference loss score and apply dataset-stratified selection to preserve the diversity of samples. Across various TSFM architectures, our method outperforms random selection by an absolute margin and even improves both relative MASE and CRPS over full-data pretraining by retaining fewer candidate pretraining windows. Further analyses show strong cross-scale and cross-architecture score correlations, indicating that a small reference model can often select data for larger targets, provided that the reference and target share compatible difficulty orderings.