Spectral predictability alone does not ensure synthetic pretraining success
Is spectral predictability enough? Testing Ω-matching for synthetic pretraining under distribution shift
Machine Learning
Summary
When training AI models with made-up (synthetic) data, it helps if the data shares certain predictable patterns with real data. The authors looked at whether matching a measure called spectral predictability alone is enough to pick the best synthetic data for training. They found that while it sometimes helps in simple settings, it doesn’t reliably predict success when using real-world data. This means other qualities of the data also matter when deciding what synthetic data to use for training AI models.
What this means in practice
- •For machine learning engineers: Select better synthetic datasets for pretraining models under distribution shifts by considering factors beyond spectral predictability.
- •For time series analysts: Improve forecasting by recognizing that spectral similarity alone does not guarantee beneficial transfer from synthetic to real data.
Authors
Edric Castel C. Hao, Juan Raphael S. Seña, Neon S. De Jesus
Abstract
Synthetic pretraining under distribution shift requires deciding what synthetic data to use. A natural criterion is to match the spectral predictability $Ω$ of the source to the target, but is $Ω$ sufficient? We test this hypothesis, which we call predictability-aware pretraining, across three architectures (LSTM, PatchTST, TCN) and three seeds. Under ideal synthetic-to-synthetic conditions, $Ω$-matched pretraining can outperform even an oracle baseline (+46.8% MSE reduction, LSTM), though the benefit is already architecture-dependent. On the real-world ETTm1 benchmark, however, this relationship does not hold: sine-plus-noise pretraining at matched $Ω$ yields only a marginal mean improvement +2.3% averaged over all three architectures, up to +65.5% on individual channels), whereas $Ω$ matched chaotic ODE trajectories systematically degrade performance -13.2%). The two sources agree on the direction of transfer for only 53% of configurations across all three architectures (Pearson r=-0.02), indicating that $Ω$ carries little information about which source will transfer beneficially. We further observe an architecture-dependent asymmetry in the response to the direction of the $Ω$ gap. We conclude that $Ω$ matching can help in controlled settings but is not sufficient to predict transfer across sources, and we discuss additional signal properties that a source-selection criterion may need to consider.