Time series models improved by preserving meaning and structure during fine tuning
SIFT: Enhancing Time Series Foundation Models via Semantic Invariance and Structural Fidelity Fine-Tuning
Machine Learning
Summary
Time series models process sequences of data over time but can struggle to learn well when adjusted for new tasks, often making dull average guesses. The authors propose a method called SIFT that helps these models stay true to the important meanings and structures in the data while fine-tuning. They use special techniques to add minor changes that don't change meaning but make the model stronger and to keep the parts of the data intact during learning. Their tests on many real datasets show this method helps the models perform better.
What this means in practice
- •For data science teams: Improve predictive accuracy of time series models by fine-tuning without losing important data meanings or structures.
- •For financial analysts: Enhance forecasting models for stock prices or economic indicators by preventing naive fine-tuning pitfalls in time series data.
Authors
Yi Tang, Tengxue Zhang, Yang Shu, Chenjuan Guo, Chenchen Sun, Yisheng An
Abstract
Time Series Foundation Models (TSFMs) have achieved remarkable zero-shot performance through extensive pre-training on massive time series datasets. Nevertheless, due to the low-dimensional properties and diverse structural patterns of time series data, performing naive fine-tuning on TSFMs often leads to overfitting and falling into the mean-prediction trap. To address these challenges, we propose SIFT, a robust adaptation method that enhances time series foundation models by preserving Semantic Invariance and structural Fidelity throughout the fine-Tuning process. We employ semantic-invariant adversarial augmentation, which utilizes semantic spectrum decomposition to partition the semantic space and then generates perturbations within the non-core semantic subspace to bolster the model's robustness against these perturbations, mitigating overfitting. We implement a component-based structural fidelity enhancement, which facilitates component-wise mixup and imposes a reconstruction objective to improve the model's ability to preserve structural fidelity, alleviating the mean-prediction trap. Extensive experiments on representative TSFMs covering 10 real-world datasets demonstrate that SIFT can significantly enhance the performance of TSFMs.