Ts2vec foundation model matches state of the art for time series
Turning TS2Vec into a Foundation Model for Time Series Classification
Machine LearningArtificial Intelligence
Summary
Time series classification sorts sequences of data points like heartbeats or stock prices. The authors show that TS2Vec, a 2022 approach that trains fresh for each dataset, can be trained once and used across many datasets like a foundation model. By training on synthetic data and combining two kinds of input views into one model, their new version, TS2Vec-FM, performs almost as well as the best current methods while keeping a simpler convolutional design. This approach works on variable-length inputs without extra processing and doesn’t rely on newer techniques like attention mechanisms.
What this means in practice
- •For data science teams: Use a single pretrained TS2Vec-FM model to classify time series from varied domains without retraining on every new dataset.
- •For industrial monitoring teams: Deploy a compact and robust model for real-time anomaly detection in sensor data streams of varying lengths.
Authors
Vasilii Feofanov, Zhenghan Tai, Andrei Ermakov, Irina Rish
Abstract
TS2Vec is one of the strongest self-supervised methods for time series classification, introduced in 2022. It is designed to be trained anew on every dataset: self-supervised pre-training followed by linear probing. Since then, the field has shifted towards foundation models that reuse one encoder across datasets. Yet TS2Vec offers a compact convolutional architecture that operates directly on temporal samples and accepts variable-length inputs without resampling. We ask whether these properties can be retained when the encoder is pre-trained only once, and show that they can: TS2Vec can essentially be turned into a foundation model. Three changes suffice: pre-train on a synthetic corpus, feed the encoder a raw amplitude view and an instance-normalized shape view of the input through two small stems, and distill one specialist encoder per view into a single encoder. The result, TS2Vec-FM, is a 1.6M-parameter dilated convolutional network that, kept frozen, reaches 83.3% on UCR-128 and 75.3% on UEA-27 under linear probing, within 0.3 pp of MantisV2, the current state of the art, and above TS2Vec trained separately on each target dataset. What we find most interesting is how different TS2Vec-FM is from other foundation models in design: near-parity is reached without attention, without a patch tokenizer, and without intermediate-layer extraction, using an "old-school" convolutional encoder and a two-layer MLP stem.