Summary
Time-series forecasting predicts future values from past numerical data, which is different from the text language models are designed to understand. The authors show how to adapt a text-based language model, GPT-2, to forecast time series by projecting chunks of numerical data into the model’s input format without turning numbers into text. They find this method works better than converting numbers to text and that only updating a small part of the model leads to efficient and accurate forecasts on several real-world datasets. This approach keeps most of the pre-trained language model unchanged, saving computing resources while maintaining good accuracy.
What this means in practice
- •For energy analysts: Improve energy consumption forecasts by adapting pretrained language models with minimal retraining for accurate and efficient predictions.
- •For financial modelers: Use language model adaptation techniques to enhance univariate financial time-series forecasting while reducing computational costs.
Abstract
We study the adaptation of pretrained language models to univariate time-series forecasting through a parameter-efficient transfer learning framework, with the goal of understanding which design choices drive effective cross-modal transfer. While language models operate on discrete textual tokens, time series consist of continuous numerical observations with temporal dependencies. To bridge this modality gap, we project fixed-length time-series patches directly into the embedding space of a pretrained GPT-2 backbone, bypassing textual tokenization and treating the Transformer as a generic sequence encoder. Through controlled ablation studies on seven benchmark datasets spanning energy, weather, traffic, and finance, we analyze the effects of (i)~representation strategy (continuous embeddings versus textual serialisation), (ii)~adaptation regime (frozen backbone versus partial or full fine-tuning), (iii)~architectural components such as adapters, pooling strategies, and prediction heads, and (iv)~input context length. Continuous patch-based embeddings consistently outperform textual prompting and randomly initialised backbones. The adapted pipeline attains MASE within the range of specialised forecasting architectures while updating less than 1\% of total model parameters. Results further indicate that freezing the pretrained backbone and training lightweight projection and adapter modules provides a favourable accuracy--efficiency trade-off with stable behaviour across varying context lengths.