Progressive memory transformer improves multiscale time series analysis
Progressive Memory Transformer: Memory-Aware Attention for Time-Series
Machine Learning
Summary
Time-series data, like stock prices or heartbeats, have important patterns at different levels, from small details to big trends. Many existing methods only look at the whole series or small parts, missing the middle-range features. The authors propose a new approach that teaches a model to recognize patterns at three levels: local, mid-range, and global. To do this, they designed a special transformer with a memory that keeps track of middle-sized windows of data. Their method performs well on various tasks like classifying data with few labels and forecasting future values.
What this means in practice
- •For data scientists: Develop models that capture local, mid-range, and global patterns in time-series data for improved classification and forecasting tasks.
- •For financial analysts: Improve mid-term pattern recognition in financial time-series for better predictive insights in market trends and asset behaviors.
Authors
Tord Sture Stangeland, Andreas Köhler, Steffen Mæland, Adín Ramíres Rivera
Abstract
Time-series carry structure simultaneously at multiple scales (fine-grained variation, mid-range motifs, and global properties) and downstream tasks operate at correspondingly different scales. Most existing self-supervised learning approaches supervise representations globally via instance-level contrastive losses and limited temporal neighborhood supervision, but do not explicitly exploit the structural hierarchy. We propose a learning framework that explicitly enforces a structural hierarchy across three scales independently: a local objective for token continuity, a mid-range objective for window-level motifs, and a global objective for sequence-level agreement. Realizing this framework requires the backbone to expose a representation at each scale; we introduce \textbf{Progressive Memory Transformer} (PMT), which augments a transformer with writable, window-aligned memory that exposes the mid-range scale alongside the token and sequence-level representations conventional transformers already provide. Across seven UCR/UEA/UCI classification benchmarks, a cue-retention probe, and forecasting benchmarks, PMT learns representations that probe well at the global, mid-range, and local scales---strong low-label classification (1--5\% labels), competitive forecasting performance across multiple horizons, and quantitative and qualitative evidence that memory states capture mid-range motifs.