Fractional memory models improve handling of long sequences
Fractional State Space Transition for Long Sequence Modeling
Machine LearningComputation and Language
Summary
Models that remember past information usually forget things too quickly, which makes it hard to handle very long sequences of data well. The authors designed a new model called FRAC that remembers past information for a longer time by using a special kind of memory called fractional dynamics. They made this memory efficient and practical by combining many simpler memories spaced in a smart way. Their tests show FRAC works better than similar models for long sequences while still doing well on short ones.
What this means in practice
- •For language model engineers: Build language models that keep track of long text context more effectively to improve understanding of lengthy documents.
- •For time series analysts: Use fractional memory models to better capture long-range dependencies in time series data for forecasting or anomaly detection.
Authors
Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi, Lifeng Shang, Yufei Cui
Abstract
State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.