Recurrent models improved to handle much longer sequence lengths
Learning Length-Extrapolatable Recurrent Models
Machine LearningComputation and Language
Summary
Recurrent neural networks struggle to learn from sequences longer than they were trained on. The authors found that the usual ideas about gradient decay don’t fully explain this problem. Instead, they focused on how feedback signals are passed back in time to earlier states, calling this 'state credit.' They introduced a technique that stabilizes these feedback signals during training, without changing how the model processes data forward. This method helps the models perform better on sequences up to 128 times longer than they saw during training.
What this means in practice
- •For machine learning engineers: Train recurrent models that maintain accuracy on much longer sequences than the training data provides, improving performance on long-text or time-series tasks.
- •For speech recognition developers: Build speech models that effectively remember information across very long utterances, enhancing recognition accuracy over extended speech segments.
Authors
Hanwen Jiang
Abstract
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.