Recurrent models can learn to remember much longer with new training tricks
Learning Length-Extrapolatable Recurrent Models
Machine LearningComputation and Language
Summary
Recurrent neural networks are designed to understand sequences over time but usually struggle to remember important information beyond a certain length they are trained on. The authors found that this problem is not just due to weak signals fading away but is related to how future learning signals are passed back to earlier steps. They introduced a new method called Credit Stabilization through Time, which carefully adjusts these signals during training to keep them stable. This adjustment helps the model remember for much longer, improving performance even when looking far beyond its original training length.
Recurrent neural networksBackpropagation through timeGradient vanishingGradient explodingState creditCredit stabilizationSequence modelingLong-context learningTemporal dependenciesParameter updates
Authors
Hanwen Jiang
Abstract
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.