Language models need different tools for stable and long-term memory
Separating Stream Stability from Long-Term Recall in Language Models
Computation and Language
Summary
Language models sometimes seem good at remembering long conversations, but they actually use different methods to stay stable and to recall old information. The authors explain that staying steady while predicting words (stability) is not the same as remembering content over time (recall). They introduce a way to measure three types of memory: how long the model stays stable, how long it can access past information, and how long it can perform a task well. Their tests show that models can be stable forever but only remember recent content, meaning other memory strategies are needed for true long-term recall.
language modelautoregressive generationattention sinkstability horizonaccess horizonutility horizonlong-term memorycacherecurrent stateretrieval
Authors
Peipei Cao, Xin Zhang, Jie Tang, Xiao Li, Siying Li, Qing Pei
Abstract
Methods for streaming language models are often discussed alongside long-context and memory systems, although they solve different problems. An attention sink can stabilize autoregressive generation over an indefinitely long stream while the model remains unable to use content that has left its recent-token cache. We argue that this distinction should be explicit in system claims and evaluation. We introduce three horizons: the stability horizon, over which predictive behavior remains well behaved; the access horizon, over which past content can still causally affect the output; and the utility horizon, over which a task retains acceptable performance. We show constructively that the stability horizon can be infinite while the access and utility horizons are finite. We then propose ThreeH, an evaluation contract that measures all three horizons under a common state and compute budget. Applying the framework to attention-sink streaming clarifies its strength, constant-memory, stable generation, without treating anchor tokens as semantic memory. The framework exposes roles for cache policies, recurrent state, retrieval, and external memory. Experiments on 128K-token streams, delayed binding recall, and delayed decisions show that attention sinks preserve local modeling but not content beyond the active cache; recurrent and retrieval state extend the semantic horizon.