Language model stability and memory horizons explained
Separating Stream Stability from Long-Term Recall in Language Models
Computation and Language
Summary
Language models that generate text over long streams can stay stable in their output even when they forget details from earlier parts. The authors point out that stability in generating text is different from remembering past content or maintaining task performance over time. They define three 'horizons' to measure these factors separately: how long the output stays stable, how far back the model can access content, and how long tasks perform well. Their tests show a model can be perfectly stable but still forget older information unless additional memory mechanisms are used.
What this means in practice
- •For language model engineers: Optimize streaming text generation systems by separately measuring stability and recall to improve model robustness and memory use.
- •For chatbot developers: Design chatbots that maintain stable conversation flow even when long-term memory is limited, by leveraging attention sink mechanisms.
Authors
Peipei Cao, Xin Zhang, Jie Tang, Xiao Li, Siying Li, Qing Pei
Abstract
Methods for streaming language models are often discussed alongside long-context and memory systems, although they solve different problems. An attention sink can stabilize autoregressive generation over an indefinitely long stream while the model remains unable to use content that has left its recent-token cache. We argue that this distinction should be explicit in system claims and evaluation. We introduce three horizons: the stability horizon, over which predictive behavior remains well behaved; the access horizon, over which past content can still causally affect the output; and the utility horizon, over which a task retains acceptable performance. We show constructively that the stability horizon can be infinite while the access and utility horizons are finite. We then propose ThreeH, an evaluation contract that measures all three horizons under a common state and compute budget. Applying the framework to attention-sink streaming clarifies its strength, constant-memory, stable generation, without treating anchor tokens as semantic memory. The framework exposes roles for cache policies, recurrent state, retrieval, and external memory. Experiments on 128K-token streams, delayed binding recall, and delayed decisions show that attention sinks preserve local modeling but not content beyond the active cache; recurrent and retrieval state extend the semantic horizon.