Residual memory tokens improve long text handling in large language models
REMORY: Learning Residual Memory for Context Compaction
Computation and LanguageArtificial Intelligence
Summary
When large language models try to understand or respond to very long texts, they often need to shorten or summarize the earlier parts because they can only handle so much at once. The authors introduce REMORY, a method that adds special extra tokens to remember important details beyond the summary. These tokens help the model act more like it has access to the full original text without needing to process all of it again. This approach improves accuracy across several tests while using far less memory and input space.
What this means in practice
- •For software developers: Improve AI assistants' ability to recall and use long user histories in constrained memory environments.
- •For ai product teams: Reduce errors and repetitions in multi-step AI workflows involving external tools by enhancing memory representations.$Commercial implications: Enables more reliable AI-powered automation and interaction tools that retain long context accurately.
Authors
Hanchen Xia, Baoyou Chen, Yutang Ge, Naihao Deng, Senqiao Yang, Zilong Dong, Weihao Yuan, Siyu Zhu
Abstract
Long-horizon agents compact their history to continue within a finite context window, but a textual summary alone may not support every subsequent decision. We introduce REMORY, a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Given the history and summary, the network learns to generate tokens that help a frozen LLM approximate the continuation it would produce with the full history. The tokens are conditioned on the summary and appended after it, forming an analogue of a residual connection along the sequence dimension. On SummHay, REMORY improves source attribution at nearly unchanged insight coverage and approaches the full-context joint score using only 5.2% of the input positions. Across long-horizon agent benchmarks, Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory. Both models also exhibit substantially fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.