LLMs recall facts in huge texts using less memory
Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size
Artificial IntelligenceComputation and Language
Summary
Large language models struggle to remember information in very long texts because they use more memory as the text grows. The authors present a way for these models to recall specific facts from massive documents without needing extra memory or retraining. They do this by using hidden signals inside the model’s own layers, which keep track of facts in a compact form. As a result, the model can answer questions about documents millions of words long when previous methods could not.
What this means in practice
- •For machine learning engineers: Build systems that handle extremely long documents for answering questions without increasing memory requirements.
- •For chatbot developers: Create chatbots that maintain context over very long conversations without needing larger models or retraining.
Authors
MyungHoon Ryu, XinYu Piao, Jong-Kook Kim
Abstract
Large language models (LLMs) process long contexts, including long documents and lengthy conversations, but face token-level memory usage that increases proportionally to input length. Although model optimization and lossy prompt compression are widely used, these methods still fail to solve the long-context recall problem beyond pretrained and size-constrained context windows. This paper proposes a long-context recall method that maintains near-constant GPU memory usage as context length increases, without additional training. The main idea is to reconstruct facts using parameter activations in the LLM's feed-forward layers, which store residual vectors representing facts from the source document. Utilizing residual vectors allows the LLM to deterministically reconstruct query relevant facts without referencing the original document, preserving high fidelity and reducing memory usage without fine-tuning weights. Experimental results show that the proposed method enables answering single-fact questions in two-million-token story contexts where previous methods fail.