Memory management improves long context use in large language models
From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models
Computation and Language
Summary
Handling very long texts is hard for large language models because reading and remembering everything takes too much computing power and memory. The authors collected and organized existing ideas about how these models store and recall parts of their memory during use. They show it’s important to decide what information should be actively used versus stored away and retrieved later, rather than just trying to squeeze in as much as possible. Their framework treats memory like managing resources with costs and helps unify many different approaches to making large models handle long texts better.
What this means in practice
- •For ai infrastructure teams: Design more efficient systems for serving large language models that handle long inputs while controlling compute and memory costs.
- •For cloud platform operators: Implement lifecycle-aware memory policies to improve scalability and responsiveness of large language model APIs for users requiring extended context use.
A survey. It maps existing work.
Authors
Chaogui Gou
Abstract
Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, efficient serving, retrieval augmentation, and agent memory is usually discussed in isolation, leaving the flow of information between storage and computation insufficiently understood. This survey reframes long-context compression as a memory-management problem for allocating inference-time resources. We distinguish compute-active memory, which directly participates in model computation, from compute-inactive memory, which lies outside the computation path but can be stored, retrieved, updated, or forgotten, and we characterize offloaded KV caches, persistent prefix caches, and precomputed KV states as intermediate states on the activation path. On this basis, we formulate writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting as state transitions within an information lifecycle, and introduce a budgeted cost model that places these operations in a single cost account. The resulting framework unifies short-term context compression, long-term memory management, and the transitions between them, and shows that the central challenge is not merely reducing information volume but deciding which information should participate in costly inference, when, and in what form. We further summarize evaluation dimensions, benchmark gaps, and open challenges, and argue that future research should move from static, pointwise compression toward dynamic, governable, lifecycle-aware memory management.