Increasing context size improves memory accuracy more than deeper structures
Memory Depth and Reconstructed Context Width: A Controlled Evaluation of Hierarchical Retrieval
Computation and Language
Summary
Remembering long conversations is important for AI systems like chatbots. The authors studied how two ways of organizing memory affect accuracy: how deep the memory structure goes and how much recent context is given to the AI. They found giving the AI access to a larger chunk of recent conversation helps much more than making memory structures deeper. After a certain point, bigger context helps only up to a limit, but some methods keep high accuracy when looking at the entire memory. This suggests using larger connected pieces of memory instead of more complex hierarchies might be better.
What this means in practice
- •For chatbot developers: Design conversational agents that use larger recent context windows to improve answer accuracy instead of relying on deeper memory hierarchies.
- •For digital assistant engineers: Optimize memory retrieval in assistants by prioritizing wide context input to maintain accuracy over extending memory depth.
Authors
Michael Andreev
Abstract
Long-term conversational memory is becoming an integral component of modern LLM systems. Proposed architectures group records by topics and events, construct hierarchies and graphs, and connect facts through causal and temporal relations. We experimentally study the interaction between two memory parameters: structural depth and the width of context supplied to the answer model. Using EverMemBench, we evaluate depths D1-D4, core budgets of 1,024/2,048/4,096 tokens, and additional Production and Oracle conditions up to the full archive. Increasing width from 1K to 4K improves Accuracy by 10.11-17.98 percentage points, whereas increasing depth provides no monotonic gain. Beyond 8-16K, Production performance reaches a plateau while tokens per correct answer continue to increase; Oracle preserves quality on full archives of 68-71K tokens. These results motivate further investigation of large, coherent context blocks instead of progressively deeper memory structures.