Persistent memory adds cache cost but no accuracy gain in multi-agent llm inference
Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
Artificial IntelligenceComputation and LanguageDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Large language models (LLMs) can split long tasks across multiple agents to save memory, but adding a layer that remembers past reasoning doesn’t always help accuracy. The authors tested a system with three memory layers and found that while splitting tasks reduced memory use, the extra persistent memory used a bit more but didn’t improve correctness. They explain this is because the test tasks treated every question independently, so recalling past information wasn’t useful. The study also highlights subtle measurement errors that can make memory seem more helpful than it really is.
What this means in practice
- •For llm system engineers: Avoid adding persistent memory layers when the tasks are independent questions, as they increase cache requirements without improving accuracy.
- •For benchmark designers: Design multi-question benchmarks that enable meaningful evaluation of persistent memory use in agent systems by providing shared evidence across queries.
Authors
Hochan Son, Kyungdoe Han, Jaehan Koh, Xiaowu Dai, Wenlu Xu, Guang Cheng
Abstract
Decomposing long-context inference across cooperating agents bounds the active KV cache per call rather than total evidence, which matters when KV-cache memory binds. Many such systems add a persistent tier storing and recalling reasoning traces, usually validated by an ablation reporting an accuracy gain. We measure both on one three-tier agent architecture. Decomposition delivers: peak KV working set of 14.3 MiB per query against 35.5 and 35.3 MiB for single-pass and retrieval-augmented baselines. The persistent tier does not: across eight controlled dataset pairs at n=100 per arm it costs +0.368 MiB [+0.167, +0.590] of peak cache and produces no detectable accuracy change (+0.015, 95% CI [-0.011, +0.046]). We argue the null is structural: single-question benchmarks supply each item with its own evidence and score it independently, and correctness requires resetting stored traces between conditions, so recall has nothing informative to retrieve. Reaching it took four measurement corrections -- three inflating the apparent benefit, the fourth making an effect that size look resolvable -- none visible in the results table. We give the conditions an agent-memory ablation must satisfy and detection procedures that need no knowledge of the specific defect.