Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents
2026-08-31 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how different types of information in an agent's working memory, like instructions or tool outputs, behave differently in how long they stick around and how they can be compressed. They found that memory management should consider these differences, not just treat all information the same. They tested two ways to manage memory based on the meaning of the info but found that improvements in one task don’t always help in others, and simply having the same memory size limit doesn’t guarantee the same effectiveness. Their findings highlight the importance of understanding the meaning of stored memory when designing and evaluating agent memory systems.
working memorysemantic heterogeneitymemory managementcoding agentscompression policyretrieval policyretention behaviortoken budgetcontext deliverymemory evaluation
Authors
Le Chen, Zishen Wan, Baixi Sun, Xiaolong Ma, Chih-Hsuan Yang, Feng Yan, Sheng Di, Franck Cappello, Rajeev Thakur
Abstract
Agent working memory is heterogeneous. Objects such as instructions, artifacts, tool outputs, and agent-generated state play different semantic roles and exhibit different size, retention, and representation profiles. Recent work has begun to explore memory-management mechanisms that account for such heterogeneity. This work focuses on semantic heterogeneity and studies how it should shape the management and evaluation of working memory in coding agents. Across 55 archived coding-agent trajectories, we find that semantically different working-memory objects exhibit distinct retention and compression behavior. This heterogeneity motivates semantically informed memory management. We study two semantically informed strategies: an object-aware compression policy and a retrieval-based policy. Their evaluation shows that calibration gains may not transfer to held-out tasks, and that equal token budgets do not imply equal delivered context or management cost. A real-system replay further exposes serving limits that nominal budgets alone do not capture. Together, these results show why semantic structure matters for agent working memory and why evaluating memory-management strategies requires more than a nominal token budget. We organize these lessons into four levels: stored state, delivered context, management work, and task or process outcome.