Cache aware object context reduces input cost for long horizon agents
CADOC: Cache-Aware Dynamic Object Context for Long-Horizon Agents
Artificial Intelligence
Summary
Long-horizon agents must remember lots of past information, which slows them down and limits how long tasks can be. The authors show a way to compress this memory by replacing detailed objects with small summary cards that can be expanded when needed. Their method, called CADOC, decides when to update these cards in batches to save computing effort while keeping the original information accessible. This approach cuts data input costs by about 40% without hurting task performance much. It shows that managing memory well involves both compression and smart scheduling of updates.
What this means in practice
- •For ai platform engineers: Optimize memory management for AI agents performing long tasks by reducing redundant data input and maintaining quick access to detailed information.
- •For cloud infrastructure teams: Lower data processing and input bandwidth costs for systems handling continuous agent histories by scheduling efficient batch updates of context data.
Authors
Junjie Yao, Zhangchen Zhou, Zhi-Qin John Xu
Abstract
For a long-horizon agent, context is the bottleneck: the history is resent with every request, the window caps task length, and reasoning degrades as the history grows. Replacing structured objects with compact retrieval Cards shortens the prompt and keeps the exact originals retrievable, but editing the history can break prefix-cache reuse, and prior recoverable methods time their edits by forecasts of future reuse or by preset intervals. We propose CADOC (Cache-Aware Dynamic Object Context), an online algorithm that replaces structured objects with compact Cards while preserving exact, on-demand retrieval of their original contents. CADOC schedules replacements in batches by balancing accumulated waiting cost against shared cache-reconstruction cost. Its scheduling rule follows from an economic order quantity trade-off, recovers the optimal integer batch under stationary assumptions. Across evaluation, CADOC consistently achieves the lowest aggregate input cost among the compared configurations, which reduces input cost by approximately 40\% on average while maintaining task performance close to full context. CADOC thus provides a cost-derived approach to compressible context management, demonstrating that efficient compression depends not only on shortening prompts but also on scheduling edits to preserve cache reuse.