AI agents use memory just in time to boost task accuracy and speed
Just-In-Time Agent Memory with Runtime Agentic Research
Computation and LanguageArtificial IntelligenceInformation RetrievalMachine Learning
Summary
AI agents need to remember past information to make good decisions, but traditional methods keep memory ready all at once before a request, which can lose important details. The authors created a new way for AI agents to build and search their memory only when a specific question arrives, saving effort and capturing more relevant facts. They developed tools to train and test this system across many tasks, showing it works better and faster than older memory methods. Their open-source code can help others build smarter AI agents.
What this means in practice
- •For software engineers building chatbots: Improve chatbot responses by dynamically retrieving relevant conversation history only when needed to increase accuracy and reduce computational cost.
- •For developers of intelligent personal assistants: Enable assistants to selectively recall detailed past user interactions at runtime, enhancing personalization without preloading extensive memory.
Authors
Bingyu Yan, Chaofan Li, Hongjin Qian, Shuqi Lu, Chaozhuo Li, Zheng Liu
Abstract
Memory is critical for AI agents. Many existing agent-memory systems follow an Ahead-of-Time (AOT) design, constructing memory before a specific request arrives. While this reduces online serving cost, such request-agnostic memory construction can discard fine-grained information that later becomes important. To address this limitation, we propose Just-In-Time Agent Memory (JAM), a trainable framework for query-conditioned context construction at runtime. A Memorizer preserves complete raw histories in a hierarchical page-store with compact navigational summaries, while a Researcher iteratively retrieves, inspects, and integrates evidence for each request. To train these memory-use behaviors, we introduce Memory-Gym, an evidence-grounded data synthesis pipeline covering nine task types across six domains, and optimize the Researcher through verified-trajectory supervised fine-tuning followed by Hint-guided Group Relative Policy Optimization. We demonstrate the effectiveness of JAM across a variety of benchmarks on agent memory and long-context processing, where it achieves stronger task performance than AOT-style memory systems while remaining substantially more efficient than prior trained agentic memory approaches. To support reproducibility and future research, we release our anonymized source code at https://github.com/VectorSpaceLab/general-agentic-memory.