Memory size matters differently for strong and weak AI agents
How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents
Artificial Intelligence
Summary
AI agents that handle long tasks often make repeated mistakes, like mixing up steps or giving wrong answers. The authors created a method where the AI learns from its own past mistakes by turning them into lessons and then into shorter guidelines. They found that how much memory these agents should use depends on how capable the AI model already is. Strong AI models get better when given many guidelines, while weaker ones perform best with fewer, carefully chosen guidelines. The paper shares practical advice based on this experience for teams building AI agents that remember past actions.
What this means in practice
- •For software engineering teams: Improve AI agents’ memory use by tailoring guideline quantity to model strength, enhancing task reliability in complex workflows.
- •For automation platform developers: Integrate automatic lesson distillation to refine deployed AI assistants and reduce repeated procedural errors across long sequences.$Commercial implications: Enables improved AI assistant products with adaptive memory management, increasing task success rates and user satisfaction.
Authors
K. R. Jayaram, Vatche Isahagian, Aswath Vaithinathan Aravindan, Vinod Muthusamy, Punleuk Oum, Gaodan Fang, Michal Ulewicz, Gegi Thomas, Evelyn Duesterwald
Abstract
LLM agents on long-horizon tasks fail in recurring, procedural ways. They mis-paginate an API, resolve the wrong entity, or answer when no answer was asked. We built a pipeline that lets an agent learn from its own history. It first mines short \emph{lessons} from the agent's own trajectories, one to three per episode. It then consolidates those lessons into a smaller set of \emph{guidelines}, where each guideline records how many lessons it merged. At inference the agent receives either all guidelines or a selected subset. The pipeline ships in the open-source altk-evolve library and is deployed in IBM's CUGA agent and watsonx Orchestrate platform. We report what we learned building, measuring, and productizing it. Our primary instrument is the AppWorld benchmark, across models spanning 40--92\% baseline task completion. The central finding is that the right amount of memory is a property of the model, not the memory. A strong model keeps improving as guidelines are added. DeepSeek-V3.2 gains +9.5 points of task completion and +16.1 points of scenario completion with its full 191-guideline set. A weak model gains less from the same full set than from a small selected subset. gpt-oss-120b starts at 39.9\% task completion. It reaches 51.2\% with all 199 of its guidelines but 55.4\% (mean of two runs) with a selected subset of about 29. Around that result we report what went wrong: a high-support guideline that backfired, provenance silently dropped during consolidation, a swallowed API error that failed every task while producing a plausible log, and a baseline confound that nearly produced a misleading comparison with a prior method. We distill these into recommendations for teams deploying agent memory.