Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation
2026-06-22 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageSoftware Engineering
AI summaryⓘ
The authors created AFTER, a benchmark with 382 real-world workplace tasks to test how well procedural memory helps AI agents learn reusable skills across different jobs and models. They found that refining the AI's skills even once improved performance by a few points, and combining learning from multiple AI models led to better accuracy than using just one. Some skills worked well across various tasks and AI models, while others were useful only in specific job roles. Their work helps guide how to build and test AI systems that remember and reuse procedures in practical settings.
procedural memorybenchmarktransfer learninglarge language modelsworkflow automationcross-task transfercross-role transfermodel generalizationskill refinemententerprise AI
Authors
Julia Belikova, Rauf Parchiev, Evgeny Egorov, Grigorii Davydenko, Gleb Gusev, Andrey Savchenko, Maksim Makarenko
Abstract
Procedural memory is increasingly used to improve LLM agents on recurring workplace tasks, yet its ability to produce reusable skills remains poorly understood. We introduce AFTER, a benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills, designed to evaluate how skills transfer across tasks, roles, and model backbones. The benchmark includes controlled evaluation settings for local improvement, cross-task transfer, cross-role transfer, and cross-model generalization. Experiments show that procedural memory delivers consistent gains in industrial workflows: a single refinement round improves aggregate performance by 3.7-6.7 points, while skills evolved from diverse multi-model execution traces achieve 73.1% cross-model test accuracy, outperforming all single-model trace sources. We further find that some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer. These results provide practical guidance for building, evaluating, and deploying procedural memory systems in production agent platforms.