Self evolving agents improve consistency in repeated language tasks
Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course
Artificial Intelligence
Summary
Sometimes AI agents that use big language models like GPT-4 can get answers right most of the time but still change their minds on the same task when trying it multiple times. This inconsistency can be a problem if we want AI agents to be reliable and trustworthy. The authors created a method where the AI learns from its past mistakes by remembering where it was unsure and creating guidelines to avoid those mistakes in the future. This approach helped the AI become more consistent on repeated tasks and when facing similar new tasks.
What this means in practice
- •For software developers: Improve reliability of AI-powered assistants by reducing answer inconsistency across repeated queries.
- •For customer support teams: Deploy more consistent AI chatbots that maintain stable responses to repeated customer issues.
Authors
Evelyn Duesterwald, Benjamin Elder, Lilian Ngweta, Shashanka Ubaru, Malgorzata Zimon
Abstract
Large language model (LLM)-powered agents can be accurate on average yet unreliable in production, a discrepancy that has been observed but remains largely unaddressed. When given the same task five times, a ReAct agent on the AppWorld benchmark using GPT-4.1 succeeds in all five runs only 53% of the time, even though its per-run pass rate averages 77%. We call this 24-point shortfall the consistency gap, and we argue that addressing it is a precondition for trustworthy AI agent deployment. We present a self-evolving agent framework that reduces this gap by identifying unstable, low-consistency steps in agent trajectories and converting them into episodic memory the agent can draw on in future runs. At its core is a Consistency Analyzer that pinpoints where and why a trajectory is likely to flip across executions, and a Guideline Generator that converts the diagnosis into targeted guidelines, committed to memory and injected into future agent executions on similar tasks. On AppWorld with ReAct/GPT-4.1, our framework raises the fraction of tasks that succeed in all five runs by +16 points on same-task evaluation and +13 points on similar-task generalization.