Self evolving agents improve reasoning by using past versions
Self-Evolve With a Reference:Anchored Training of Tool-Integrated Agents
Artificial Intelligence
Summary
Teaching AI agents to learn from their own generated tasks has challenges because feedback based only on their current abilities can become less useful over time. The authors propose a new method called AnchorLoop, which keeps a fixed copy of a past version of the AI as a reference point. This helps the AI compare its current answers with historical ones, leading to better task selection and learning. Tested on 13 reasoning tasks, this approach improved performance and maintained learning progress better than previous methods.
What this means in practice
- •For ai system developers: Build AI agents that improve over time by referencing past versions to enhance learning without needing external supervision.
- •For automated reasoning teams: Enhance reasoning abilities of tool-integrated agents by stabilizing curriculum learning with historical reference models.
Authors
Wenjie Liao, Liangjie Zhao, Zehong Cao
Abstract
Self-evolving tool-integrated agents learn from tasks and feedback generated within their own training loop. A Curriculum Agent generates tasks, while an Executor Agent learns from self-consistency signals through reinforcement learning. However, relying solely on the current Executor for feedback has two limitations: group-relative advantages vanish under full consensus, while uncertainty-based curriculum rewards favor disagreement without showing whether the generated tasks support further learning. These limitations motivate an additional reference beyond the current Executor. We propose \textit{AnchorLoop}, which introduces a frozen copy of the previous iteration's Executor as a historical reference and reuses it on both sides of the training loop. For the Executor, the anchor provides a cross-reference advantage that evaluates current outputs against both current and historical majority answers. For the Curriculum, it provides an agreement-based reference based on differences in sampled majority agreement. Since the Executor and anchor have identical parameters during Curriculum training, this comparison serves as a proxy for task selection rather than evidence of inter-version improvement or correctness. Across 13 reasoning benchmarks, AnchorLoop improves over Agent0 by 2.5\% on mathematical reasoning and 2.8\% on general reasoning tasks. It also maintains higher effective-advantage variance and continues improving in later iterations as the unanchored baseline shows diminishing gains. These results demonstrate the benefit of introducing a lightweight historical reference into self-evolving tool-integrated agents without external task or answer supervision.