Agent continual learning improves multi-stage capability integration
ACLArena: Agent Continue Learning in Multi-stage Post-training
Artificial Intelligence
Summary
Building intelligent agents that can learn multiple skills step-by-step is hard because learning new things often makes agents forget old skills. The authors created ACLArena, a tool to study how agents can keep old skills while learning new ones. They tested different ways to combine skills and found that using offline replay with specialized small networks helps agents learn better across many tasks. Their experiments show this approach works well both when tasks are familiar and when agents face new types of challenges.
What this means in practice
- •For industrial ai developers: Build AI agents that retain previously learned skills while acquiring new capabilities across multiple deployment stages.$Commercial implications: Enables creation of versatile AI agents for different industrial tasks, improving product offerings requiring adaptable AI solutions.
- •For robotics engineers: Create robots that continuously learn and adapt to new tasks without losing proficiency in previously trained behaviors.
Authors
Haixin Wang, Xiaoxuan Wang, Junkai Zhang, Han Zhang, Renliang Sun, Alexander K Taylor, Yidan Shi, Haoran Deng, Chenguang Wang, Jason Cong, Yizhou Sun, Wei Wang
Abstract
Building general-purpose agents for industrial deployment requires integrating multiple capabilities, each typically acquired at a distinct stage of training. Yet there is currently no well-established recipe for Agent Continual Learning (ACL), with little understanding of the trade-offs among existing integration paradigms. To address this gap, we introduce ACLArena, a framework for comprehensively studying, analyzing, and evaluating ACL. We first build a sequential training pipeline and conduct an in-depth analysis that explains the mechanisms of forgetting and generalization from two complementary perspectives, the model level and the token level. Guided by these analyses, we systematically compare multi-teacher on-policy distillation, self-distilled fine-tuning, and model merging to assess their ability to recover previously learned capabilities while preserving newly acquired ones. Through extensive experiments, we develop a detailed understanding of how capabilities transfer across stages. Finally, we propose a new ACL recipe that combines offline replay over high-quality trajectories with a routed network of multiple LoRA experts each specialized via RL, substantially improving the agent's ability to learn across multiple domains. Comprehensive experiments on four reasoning and agentic tasks, evaluated under both in-domain and out-of-domain settings, demonstrate the value of our analysis and the effectiveness of our approach.