Reinforcement learning improves agent skills with self-retiring teacher help
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Computation and LanguageArtificial Intelligence
Summary
Training AI agents that learn from rewards for long tasks can be tricky because they get only a single reward at the end. The authors found that teaching an agent by having it imitate a better teacher agent only works sometimes, and it depends on how far along the training is. They created a new method called RetireOPD where the student agent learns from a skilled teacher but stops copying it once it gets good enough, then trains on its own. This method helped AI agents perform better on challenging tasks compared to traditional approaches and even beat their teacher agents in success rates.
What this means in practice
- •For ai development teams: Improve training methods for AI agents performing complex sequential tasks by using adaptive teacher-student learning to increase success rates.
- •For game ai programmers: Create more capable game agents by integrating self-retiring on-policy distillation to efficiently learn from privileged skill models.
Authors
Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen
Abstract
Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.