Policy and skills evolve together to improve AI agent success rates
RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
Computation and Language
Summary
Long tasks with many steps are hard for AI agents to learn because it’s tricky to figure out which actions help in the end. The researchers created a new method called RSIPO, where the AI not only learns a policy for actions but also develops skills from its experiences as it trains. This method also helps the AI decide how much to trust feedback from a teacher model, which sometimes can be misleading. Their tests show RSIPO helps AI agents succeed more often than previous methods on tasks involving interaction and search.
What this means in practice
- •For robotics developers: Enhance robot decision-making for complex tasks by enabling simultaneous skill extraction and policy improvement from interactions.
- •For game ai engineers: Improve agent behavior in games requiring extended sequences of actions by evolving skills and policies together during training.
Authors
Fan Zhang, Hao Wu, Jinyang Wu, Hongyu Qu, Zheng Lian, Xiao Luo, Yefeng Zheng, Bo Zhang, Haoxuan Li, Pheng-Ann Heng
Abstract
Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6\% on ALFWorld, yielding absolute improvements of 2.5\% and 17.4\% over the second-best baseline and standard GRPO, respectively.