Dynamic Dual-Granularity Skill Bank for Agentic RL
2026-03-30 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors propose D2Skill, a new method for reinforcement learning that organizes past experience into two types of skills: big-picture task skills and detailed step skills to help agents make better decisions. Their approach trains both the agent's policy and the skill bank together by comparing normal runs with skill-enhanced runs, using the differences to improve both. The skill bank grows and adapts over time by keeping the most useful skills. Tests showed D2Skill helps agents succeed more often than methods without skills, and the skills work well across different tasks with only small additional training costs.
reinforcement learningskill bankagentic RLpolicy optimizationtrajectoryhindsight utilityskill maintenancetask skillsstep skillstraining overhead
Authors
Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, Dongbin Zhao
Abstract
Agentic reinforcement learning (RL) can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld and WebShop with Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 show that D2Skill consistently improves success rates over skill-free baselines by 10-20 points. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.