ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL
2026-06-01 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors present ReSkill, a new approach that helps reinforcement learning agents create and improve reusable skills while learning a policy. Unlike previous methods that separate skill creation from learning, ReSkill updates skills based on past failures and tests different skill versions together with the policy. They use special techniques to decide which skills to keep or change as the agent gets better. Their experiments show ReSkill leads to better performance, especially on new tasks, by allowing skills and policies to evolve together smoothly.
reinforcement learningmodular skillspolicy optimizationGRPOThompson Samplingskill creationskill refinementskill pruningagent learningexploration-exploitation
Authors
Zelin He, Haotian Lin, Boran Han, Wei Zhu, Haoyang Fang, Bernie Wang, Xuan Zhu, Runze Li, Matthew Reimherr
Abstract
Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.