SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training
2026-06-01 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors introduce SIRI, a new method that helps long-task language model agents learn useful skills on their own without needing extra tools or stored skill databases during use. Their approach involves three steps: warming up basic abilities, discovering and testing new skills from the agent's own experiences, and then mixing those good skills back into the agent's main model. SIRI improves performance on two tasks compared to other methods and even matches skill learning done with more powerful models. This means agents can become better at complex tasks with less technical hassle.
Long-horizon agentsLarge language modelsSkill discoveryReinforcement learningPolicy distillationIntrinsic skillsTrajectory rolloutsALFWorldWebShopGiGPO
Authors
Zhongyu He, Yuanfan Li, Fei Huang, Tianyu Chen, Siyuan Chen, Xingyang Li, Meng Hsuan Yu, Xiangrong Liu, Leyi Wei, Lu Pan, Ke Zeng, Xunliang Cai
Abstract
Long-horizon LLM agents can benefit from reusable skills, yet existing skill-based methods often rely on external skill generators during training or persistent skill retrieval at inference, increasing engineering complexity, context length, and deployment latency. We propose Self-Internalizing Reinforcement learning with Intrinsic skills (SIRI), a three-phase framework that enables agents to discover, validate, and internalize skills without external skill generators or inference-time skill banks. SIRI first warms up the policy with GiGPO to acquire basic interaction ability and collect successful skill-free trajectories. It then performs self-skill mining, where the current policy summarizes compact skills from its own successful plain rollouts and validates them through paired skill-augmented and skill-free rollouts. Finally, SIRI distills only beneficial skill-guided action tokens into the plain policy using trajectory-level utility and action-level advantage. At inference, the agent runs with the original prompt only. On ALFWorld and WebShop with Qwen2.5-7B-Instruct, SIRI improves GiGPO from 0.908 to 0.930 on ALFWorld and from 0.728 to 0.813 on WebShop, outperforming prompt-based, RL-based, and memory-augmented baselines. Further analysis shows that our self-mining strategy can achieve performance comparable to distillation with closed-source large model. Our code is available at https://github.com/kirito618/SIRI.