AI summaryⓘ
The authors study how large language models (LLMs) can better use external natural-language skills to solve tasks. They point out that standard training methods focus only on final task rewards, which miss details about how well the model uses these skills. To fix this, the authors propose BCSD, a new training approach that looks at each action sequence from two perspectives—one adding extra high-level guidance and one focusing on task-specific skills—and combines these views to improve learning. Their experiments show BCSD helps LLM agents use skills more effectively across different model sizes. They also confirm through tests that both the added and reduced guidance views are important for the method's success.
Large Language Model (LLM)External SkillsReinforcement LearningSelf-DistillationPolicy LearningSkill UtilizationBidirectional ContextMeta-Skill GuidanceALFWorldWebShop
Authors
Tianjun Pan, Yuan Li, Hongda Wang, Linbo Jin, Mengfei Song, Lei Gao, Qiming Shi, Shaokang Fu, Jiarong Zhao, Chengyu Wang, Chengfu Huo
Abstract
External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.