Language model skills help smaller AI work like bigger models
D-Skill: Distilling Teacher Knowledge through Agent Skills
Artificial Intelligence
Summary
Big language-model agents are good at complex tasks but are costly to run. The authors present D-Skill, a way to teach smaller models by capturing the behavior differences between a big and a small model in natural language 'skills' without re-training the smaller model’s parameters. This method uses past task examples from both models and creates easy-to-understand skill guides that help the smaller model perform closer to the bigger one, saving time and computing power. Their tests show that D-Skill can close most of the performance gap while making the smaller model much cheaper to run.
What this means in practice
- •For ai system builders: Provide smaller AI models with interpretable skill documents to improve their task performance without retraining.
- •For game engine developers: Use skill-based knowledge distilled from large agents to make lightweight game AI that performs close to heavy models with lower computational needs.
Authors
Shuo Yuan, Dacheng Li, Joseph E. Gonzalez
Abstract
Frontier language-model agents excel at complex tasks, but high inference costs make transferring their capabilities to smaller models essential. Traditional knowledge distillation achieves this through parameter updates, requiring access to the student's weights and additional training compute while encoding the transferred knowledge implicitly in the parameters. We introduce D-Skill, a framework that treats the teacher as a black box and distills its behavioral advantage into an interpretable natural-language skill by analyzing paired rollouts and identifying behavioral divergences and agreements. The resulting lessons are consolidated offline, with any compatible skill-update backend, into a skill document that is provided to the student at inference time. D-Skill constructs the skill entirely offline from fixed trajectories, without labels, further environment interaction, or weight updates after rollout collection. Across SpreadsheetBench-Verified, ALFWorld, and chess puzzles spanning multiple teacher-student pairs, D-Skill closes 75%-95% of the teacher-student gap without touching model weights. After a single offline pass over the paired traces, D-Skill outperforms one-epoch SFT baselines and remains competitive with multi-epoch SFT. Under the same model family, the resulting student also incurs 5.7x-8.3x lower inference cost per task than the teacher. Our results demonstrate that natural-language skill distillation offers a performant, interpretable, and efficient alternative to weight-based capability transfer.