ActionOPD improves small AI agents by focusing on decisions not words
ActionOPD: Distilling Decisions, Not Tokens
Artificial Intelligence
Summary
When making smaller versions of big AI language models, it's tricky to copy how the bigger model chooses what to do next. The paper introduces ActionOPD, a way to train smaller models to imitate the big ones based on their actual decisions (actions) rather than trying to copy every single word they produce. This method works even if the smaller and bigger models use different vocabularies or internal setups. ActionOPD shows better performance on tasks the models haven't seen before, making it a useful technique for building efficient AI tools.
What this means in practice
- •For ai model developers: Train smaller AI agents to mimic larger models more effectively by focusing on action-level behavior alignment without matching token outputs.
- •For software engineers building ai systems: Improve tool-using AI systems by distilling decisions from powerful models into smaller, faster ones that handle diverse tasks reliably.
Authors
Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen, Xiao-Ping Zhang, Yu Wang, Chao Yu
Abstract
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On $τ^2$-bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to agent distillation.