Online distillation reduces excessive response length in reinforcement learning
Mitigating the Length-Scaling Tax with Online Distillation
Machine LearningArtificial Intelligence
Summary
Sometimes, when AI systems learn by themselves, they start giving longer answers even when short ones worked fine before. The paper shows this extra length can be unnecessary and calls it a length-scaling tax. The authors introduce a way to teach the AI to keep answers short when the problem is easy, while still allowing longer answers on harder problems. Their method uses the AI's own past behavior to guide learning, improving efficiency without making answers too wordy.
What this means in practice
- •For chatbot developers: Improve chatbot efficiency by reducing unnecessary verbosity on simple queries while keeping detailed responses for complex interactions during reinforcement learning upgrades.
- •For virtual assistant engineers: Maintain concise answers on routine commands and preserve adaptive reasoning on complex tasks when refining assistant behavior via reinforcement learning.
Authors
Xu Wan, Wenyue Xu, Shengjie Zhao, Mingyang Sun
Abstract
Length scaling during reinforcement-learning (RL) post-training is often viewed as a sign of improved reasoning ability, especially on difficult problems, but may also make responses to already-solved problems unnecessarily verbose. We quantify this side effect as the length-scaling tax (LST): excess response length on already-solved queries without a commensurate accuracy gain. To mitigate LST, we propose Length Self-Distillation (LSD), which routes solved prompts to on-policy distillation and retains the original RL objective for unsolved prompts. LSD uses an exponential moving average of the online policy as its teacher, requiring no external model. We find that LSD achieves comparable or better performance than RL across multiple variants, while substantially curbing response-length growth on easy queries. LSD reduces LST from 19.0% to -3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks, demonstrating that LSD effectively preserves concise response patterns on easy queries while supporting efficient exploration on difficult queries during RL post-training.