Hierarchical guidance improves long task learning for smaller language models
Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation
Computation and Language
Summary
Training smaller AI models to behave like larger ones can be tricky, especially when tasks take many steps to complete. The authors found that simply copying every detail uniformly doesn’t help these smaller models learn effectively. They designed a method called LENS-OPD that first finds where the smaller model struggles, checks if giving advice there actually helps, and then focuses teaching efforts on the most important parts. Testing this approach showed it helps smaller models perform better on complex tasks over longer times.
What this means in practice
- •For ai model developers: Improve training efficiency for smaller language models handling multi-step tasks by allocating supervision where it most benefits learning.
- •For robotics software engineers: Enhance agent decision-making over long tasks by guiding policy training through dynamic hierarchical intervention points.
Authors
Yuhao Sun, Binrui Wu, Zhuoer Xu, Ming Wen, Haoxiang Xu, Bin Chen, Yan Lin, Qianzijing Zhang
Abstract
On-policy distillation (OPD) transfers the capabilities of a large language model to a smaller student by providing teacher supervision on the student's own rollouts. In long-horizon agentic tasks, however, uniform token-level matching can allocate supervision poorly: a large local discrepancy need not improve future behavior, while consequential guidance may be beyond the current student's reach or fail to persist without privileged input. We formulate long-horizon OPD as hierarchical supervision allocation and argue that productive guidance lies at the intersection of future utility and current learnability. Crucially, this intersection evolves as the student learns. Based on this principle, we propose LENS-OPD, a coarse-to-fine framework that organizes supervision through Locate, Validate, and Refine. Locate adapts trajectory exposure to the student's evolving competence and proposes a candidate decision for intervention. Validate tests whether teacher guidance at that decision improves the same student's subsequent behavior. Refine internalizes the beneficial guided behavior into the deployable policy and concentrates token-level supervision on decisive teacher-student conflicts within the validated turn. These stages are nested: each finer allocation is conditioned on the coarser decision, rather than being optimized as an independent importance score. Experiments across multiple long-horizon agent benchmarks and student-teacher configurations show that LENS-OPD consistently improves task performance over vanilla OPD and strong curriculum- and selection-based baselines. Our results suggest that effective long-horizon distillation requires teaching at the right depth, the right decision, and the right token.