Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
2026-08-03 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors studied a way to help AI agents learn better by having a 'teacher' AI guide a 'student' AI using states it actually visits. They found that over time, the student's path can drift to areas where the teacher's advice is less useful. To fix this, they created FutureBridge-OPD (FTB), which tries a short teacher guide at tricky points and checks if it helps the student get better guidance overall. Testing on several tasks, FTB improved learning compared to older methods and worked well with different AI sizes and teacher versions.
on-policy distillationagentic tasksteacher-student learningtrajectory distributiondisagreement statesdistillation signalsALFWorldWebShopScienceWorldQwen models
Authors
Chishui Chen, Yaoyou Fan, Te Sun, Yi Yang, Chenghao Sun, Delin Mao, Hongbo Qiao, Zuowei Zhang, Junxi Wang, Chenxing Sun, Yangen Hu, Lu Pan, Xuyang Liu, Linfeng Zhang
Abstract
On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference. However, in multi-turn agentic tasks, student deviations may accumulate over time, gradually moving the trajectory away from states where teacher guidance remains effective. Our quantitative analysis further shows that high-disagreement states offer promising opportunities for teacher guidance, but determining whether such guidance is beneficial requires examining its effect on subsequent student trajectories. We propose FutureBridge-OPD (FTB), which executes a short teacher bridge at a high disagreement state and uses the resulting student continuation to assess whether the bridge increases the density of positive distillation signals relative to the teacher. On ALFWorld, WebShop, and ScienceWorld, under the main Qwen3-32B teacher to Qwen3-1.7B student setting, FTB outperforms vanilla OPD and TCOD by an average of 16.6 and 7.6 points, respectively, and remains effective across student scales and teacher settings. Our code is publicly available at https://github.com/ChenChiShui/FutureBridge-OPD.