Teacher guidance improves student model training in reasoning tasks
From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Computation and LanguageMachine Learning
Summary
Training AI students to think like stronger teachers can help improve their problem-solving, but too much teacher involvement can confuse the student because it changes the way the student learns. The authors show that finding the right balance of teacher help depends on the task and where the teacher steps in. They created a system called MAESTRO that decides when and how long the teacher should guide the student based on how much the teacher and student's thoughts differ. This approach improved accuracy and made training responses shorter on math reasoning tests.
What this means in practice
- •For machine learning engineers: Train AI reasoning models more effectively by dynamically adjusting teacher guidance during learning to boost accuracy and reduce training time.
- •For educational technology developers: Design AI tutoring systems that adapt when to provide hints or intervene based on the learner’s current understanding level.
Authors
Yuhao Wang, Ruiyang Ren, Yinan Zhang, Ruiqing Zhang, Jing Liu, Chunyan Miao
Abstract
On-policy distillation (OPD) trains a student on its own reasoning trajectories using feedback from a stronger teacher. Teacher interventions can improve these trajectories, but also change the distribution on which the student learns. Our controlled studies show that rollout quality alone is an incomplete criterion for allocating teacher guidance. Deeper intervention yields diminishing gains in rollout accuracy while increasing off-policy load. In a training probe with a restricted rollout horizon, peak student accuracy and performance retention favor different intervention strengths. The preferred intervention depth and placement also vary across benchmarks. These findings motivate MAESTRO, which uses local policy disagreement to jointly adapt when the teacher takes over and how long it generates. Its {policy disagreement score} combines teacher-weighted candidate coverage with local distribution similarity and is aggregated within reasoning paragraphs. Across eight mathematical reasoning benchmarks, MAESTRO achieves the highest macro-average accuracy among the compared methods for both 0.6B and 1.7B Qwen3 students, with the 1.7B student leading on every benchmark. MAESTRO also reduces average training response length by 67.3\% relative to standard OPD. The code is available at https://github.com/yhao-wang/MAESTRO.