Trajectory intervention improves teacher guidance in self-distillation for coding and science models
TISD: On-Policy Self-Distillation with Trajectory Intervention
Artificial IntelligenceMachine Learning
Summary
Sometimes, when a computer learns from its own past decisions, it only gets feedback on the actions it actually took, missing important alternatives. The authors found that encouraging the computer to try different paths chosen by a guiding teacher helps it learn better by exposing more useful situations. They developed a method called Trajectory-Intervention Self-Distillation (TISD) where the teacher forces a better choice at a critical step, letting the student continue from there and learn from the full new experience. Their tests show this approach improves the quality of coding and science problem-solving models under equal training effort.
What this means in practice
- •For machine learning engineers: Enhance training of language models by incorporating teacher-guided branching to improve exposure to useful sequences during self-distillation.
- •For ai tool developers: Improve coding assistant models by applying TISD to boost their problem-solving accuracy within the same training time or data budget.
Authors
Taeckyung Lee, Rinat Amankos, Jeonghye Kim, Hyungjun Yoon, Woogyeol Jin, Sung-Ju Lee
Abstract
On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher-student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation.