Trajectory dropout improves teaching of reasoning in ai models
What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillation
Artificial Intelligence
Summary
When AI models learn to reason step-by-step, they often train by comparing their own answers to a smarter teacher model one word at a time. The authors found that when the student model’s early answers are too confident or correct, it gets weaker signals to improve later answers. To fix this, they introduced a method called Trajectory Dropout, which hides parts of the student's reasoning during training. This makes the teacher's corrections stronger and helps the student improve more effectively. The method works across different models and math problems without extra computing cost.
What this means in practice
- •For ai engineers: Improve the training of language models for reasoning tasks by strengthening feedback signals during step-by-step learning.
- •For software developers: Enhance AI systems performing complex sequential predictions by integrating Trajectory Dropout to reduce overconfidence errors in outputs.
Authors
Zzizhuo Lin, Quanling Liu, Yi Yang, Yawei Luo
Abstract
On-policy distillation (OPD) trains a student model on its own trajectories using dense token-level feedback from a stronger teacher model. Since each update is conditioned on the reasoning prefix already generated by the student, the prefix also shapes how effectively teacher feedback is converted into learning. We find that shared prefixes can lead to weak token-level updates, a phenomenon we call Prefix-Induced Supervision Attenuation (PISA). This attenuation arises in two common cases. (i) High student confidence can weaken corrective gradients even when the teacher disagrees. (ii) Tokens that rely on earlier reasoning can receive learning signals as weak as those for simple local continuations. To solve this problem, we propose Trajectory Dropout, a simple training-time intervention that exposes these weakened signals. The student first performs a standard full-context rollout to generate a complete trajectory. During training, we randomly drop a certain proportion of the student's reasoning trajectory, while the teacher continues to observe the complete trajectory for token-level supervision. This intervention strengthens corrections for overconfident predictions and introduces additional supervision at prefix-sensitive positions. Trajectory Dropout consistently improves average performance across teacher--student model pairs of different scales and six mathematical reasoning benchmarks, while also yielding gains on two out-of-domain benchmarks. It can also be flexibly integrated into existing OPD variants with negligible computational overhead, further improving their performance. These results demonstrate that Trajectory Dropout provides a simple mechanism for strengthening token-level supervision across model scales and OPD objectives.