AI summaryⓘ
The authors studied how language models that use reasoning steps learn better by teaching themselves with extra hints (privileged information). They found that when these hints are given during learning (self-distillation), models that do long and complex reasoning tasks actually get worse, especially on harder problems. This problem happens mostly when there are many possible next steps in reasoning, and the hints make the models less flexible in exploring different ideas. Interestingly, this negative effect is not seen in simpler models tuned only with instructions. The authors suggest that teaching strong reasoning models needs careful handling of the details in the reasoning process, especially the parts where the model rethinks or corrects itself.
self-distillationlanguage modelsprivileged informationreasoning traceon-policy distillationhigh-entropy forkingthinking modelsinstruction-tuned modelsbacktrackingtoken-level signal
Authors
Simran Kaur, Narutatsu Ri, Yinghui He, Liam Fowl, Sanjeev Arora
Abstract
Self-distillation is a promising recipe for self-improvement in language models. In this setting, a model can serve as its own teacher when given privileged information, such as a solution to a math problem. This seems especially appealing for thinking models, which can use test-time reasoning to absorb the privileged information. Surprisingly, we show that privileged self-distillation degrades thinking models on long reasoning traces: across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode is not specific to self-distillation: on-policy distillation (OPD) improves thinking models, but privileged OPD reverses these gains. Our diagnostics link this failure mode to how privileged teacher context reshapes learning at high-entropy forking positions, where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction-model rollouts. This leads to an interesting dichotomy, where privileged context can help instruction-tuned models but hurts stronger thinking models. The effect is visible when the student begins a self-correction branch, where privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that self-distillation for strong thinking models requires attention to token-level signal, especially around correction and reasoning steps.