Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
2026-08-10 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors study how to better evaluate and learn from detailed reasoning steps in AI models. They point out that simply measuring changes in token likelihood doesn't clearly show which intermediate steps deserve credit. By carefully separating how scores are made and what the training focuses on, they show that scoring a model’s own output can be misleading. Their experiments with a large model on math problems find that some scoring methods perform almost like random guesses and can even favor wrong answers. They suggest that understanding how scores relate to feedback and training goals is important before using token likelihood as a sign of credit.
outcome verifierself-distillationtoken likelihoodhindsight feedbackrollouttraining losspaired comparisoncredit assignmentAIME 2025large language model
Authors
Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao, Anurag Koul, Zeyu Liu, Shafiq Joty
Abstract
Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens. Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information. A token likelihood change, however, is not automatically outcome credit. We separate three questions: whether the score tracks better actions, whether feedback construction changes what is compared, and what behavior the training loss reinforces. We establish these distinctions formally. When a rollout is scored using hindsight feedback written about that same rollout, its content determines both the tokens and the scoring context, creating direct self-dependence. Using feedback from another rollout of the same problem removes this dependence but does not guarantee a useful score. In matched experiments with a 20B model on AIME 2025, the implemented additive score is near chance (AUC=0.505) and slightly favors incorrect traces after length adjustment. In the paired comparison, the outcome-only control records 64.2\%, versus 24.2\%--33.9\% for five token-score variants. The results motivate validating score meaning, feedback construction, and training behavior separately before calling a likelihood signal credit.