Improving language model reasoning by learning from self-generated answers

Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning

Artificial Intelligence

Summary

It can be hard for AI to learn from its own answers when there’s no clear way to check if those answers are right. The authors came up with a method where the AI judges how reliable its own answers are, and also considers how uncertain it is about parts of those answers. This way, the AI teaches itself better by focusing more on trustworthy and confident answers. Their approach worked better than other methods across many reasoning tests.

What this means in practice

Authors

Jinhao Pan, Bowen Wei, Yuqing Zhou, Ziwei Zhu

Abstract

Reference solutions and question-specific rubrics are not always available to guide the teacher in self-distillation. A large language model's own solution attempts offer an alternative, but their quality is difficult to assess in open-ended tasks without automatic correctness checks, and even correct attempts can involve uncertain token predictions. We propose Reliability--Uncertainty Self-Distillation (\method{}) to learn from these attempts. The same model serves as teacher and student: the teacher receives the question and completed attempt, and the student matches the teacher's token-level predictions using a question-only prompt. \method{} weights each attempt's distillation loss by additively combining judged reliability, estimated from a self-judge's score and confidence, with student predictive uncertainty. Across eleven reasoning benchmarks, \method{} outperforms all five baselines in average accuracy, reaching 70.47\% on Qwen3-4B and 76.69\% on Qwen3-8B (8.90 and 2.10 percentage points above the base models). Data and code are available at https://github.com/JP-25/RUSD.