Improving language model reasoning by learning from self-generated answers
Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning
Artificial Intelligence
Summary
It can be hard for AI to learn from its own answers when there’s no clear way to check if those answers are right. The authors came up with a method where the AI judges how reliable its own answers are, and also considers how uncertain it is about parts of those answers. This way, the AI teaches itself better by focusing more on trustworthy and confident answers. Their approach worked better than other methods across many reasoning tests.
What this means in practice
- •For natural language processing teams: Enhance reasoning accuracy in large language models by training them to assess and learn from their own answer reliability and uncertainty.
- •For software developers integrating ai assistants: Improve the quality of AI-generated responses in open-ended tasks by incorporating self-evaluation to guide model refinement.
Authors
Jinhao Pan, Bowen Wei, Yuqing Zhou, Ziwei Zhu
Abstract
Reference solutions and question-specific rubrics are not always available to guide the teacher in self-distillation. A large language model's own solution attempts offer an alternative, but their quality is difficult to assess in open-ended tasks without automatic correctness checks, and even correct attempts can involve uncertain token predictions. We propose Reliability--Uncertainty Self-Distillation (\method{}) to learn from these attempts. The same model serves as teacher and student: the teacher receives the question and completed attempt, and the student matches the teacher's token-level predictions using a question-only prompt. \method{} weights each attempt's distillation loss by additively combining judged reliability, estimated from a self-judge's score and confidence, with student predictive uncertainty. Across eleven reasoning benchmarks, \method{} outperforms all five baselines in average accuracy, reaching 70.47\% on Qwen3-4B and 76.69\% on Qwen3-8B (8.90 and 2.10 percentage points above the base models). Data and code are available at https://github.com/JP-25/RUSD.