Language model improves by learning from its own uncertain answers
Learning from Unverified Peers via Shared Residual Distillation
Artificial Intelligence
Summary
Language models create different solutions to problems, but not all are correct, and it's hard to know which ones are right. The authors propose a method where the model learns from all its attempts, even uncertain ones, by focusing on consistent changes in predictions shared across solutions. This helps improve the model without needing external feedback or correct answers. Their approach enhanced performance significantly on specific tests without harming general language abilities.
What this means in practice
- •For natural language processing engineers: Improve language model accuracy by training models to learn from their own uncertain outputs without relying on verified answers.
- •For machine learning platform teams: Integrate self-distillation techniques that enhance model reasoning through shared residual adjustments to boost performance on complex tasks.
Authors
Feiyang Chen, Runze Ma, Zichen Wang, Jiong Lou, Zihan Wang, Shuangjia Zheng, Jie Li
Abstract
A language model can generate several solutions to the same problem, some leading to correct answers and others to incorrect ones. Many post-training methods turn these trajectories into learning signals using supervision from reference answers, verifiable feedback, or external teachers. Without such supervision, learning from the model's own generations becomes less straightforward, as their correctness cannot be reliably assessed. An incorrect solution may nevertheless contain useful intermediate reasoning, so its value as supervision need not be determined solely by its final answer. Can a model improve by learning from its unverified peer solutions without first determining which are correct? We propose Shared Residual Distillation (SRD), an on-policy self-distillation method that learns from the model's own unverified peer trajectories. SRD compares how peers change the prediction distribution relative to a control, retaining adjustments that agree in direction and are conservative in magnitude. This yields supervision without treating any complete solution as correct, and the trained model requires no peers at inference time. Starting from the corresponding base models, SRD raises mean AIME24--26 avg@12 from 11.02 to 16.20 on Qwen3-1.7B, from 19.07 to 45.46 on Qwen3-4B, and from 21.39 to 54.44 on Qwen3-8B. Controlled ablations support the value of the shared-residual construction, and we observe no degradation relative to the base models on two general-capability benchmarks.