Selective internal feedback improves language model reasoning accuracy
Learning to Reason with Selective Internal Feedback
Artificial Intelligence
Summary
Language models that solve problems can sometimes become too confident even when they are wrong. The authors studied a way to teach these models by using their own uncertain feelings, rather than relying on correct answers from outside. They developed a method called SIF that carefully picks which of the model’s self-assessments to trust and use for learning. This method helped models get better at math and science tasks, keeping accuracy high even after more training.
What this means in practice
- •For ai engineers: Improve language model training so internal confidence better guides accurate reasoning without external answers.
- •For educational software developers: Create smarter tutoring systems that learn to evaluate their reasoning process internally, improving problem-solving accuracy.
Authors
Xun Lu, Shumeng Yang, Xinrui Wang, Songjun Tu, Linjing Li, Daniel Dajun Zeng
Abstract
Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers. However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy. We introduce SIF, a reinforcement learning method that combines comparative reward construction with selective policy updates. SIF compares ranked token-level self-certainty profiles across multiple windows and peers, assigning positive rewards only when both levels of voting provide sufficient support. Sample and group level filtering determines which responses contribute to policy optimization. Experiments on three backbones across 12 mathematical and scientific reasoning benchmarks show that SIF's final checkpoints achieve higher average accuracy than the best-validation checkpoints of Intuitor and Vigor, with a gain of 4.71 percentage points over Intuitor on Qwen2.5-Math-1.5B. Further analyses show better retention of accuracy during continued optimization, while test-time training experiments demonstrate sustained improvements with as few as four rollouts per prompt.