Confidence calibrated reinforcement learning reduces hallucinations in AI answers
CalibraRL: Confidence-Calibrated Reinforcement Learning for Mitigating Hallucinations in RAG
Computation and Language
Summary
Large language models sometimes give answers that sound right but are actually wrong, which is a problem. The authors introduce CalibraRL, a method that helps these models better judge how confident they should be about their answers and when they should choose not to answer. This approach rewards models for being both correct and well-calibrated in their confidence, leading to fewer mistakes and less chance of misleading information. Their tests show CalibraRL improves reliability in answering questions by reducing overconfidence and unnecessary abstentions.
What this means in practice
- •For search engine developers: Enhance retrieval-augmented question answering systems by reducing incorrect confident answers and unnecessary abstentions to improve user trust.
- •For ai content moderation teams: Improve confidence assessment in generative AI outputs to better detect and mitigate hallucinated or unreliable content.
Authors
Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, Zhiming Zheng
Abstract
Large language models (LLMs) often produce plausible but incorrect answers, motivating reinforcement learning methods that reward correct answers, penalize incorrect answers, and allow abstention. However, fixed outcome-based rewards do not distinguish incorrect responses with different confidence levels or account for differences in answering reliability across queries. Consequently, such rewards may inadequately penalize overconfident errors and fail to discourage unnecessary abstention. We propose CalibraRL, a reinforcement learning framework that incorporates response confidence and query-level answering evidence into reward design. For non-abstaining responses, CalibraRL combines correctness-based utility with a Brier-based penalty that encourages agreement between confidence and correctness. For abstentions, it estimates conditional answering reliability using a Beta posterior derived from the correctness of non-abstaining responses within each sampled group. The abstention reward increases with the posterior probability that the model's accuracy on non-abstaining responses is below a predefined threshold. CalibraRL reuses training rollouts without requiring additional annotations indicating whether the model knows the answer to each question. Experiments on three QA datasets demonstrate that CalibraRL improves answer precision and reduces hallucinations compared with strong baselines. Further analysis shows that CalibraRL reduces mean expected calibration error (ECE) by 30.8\% across two backbone models and three datasets, highlighting its potential to improve the reliability of retrieval-augmented generation (RAG) in web search.