Language models learn to answer only with set confidence levels
Reinforcement Learning Done Right for Explicit Confidence Targets
Machine LearningComputation and Language
Summary
Large language models sometimes give answers that sound good but might be wrong, especially when they are unsure. This can confuse people who use them. The authors worked on training a single model to decide when to answer or say "I don't know," based on a confidence level chosen by the user. Their new method keeps the training stable and effective, allowing the model to meet these confidence goals without losing its overall ability. This helps make model responses more reliable and lets users control how confident the answers should be.
What this means in practice
- •For chatbot developers: Enable chatbots to answer only when they meet user-set confidence levels, avoiding misleading responses.
- •For customer support teams: Provide customer support systems that can skip uncertain answers to improve user trust and service quality.
Authors
Zefeng He, Yichen He, Yuan Lin, Yang Yang, Yu Cheng
Abstract
Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliability. Training models to abstain with reinforcement learning is a natural approach, but most existing methods use a fixed error penalty and therefore cannot adapt their behavior to different confidence requirements at inference time. We study \emph{explicit confidence targets}, where a user specifies a target \(p\) in the prompt and the model directly chooses whether to answer or abstain, with the goal of achieving an accuracy of at least \(p\) among answered questions. However, learning to respond to different confidence targets with a single model has proven difficult, and prior attempts have failed to distinguish different targets. We show that this failure comes from two problems: the reward signal that distinguishes different values of $p$ is often lost during optimization, while the original error penalty grows without bound as $p$ approaches one. We propose RLEC (\textbf{R}einforcement \textbf{L}earning with \textbf{E}xplicit \textbf{C}onfidence Targets), which preserves the learning signal needed to distinguish confidence targets, introduces a stable reward family that keeps the desired answer condition unchanged, and uses reward scheduling to balance effective learning and training stability. Together, these designs enable the first successful training of a single LLM for explicit confidence target control. Extensive experiments across diverse benchmarks and model scales show that RLEC largely preserves general capabilities and achieves competitive calibration performance. RLEC also supports fine-grained target control without an external decision system or threshold calibration after training. By making explicit confidence targets part of the trained policy, RLEC opens a new and more direct path toward behavioral calibration.