Language models learn to gauge when they have enough evidence to answer

Each Insufficient in Its Own Way: Calibrating Language Models Across Varying Evidence States

Computation and Language

Summary

Language models often try to answer questions even when they don't have enough information, which can lead to mistakes. The authors propose a new approach called Sufficiency Confidence Estimation that helps models better judge how likely they are to be correct based on the evidence they currently have. This method fine-tunes the model's confidence levels to match how successful the model actually is when answering using different amounts or types of evidence. Tests show it improves the accuracy of confidence estimates in tasks like medical diagnosis and complex question answering. This means models can better decide whether to answer immediately or seek more information first.

What this means in practice

  • For clinical data teams: Improve medical AI systems by adjusting confidence to better reflect available patient evidence, helping decide when diagnoses are reliable.
  • For question answering system developers: Enhance multi-step question answering models to signal confidence accurately based on varying evidence inputs, improving answer reliability.

Authors

Jiayi Li, Dennis L. Shung, Bradly C. Stadie

Abstract

How can a language model know whether it has enough evidence to answer? Language models tend to answer even when the evidence they have is insufficient. Deciding whether to answer or gather more evidence requires a calibrated estimate of how likely an answer is to be correct with the available evidence. However, most calibration work treats the input as fixed, without asking how this estimate should change with the evidence. We introduce \emph{evidence-state calibration}, which requires a model's confidence before answering to match its probability of answering correctly and to change accordingly whenever the evidence changes this probability. We propose \ours{} (\textbf{Suffi}ciency \textbf{C}onfidence \textbf{E}stimation), which fine-tunes the confidence already present in a model's next-token distribution to match how often the model answers correctly from each evidence state. Across clinical diagnosis and multi-hop question answering with four models from two families, \ours{} reduces the Brier score by up to 60\% and increases AUROC by up to 0.16. Its confidence matches the empirical success rate and responds not only to the amount of evidence but also to which evidence is present, showing that models can learn to judge whether their current evidence is enough to answer.
View PDFOpen arXiv