Large language models verbalized and internal probabilities closely match
Verbalized and Internal Probabilities Are Coupled in Large Language Models
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) have two ways to show how certain they are about their answers: one is the built-in probabilities they use internally to pick words, and the other is when they say how confident they are, either by words or numbers. The paper finds that these two ways of showing uncertainty are linked and influenced by both frequency and explicit uncertainty statements in the training data. This means when a model says how confident it is, it gives a good clue about the hidden probabilities it uses inside. So, asking a model how sure it is can tell us about its real guessing process.
What this means in practice
- •For machine learning engineers: Use model verbalized confidence reports to better estimate internal decision probabilities during debugging or model refinement.
- •For ai system integrators: Incorporate LLM verbalized confidence scores as trustworthy indicators of uncertainty in decision-support or automated systems.
Authors
Sinead Williamson, Jiaxuan Li, Nick Foti, Russ Webb, Masha Fedzechkina
Abstract
Large language models carry an internal notion of uncertainty in their sampling distribution, i.e., the probabilities they place on generating one answer rather than another. They can also be asked to state a confidence, in words or as a number: a verbalized uncertainty. Prior work suggests that internal probabilities track relative frequencies in the training data, and that verbalized probabilities track explicit probabilistic assertions in the training data. However, we do not know whether these two readouts are aligned, except when frequencies and probabilistic assertions in the training data happen to align. This limits our understanding of when we can use verbalized uncertainties as a proxy for either training data frequencies, or a model's internal distribution. We resolve this gap by systematically exploring how LLMs probability readouts are impacted by training and in-context data, via intervening on the underlying uncertainty sources in the data. We find that both internal and verbalized probability readouts are impacted by both distributional and asserted uncertainty in the training data. Further, we find that verbalized and internal probabilities are aligned beyond what would be expected by independently tracking the same uncertainty sources, suggesting that verbalized probabilities can be used to probe a model's internal distribution.