Llms learn to predict accuracy differently depending on data type
LLMs learn different forms of metacognition when trained to predict their own accuracy
Computation and Language
Summary
Large language models normally always try to give answers, even when unsure, which can cause them to make things up. This paper studies how models learn to guess how accurate their answers are when trained to predict their own correctness. The authors found that for questions similar to training data, the models' confidence aligns with true accuracy, but for other questions, confidence mostly reflects how consistent the model's answers are rather than their correctness. This shows that models may not truly learn to recognize when they are confidently wrong, raising questions about how well they understand their own knowledge.
What this means in practice
- •For ai model trainers: Improve training processes to enhance how models estimate their accuracy on familiar data, potentially leading to more reliable model confidence scores.
- •For ai safety engineers: Develop better evaluation metrics for identifying confidently wrong answers by understanding limits of current calibration training.
Authors
Nicolas Yax, Stefano Palminteri, Pierre-Yves Oudeyer
Abstract
Large language models are trained to always produce an answer, regardless of whether they possess the relevant knowledge, which leads them to fabricate facts. Prior work has shown that LLMs' confidence estimates correspond poorly to their actual performance, and that fine-tuning can substantially improve them. However, what models actually learn during such training remains poorly understood. We investigate how LLMs acquire metacognitive monitoring, the ability to know what one knows, by training 10 open-weight LLMs to predict their own accuracy on factual multiple-choice questions before answering them. We find that trained confidence reflects two distinct signals. While on questions close to the training data, it tracks the model's true accuracy, in other domains, it instead tracks output consistency: the concentration of the model's answer distribution. Output consistency tracking emerges early in training and generalizes across datasets, whereas accuracy tracking develops later and remains local to the training distribution. These results suggest that calibration training may not teach models to generally detect errors they commit confidently, and they raise broader questions about the nature of metacognition in artificial systems.