Post-training increases confident wrong answers in language models
The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models
Computation and LanguageArtificial Intelligence
Summary
Large language models sometimes give wrong answers that sound very sure, which makes it hard to tell when they are mistaken. This paper finds that fine-tuning these models to follow instructions actually makes them more confident in their incorrect answers, especially for rare facts. The authors identify that this overconfidence appears in the last layers of the model after fine-tuning. They propose a way to limit this overconfidence during post-training, which reduces confident errors while keeping the model’s reasoning abilities intact.
What this means in practice
- •For llm developers: Reduce high-confidence factual errors during fine-tuning by bounding margin growth in model outputs.
- •For ai safety engineers: Improve reliability of uncertainty-based error detection in deployed language models by mitigating confident hallucinations post-training.
Authors
Qingjia Huang, Yakai Li, Jianguo Wu, Qihang Zhou, Aimin Yu, Xiaoqi Jia, Luping Ma, Weijuan Zhang
Abstract
Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the \textbf{Alignment Paradox}. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors ($p \ge 0.95$) by more than an order of magnitude (10$\times$ to 35$\times$). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3\% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.