Diversity-Oriented Fine-Tuning for Uncertainty-Based Hallucination Detection
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
The authors explore ways to improve how models detect their own mistakes, called hallucinations, by changing the training process rather than just checking outputs later. They notice that wrong answers often look very similar each time, making them harder to catch. To fix this, the authors fine-tune the models to produce more varied wrong answers using two methods, which helps their detection technique work better. Their experiments show that this approach makes it easier to spot hallucinations without losing accuracy.
Authors
Qiuyuan Li, Hongliang Dai, Piji Li
Abstract
Existing hallucination detection methods are typically conducted at the inference stage, without making any modifications to the model itself. In this paper, we are interested in exploring fine-tuning strategies that enhance the detectability of hallucinations in the resulting model. Focusing on semantic-entropy-based detection, we observe that many erroneous outputs remain undetected because the model produces nearly identical incorrect answers across multiple runs. To address this, we propose diversity-oriented fine-tuning to encourage more varied generations. We introduce two specific strategies: one based on Supervised Fine-Tuning (SFT) and the other on Direct Preference Optimization (DPO). Extensive experiments are conducted to evaluate our approach and analyze the behavior of the models before and after fine-tuning. We find that after adopting our fine-tuning methods, the models become less likely to produce low semantic entropy responses for hallucinated answers, thereby improving the effectiveness of hallucination detection, eventually yielding results better than or comparable with state of the art methods. The code will be publicly released.