Hallucination detection labels differ based on faithfulness versus factual correctness

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

Computation and Language

Summary

Figuring out when AI language models make things up—called hallucinations—is tricky because there are different ways to judge if an answer is wrong. One way is to check if the answer matches a given reference, and another is to see if the answer is factually true regardless of that reference. The authors found that automatic systems often get confused between these two and label answers inconsistently compared to human judgment. They showed that changing how the question is asked to AI can improve agreement with humans. This means careful design and clarity about what kind of error is being labeled are very important.

What this means in practice

  • For ai model evaluators: Improve reliability of hallucination detection in language model benchmarks by explicitly choosing and validating label criteria.
  • For qa system developers: Design detection systems that distinguish factual errors from lack of reference support to reduce false alarms in automated answer verification.

Authors

Jorma Valjakka, Juhani Kivimäki, Juha Mylläri, Jukka K. Nurminen

Abstract

In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked with open-domain question answering (QA) datasets containing questions and corresponding short reference answers. First, an LLM is used to generate answers to questions within the QA dataset. Then, some automated labeling strategy is used to label these answers as hallucinated or not by comparing them with the reference answers in the dataset. This evaluation setting creates a methodological ambiguity between two criteria: reference faithfulness (whether the answer is fully supported by the reference) and factual correctness (whether the answer is free from contradictions and factually false specific claims). In practice, automated labelers may apply the former criterion even when the intended target is the latter. We study this potential criterion mismatch using 900 human-labeled question-answer pairs spanning three commonly used QA datasets and three generator models, with labels targeting answer-level factual correctness. We evaluate lexical similarity metrics, a reference-entailment NLI baseline, and seven LLM judges under controlled prompt variants as automated labelers. Our experiments reveal substantial disagreement both among automated labeling strategies and between these labels and human annotations. Many strategies also exhibit strong directional error biases, and for most judge-generator pairs, replacing a faithfulness-oriented prompt with a factual-correctness prompt improves agreement with human annotations and reduces false-positive dominance, indicating that automated hallucination labels depend strongly on how the target criterion is specified. Label-source choice should therefore be considered a fundamental part of benchmark design and made explicit, validated, and matched with the benchmark goal.