Summary
Sometimes AI language models make things up, which can cause problems when accurate information is important. The authors focus on improving how we spot these made-up details by using external information retrieved during the AI’s answer process. They propose a new technique called Evidence-Aligned Entity Verification that checks if the names and facts the AI mentions match trustworthy sources it pulls in at the same time. Their method looks at this match in several ways and tests it against changes to the sources to make sure it is reliable. Experiments show their approach helps find false statements more accurately and works well across different types of tasks.
hallucination detectionlarge language modelsretrieval-augmented generationentity verificationfactual accuracycounterfactual stabilityevidence alignmentparametric knowledgeinformation retrievalmodel robustness
Authors
Runsong Jia, Zhen Fang, Mengjia Wu, Jie Lu, Yi Zhang
Abstract
Hallucination detection is crucial for large language models (LLMs), as hallucinated content creates significant barriers in applications requiring factual accuracy. Current detection methods mainly depend on internal signals like uncertainty and self-consistency checks, using the model's pre-trained knowledge to identify unreliable outputs. However, pre-trained knowledge may become outdated and has coverage limitations, especially for specialized or recent information. To address these limitations, retrieval-augmented generation (RAG) has emerged as a promising solution by retrieving relevant evidence at inference time, grounding outputs beyond the model's parametric knowledge. In this paper, we target a critical and practical learning problem RAG-based hallucination detection (RHD), where RAG is employed to enhance hallucination detection by addressing information updating challenges. To address RHD, we propose a novel method Evidence-Aligned Entity Verification (EAEV), which detects entity-level hallucinations by leveraging RAG to align generated entities with retrieved evidence contexts. Specifically, EAEV evaluates entity-evidence alignment through three complementary dimensions and introduces counterfactual stability analysis to ensure robust alignments under evidence perturbations. Experiments across multiple RAG benchmarks demonstrate that EAEV achieves consistent improvements over existing methods with strong generalization capabilities.