Evidence aligned verification improves detecting hallucinated facts in AI outputs
Evidence-Aligned Entity Verification for Hallucination Detection in Retrieval-Augmented Generation
Artificial IntelligenceInformation Retrieval
Summary
Large language models sometimes make things up, which is a big problem for using them in tasks needing true facts. The authors show that checking facts against up-to-date evidence retrieved during the model’s response can improve spotting these made-up details. They introduce a new method that aligns named things in the model’s answers with external evidence and tests how stable these alignments are. Their approach works better than earlier ones across several tests and is good at handling new or niche information.
What this means in practice
- •For ai developers: Improve the reliability of AI-generated text by reducing false information through real-time evidence verification.
- •For content moderation teams: Detect and flag hallucinated or inaccurate claims in AI-supported content generation workflows to maintain factual standards.
Authors
Runsong Jia, Zhen Fang, Mengjia Wu, Jie Lu, Yi Zhang
Abstract
Hallucination detection is crucial for large language models (LLMs), as hallucinated content creates significant barriers in applications requiring factual accuracy. Current detection methods mainly depend on internal signals like uncertainty and self-consistency checks, using the model's pre-trained knowledge to identify unreliable outputs. However, pre-trained knowledge may become outdated and has coverage limitations, especially for specialized or recent information. To address these limitations, retrieval-augmented generation (RAG) has emerged as a promising solution by retrieving relevant evidence at inference time, grounding outputs beyond the model's parametric knowledge. In this paper, we target a critical and practical learning problem RAG-based hallucination detection (RHD), where RAG is employed to enhance hallucination detection by addressing information updating challenges. To address RHD, we propose a novel method Evidence-Aligned Entity Verification (EAEV), which detects entity-level hallucinations by leveraging RAG to align generated entities with retrieved evidence contexts. Specifically, EAEV evaluates entity-evidence alignment through three complementary dimensions and introduces counterfactual stability analysis to ensure robust alignments under evidence perturbations. Experiments across multiple RAG benchmarks demonstrate that EAEV achieves consistent improvements over existing methods with strong generalization capabilities.