Hierarchical framework detects conflicting evidence in ai-generated answers
A Hierarchical Consistency Framework for Auditing Retrieval-Augmented Generation Systems
Artificial Intelligence
Summary
Many systems that answer questions by looking up information and then generating responses can produce answers that seem right but are actually supported by conflicting facts. The authors introduce a new way to check answers by examining the original information, the retrieved evidence, and the final answer separately. This method helps find where contradictions happen, even when the answer matches what’s expected. Their approach does not prove if an answer is true but makes it easier to see and understand the evidence behind it.
What this means in practice
- •For ai system developers: Audit output consistency at multiple stages to identify contradictions that answer-only checks miss in retrieval-augmented generation systems.
- •For compliance teams: Inspect and attribute conflicting evidence in AI-generated reports to support transparency and risk assessment.
Authors
Ramon Gonzalez, Antonio Diaz
Abstract
Retrieval-augmented generation (RAG) is commonly evaluated by whether the final answer is correct. That test is insufficient: an answer can match its reference while the context that produced it contains a direct contradiction, leaving the contested evidence invisible to answer-only review and retrieval relevance scores. This paper presents the Hierarchical Consistency Framework (HCF), a post-hoc, model-agnostic audit of three distinct levels of a RAG process: the knowledge corpus, the final retrieved context, and the generated answer. HCF represents corpus conflicts as source-linked atomic facts, thereby identifying the documents responsible, and returns each Answer Consistency Score (ACS) with an explanation of supporting and contradictory contextual statements. We evaluate HCF on several controlled corpora spanning five domains and 100 query-corpus instances. A human evaluator compares every generated response with its supplied ground-truth response. The results show that the three diagnostic levels can dissociate: the corpus with the highest mean retrieval similarity has the lowest mean ACS, while a structurally degraded corpus performs worse at corpus level but better at answer level. Most importantly, HCF identifies contradictory retrieved evidence in several cases where the answer still matches the ground truth. HCF does not certify factual truth; it makes the evidence supporting and challenging an answer inspectable and attributable.