Multi-hop question answering errors split between retrieval and extraction
Diagnosing the Fact-Grounding Gap in Multi-Hop Question Answering
Computation and LanguageArtificial IntelligenceInformation Retrieval
Summary
Answering complex questions often needs combining facts from multiple documents. This paper finds that mistakes happen not only when the right documents are not found but also when the needed information is in the documents but can't be identified. The authors call this second type the fact-grounding gap. They show that this gap accounts for nearly half of all mistakes and isn’t noticed by current tests that only check document retrieval. Their work suggests different solutions are needed for each problem.
What this means in practice
- •For natural language processing teams: Improve QA system evaluation by separately detecting retrieval and extraction errors to better target fixes.
- •For search engine developers: Design retrieval and extraction modules independently to address distinct failure types in multi-hop question answering.
Authors
Kevin Mo, Nathan Mo, Richard Zhu
Abstract
Multi-hop question answering requires combining information from multiple documents to answer complex questions. These systems have grown increasingly capable, yet when they fail, the error is typically attributed to not finding the right documents. Whether this holds at the level of individual reasoning steps remains largely unexamined. We investigate this across three standard multi-hop QA benchmarks and find that failures decompose into two distinct modes: retrieval failures, where the needed passage was not retrieved, and extraction failures, where the passage was retrieved but the needed fact could not be extracted - a phenomenon we term the fact-grounding gap. Extraction failures account for nearly half of all per-hop deficiencies and are invisible to standard retrieval metrics. They remain unresolved by every retrieval intervention we test, establishing a ceiling for retrieval-only improvements. The gap's severity varies across benchmarks and question types, but extraction failures appear on every dataset we measure. Our findings reveal that retrieval failures and extraction failures are fundamentally different bottlenecks requiring different solutions - a distinction absent from current evaluation practice.