Multi hop retrieval augmented generation reveals evidence search challenges
Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG
Computation and Language
Summary
When AI systems answer complex questions by gathering clues step-by-step, they sometimes miss important information either because they can’t find it or they lose track of what to look for. The authors show that many errors come not from missing clues but from the system stopping the search too early or guessing too soon. They created a tool that helps the system keep track of what questions still need answers and what evidence is needed, which revealed less missing information but new challenges in checking if the found clues are enough to answer the question.
What this means in practice
- •For ai developers: Improve multi-hop question answering systems by tracking open sub-questions to reduce retrieval errors caused by premature search termination.
- •For chatbot designers: Enhance dialogue agents to better verify whether gathered evidence sufficiently answers complex user queries, reducing incorrect early conclusions.
Authors
Jiaqi Wei, Wenxuan Huang, Chengdong Xu, Xiang Zhang, Wanli Ouyang
Abstract
Multi-hop retrieval-augmented generation (RAG) requires an agent to retrieve evidence for a sequence of dependent sub-questions. When a wrong answer lacks a required passage, the passage may be inaccessible to the retriever, or the agent may have failed to issue a query that would retrieve it. Standard retrieval evaluations conflate these two causes. We study how much apparent retrieval failure is instead caused by search drift: the search no longer tracks an unanswered sub-question. Holding the retriever, corpus, and generation model fixed, we introduce ExecutionRAG, a training-free requirement register that records open sub-questions and their supporting evidence. The register changes search guidance but not evidence access, so each gold passage it recovers certifies that the passage was reachable by the original retriever. On 1,000 MuSiQue questions, the number of wrong answers with unretrieved required evidence falls from 382 to 234 with all-MiniLM-L6-v2 and from 383 to 274 with Qwen3-Embedding-0.6B. These counts correspond to 69% to 44% and 66% to 48% of each system's wrong answers, respectively. Crucially, evidence recovery does not produce a proportional reduction in total error: premature resolutions and abstentions increase as missing-evidence failures decrease. The central finding is therefore a shift in the remaining bottleneck, from guiding search toward required evidence to verifying that the retrieved evidence is sufficient.