Better Retrieval, Worse Robustness:How Multi-hop RAG Amplifies Upstream ASR Errors
2026-08-24 • Computation and Language
Computation and LanguageInformation Retrieval
AI summaryⓘ
The authors studied how errors from speech recognition affect computer systems that answer complex questions using spoken input. They tested two methods meant to improve answering by linking entities and reformulating queries, finding these methods actually make error effects worse compared to simpler search approaches. Most problems came from misrecognizing important named entities in the questions. Even some simple fixes didn’t fully solve the issue, showing that the way the system processes information can amplify speech recognition mistakes.
Automatic Speech Recognition (ASR)Retrieval-Augmented Generation (RAG)Entity LinkingIterative ReformulationMulti-hop Question AnsweringDense RetrievalF1 ScoreNamed EntitiesError AmplificationSurface-form Mitigation
Authors
Zhenghua Bao
Abstract
Speech-based applications pass spoken queries through automatic speech recognition (ASR) before any retrieval module, so ASR errors enter the pipeline as a fixed upstream constraint. We empirically test whether two extensions to standard retrieval-augmented generation (RAG), entity-graph linking and iterative reformulation, absorb or amplify these errors. Using four English accents synthesized through neural TTS, we evaluate four RAG configurations on three multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA and MuSiQue) against a clean-text oracle. Although the structurally richer configurations generally retain higher absolute F1 under ASR input, both extensions amplify the error: the F1 gap from clean text to the highest-WER accent is 36-67% larger under their combination than under naive dense retrieval, on all three benchmarks. The dominant failure mode is corruption of one or more query entities, accounting for 87-96% of degradation cases on 2WikiMultiHopQA across all four methods. Two lightweight surface-form mitigations leave most of the gap intact, indicating that downstream retrieval structure amplifies remaining entity errors. We release code and data at https://github.com/ZhenghuaBao/spoken-multihop-rag .