Evidence Interfaces Shape How Retrieval-Augmented Readers Use Support
Artificial Intelligence
Summary
The authors studied how multi-hop question answering systems retrieve and use supporting evidence. They found that just scoring the top retrieved answers can hide problems, like missing parts of the needed evidence or the system not using the evidence well. By testing with datasets where the full support is known, they showed that missing support explains many errors and that better retrieval and ranking based on support can improve results. Their work suggests reporting answer scores alongside checks that all necessary support is included to better understand system performance.
Authors
Junchi Liao, Jiawen Deng, Fuji Ren
Abstract
In multi-hop RAG evaluation, a top-k answer score can hide two different failures: the retrieval window may drop part of the support chain, or it may contain support in a form the adapted reader does not use well. We call this reader-facing form of retrieved evidence an evidence interface. Using three support-annotated multi-hop QA benchmarks, we compare matched adapted readers trained with raw context, retrieval windows, and gold-support diagnostic renderings. These comparisons distinguish support-availability failures from remaining reader-interface effects. Top-k windows become interpretable only after checking whether the complete annotated support chain survives: when it does, short ranked windows can match or improve over raw context; when it does not, missing support explains much of the loss. Gold support-first improves matched readers; on 2Wiki and MuSiQue, a support-supervised ranker raises coverage and recovers raw-context quality at lower prompt cost, while retaining gold headroom. Support-removal checks further show that the gains rely on exposed evidence, not only answer priors. On support-annotated evaluations, top-k answer scores should therefore be reported together with complete-support coverage.