Summary
This paper shows that popular ways to judge if a conversation-answering system should refuse to answer are often wrong because they rely on incomplete labels of relevant documents. The researchers found that better retrieval systems still find answers in contexts without labeled relevant documents, so treating such questions as unanswerable is misleading. They had multiple models and human judges check if the answer was present regardless of labels and found many misses in current evaluation methods. The authors recommend using content-based labels rather than just relying on presence of gold-labeled documents to score when a system should abstain.
What this means in practice
- •For search engine developers: Improve evaluation of conversational search systems by scoring refusals against actual content presence rather than incomplete gold labels.
- •For chatbot platform engineers: Refine model abstention mechanisms in question answering by detecting when answers actually exist despite missing relevance annotations.
Abstract
Conversational RAG benchmarks such as MTRAG and RETECO evaluate retrieval and generation end to end: each system answers from its own retrieval, so abstention must be scored against whether each retrieved context is sufficient. A convenient shortcut reads sufficiency off the relevance judgments and treats a context without gold documents as unanswerable. We re-run the official baselines exactly and test this shortcut on RECOR (RETECO track~2) with four retrievers (nDCG@10 from .18 to .59) and three Qwen3 readers. Two open LLM judges and a panel of four small open models label each context by content, and a model of another family audits 250 contexts blind, with adjudication. The stronger the retriever, the larger the share of no-gold contexts that still contain the answer (28\% to 72\%) and the lower the shortcut's precision (.68 to .10; .70 to .21 under the audit). A reader agrees: Qwen3-4B answers 62--67\% of these false misses correctly, against 23--30\% of no-gold contexts judged insufficient and 1\% with no documents. In most false misses, the answer comes from a hard negative mined for the same question, another turn's gold or a neighbouring chunk of the gold page. On MTRAG, whose judgments come from annotators, precision drops from .72 to .50. Under the shortcut, answerability AUROC is up to .19 too low, and the best sufficiency detector appears to be a retrieval score that never reads the documents. We release code, runs and labels, and recommend scoring refusals against content labels, not the presence of gold.