Better deciding when to revise answers improves question accuracy
Return or Revise? Learning When Revision Helps Retrieval-Augmented QA
Computation and LanguageInformation RetrievalMachine Learning
Summary
When AI systems answer questions by first making a draft response and then retrieving extra information to revise it, it can be tricky to know if revising will actually help. The authors studied how to predict whether revising an answer will improve it or cause harm. They trained a system to estimate this 'recoverability' before attempting revisions, which led to more accurate answers than always revising or just trusting the original draft. However, sometimes harmful revisions still occur, and choosing from multiple answer options can work better in some setups.
What this means in practice
- •For ai system engineers: Improve answer accuracy by implementing policies that decide whether to revise AI-generated drafts based on predicted improvement.
- •For customer support platforms: Use better revision decision-making to enhance automated response systems that retrieve information to update draft answers.
Authors
Nicholas Kashani Motlagh, Tim Anderson, Jeremy Gwinnup, Grant Erdmann
Abstract
We consider the decision of whether to return an existing draft answer or revise it using retrieved evidence, as in answer-revision systems. Draft confidence estimates whether the current answer is correct, but the decision requires estimating the effect of a specified revision. For offline training and evaluation, we grade both the returned draft and its candidate revision under the same correctness judge, which makes repair, harm, and the gap to an oracle observable. We call this paired effect its recoverability, and we train policies to predict it before revision. On 25,870 held-out open-domain questions across three revision setups, a scorer trained on the paired outcome has greater area under the accuracy--revision-rate curve than a matched draft-correctness scorer in all nine Llama setup--seed fits, and gains 0.23--0.68 accuracy points on average at development-selected thresholds, a difference significant across training runs only for dense retrieval. The resulting policy improves on always revising and on average closes more than a third of the oracle gap, although it still applies 38--46% of the harmful revisions. When a draft-free standard-RAG answer is also available, however, choosing between the draft and that answer is stronger by about two points for Llama and four for OLMo, and adding candidate revision as a third option yields no significant gain. Recoverability describes one revision; its value as an available action also depends on the alternatives.