Predicting intermediate answer quality to improve multi-step question answering
Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation
Information Retrieval
Summary
Answering complex questions often requires gathering information in several steps and building up an answer gradually. The researchers looked inside these step-by-step processes to see how good partial answers get after each step. They found that many steps add little new value and created ways to predict when an answer is good enough to stop early. This can save time and effort while keeping most of the final answer’s quality.
What this means in practice
- •For ai system builders: Reduce computation in multi-hop question answering systems by predicting when intermediate answers are sufficiently accurate to stop early.
- •For customer support teams: Use early prediction of answer quality to speed up automated answer generation in complex query handling workflows.
Authors
Fangzheng Tian, Debasis Ganguly, Craig Macdonald
Abstract
Agentic Retrieval-Augmented Generation (RAG) has become a promising paradigm for multi-hop question answering, where a reasoning model iteratively issues queries to a retriever and incorporates newly retrieved context into subsequent reasoning steps. While this iterative process can improve final answer quality, current evaluations of agentic RAG largely focus on end-to-end outcomes and provide limited visibility into how a model's answer state changes during generation. In this work, we introduce an in-trajectory probing framework to study intermediate answer states in agentic RAG. Specifically, after each retrieval-reasoning iteration, we force an agentic model to stop reasoning and generate an intermediate answer based on its current state. This allows us to define two iteration-level measures: partial answer quality at each iteration, and partial utility as the change in partial answer quality across iterations. Our analysis across multi-hop QA benchmarks reveals that partial answer quality often plateaus before natural termination, with many later iterations contributing only small measurable improvements. Accordingly, we formulate two prediction tasks, partial answer quality prediction and partial utility prediction, and study trajectory-derived signals from intra-iteration, inter-iteration, and query-iteration perspectives. Experiments show that partial answer quality is more predictable than partial utility, with supervised models achieving Pearson's r above 0.43 for quality prediction. Finally, using predicted answer quality and utility for early stopping reduces average iteration count by about 11% while preserving about 98% of the final answer quality achieved by natural stopping.