Adaptive retrieval improves answer reliability with fewer responses
Adaptive Retrieval, Selective Answers: Evaluating Conformal Factuality in Agentic RAG
Computation and LanguageArtificial Intelligence
Summary
The paper looks at how computer programs can better get facts when answering questions by checking their own confidence in the facts they found. The authors study a system that decides whether to search for more information or stop and answer based on how certain it is. They tested this with different language models and question sets and found that as they made the system more strict, the answers it gave were more likely to be fully supported but also less often given. Their work explains the trade-off between having more reliable answers and having fewer answers overall.
What this means in practice
- •For natural language processing engineers: Design question answering systems that adaptively retrieve information while controlling answer reliability and availability trade-offs.
- •For chatbot developers: Build chatbots that can decide to gather more data or answer early to balance fact-checking confidence with response speed.
Authors
Muhammad Aimal Rehman, Chi-Kuang Yeh
Abstract
Retrieval-augmented generation (RAG) can improve factual grounding without guaranteeing that every generated claim is supported. Previous work has studied claim-level conformal factuality in single-stage RAG and, more recently, multi-hop RAG. We examine its behavior in agentic RAG, where a language-model policy adaptively issues follow-up retrieval queries or terminates retrieval before response generation. We retain the retrieval-based relevance-scoring formulation and post-generation conformal filtering and evaluate six model-dataset configurations: GPT-6 Luna and Llama 3.1 8B on HotpotQA, Natural Questions, and TriviaQA. Each configuration uses 60 fixed questions, 100 seeded 30/30 calibration-test partitions, five conformal target levels, and a no-conformal-prediction baseline. Across the six configurations, higher target levels are accompanied by higher observed full-support rates and reduced answer availability. At the 95% target, non-empty response coverage ranges from 17.23% to 60.17%, claim retention from 10.72% to 43.16%, and full support among non-empty outputs from 89.14% to 95.56%. We formulate the adaptive retrieval trajectory explicitly and characterize the reliability-informativeness trade-off when conformal factuality is applied after agent termination.