Search technique decides when web evidence is enough for answers
Knowing When the Evidence Is Enough: Requirement-Directed Search with Horn-Closure Certificates
Information RetrievalArtificial IntelligenceComputation and Language
Summary
When a search engine tries to answer questions from the web, it needs to know if it has found enough information, or if it should keep looking. The authors created a method called Certify that breaks down a question into smaller fact-checks and uses a structured approach to find and verify the right pieces of information. This method can confidently decide if the search has enough proof to answer or if more searching is needed, improving accuracy and reliability compared to other approaches.
What this means in practice
- •For search engine developers: Improve question-answering systems by providing a precise way to decide when gathered information is enough to answer without unnecessary extra searches.
- •For digital assistants teams: Make virtual assistants more reliable by structurally verifying facts from search results before responding to user queries.
Authors
Xinyu Zhang, Weiwei Sun
Abstract
After each Web search, a search-augmented model must decide whether its evidence suffices: answer, search again, or abstain. Prevailing systems leave this to holistic model judgement, which can accept a context naming every entity yet lacking one required fact. We instead make sufficiency structural: a per-query requirement hypergraph states which facts jointly yield another fact. This raises two challenges: deriving requirements from the question alone, and linking each requirement to passages about the right entity. Certify therefore decomposes the question, searches per requirement, and accepts links only after entity checks. Horn closure over the links yields a certificate, the minimal sufficient passages and their derivation, or an abstention naming open requirements. Given correct retrieval, requirements and links, the decision is sound, and complete up to the entity check. Pooled over three corpora with unanswerable twins, Certify decides significantly more accurately than search agents, graph retrieval and sufficiency judges in default configurations. On the same evidence, it matches a proprietary model's sufficiency label without an additional model call. Over 21M Wikipedia passages, entity checks add 4 to 13 points of decision accuracy where gold evidence exists. Finally, answers from its certificate are 80% correct, against 57% from all retrieved passages.