Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search
2026-07-16 • Information Retrieval
Information RetrievalComputation and Language
AI summaryⓘ
The authors show that evaluating documents for search only by how well they answer the current question misses an important part of how language model agents use documents. They find that some documents don't seem useful on their own but are crucial because they help the agent decide what to ask next. By removing documents and seeing how it affects the agent's behavior and answers, they identify these 'bridge' documents that guide the search process. This means that usefulness of documents to an agent is different from static relevance, and focusing only on relevance doesn't capture their true value in multi-step reasoning.
retrieval systemslanguage modelsReAct agentHotpotQAcounterfactual analysisstatic relevancebridge documentsquery reformulationentity relevancecausal usefulness
Authors
Debayan Mukhopadhyay, Utshab Kumar Ghosh, Shubham Chatterjee
Abstract
Retrieval systems are trained and evaluated on a static idea of usefulness: hand a document and a question to a reader model, see whether the answer improves, and score the document accordingly. The idea holds up when a document is read on its own. It breaks when a language model works as a search agent, issuing several queries and reasoning across turns, because a document can matter for what it lets the agent do next rather than for what it says about the current question. We measure that gap rather than argue it. Using a ReAct style agent over HotpotQA, we replay 1000 development questions and, for every document the agent read, delete it and re-run the rest of the trajectory from that point. Comparing the original run against its counterfactual gives a Counterfactual Trajectory Utility (CTU) score from three deltas: final answer quality, next query retrieval quality, and turn count. Crossing CTU against Static RAG Utility (SRU) over 23,322 document observations, the two are close to statistically independent (Spearman rho = -0.026). Roughly a third of the documents the agent reads are causally load bearing while looking useless to a static reader; we call these bridge documents. The pattern survives when the reader based axis is swapped for a BM25 and cross encoder proxy, giving a bridge cell of 27.2% on an evenly spread axis. A second experiment pins down the mechanism. Using the Observable Entity Relevance (OER) measure from prior work, entities that discriminate relevant from non-relevant candidates appear in the agent's next query 4.02 times more often than entities found only in non-relevant documents (6.1% vs 1.5%, n = 227,139). A bridge document earns its keep by handing the agent a discriminative entity that redirects the search. Static relevance and causal usefulness are different quantities in agentic retrieval, and optimizing the first does not deliver the second.