Adaptive search auditing improves trust in deep research agent reports
Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents
Artificial Intelligence
Summary
When AI research agents gather information, they pick documents step-by-step, which can lead to biased or incomplete conclusions. The authors found that checking only if cited sources support claims misses this bias from selective document search. They created a method called CESS that adjusts for this selection bias, making it clearer whether the gathered evidence fairly represents all available documents. Their method helps evaluate if reports truly reflect the full range of evidence and not just the documents the agent happened to read.
What this means in practice
- •For medical data teams: Audit and correct biases in AI-assisted systematic literature reviews to ensure evidence reports fairly represent all available studies.
- •For legal document analysts: Improve accuracy in AI-generated case summaries by correcting for adaptive document selection biases during evidence collection.
Authors
Shuyao Xiao, Shengling Wang, Xuan Chen, Ke Chao, Ming Cui, Feifei Qian, Chaoyang Mei, Fanlin Meng, Lulu Wang, Ziming Yu, Junxi Yin
Abstract
Deep Research agents synthesize evidence into cited reports, yet a well-cited report can still reach a misleading conclusion. Citation correctness checks whether cited sources support individual claims. It does not show whether adaptive search exposed a representative view of all documents made available for evaluation, which we call the candidate pool. Early findings redirect later queries, document choices, and stopping, so the documents an agent reads form a selective sample. Existing evaluations rarely account for this selection. We formulate the problem as adaptive evidence sampling and introduce Causal Evidence Selection Correction (CESS). CESS predicts each candidate document's evidence direction and corrects the candidate-pool average using the logged probabilities of selecting each document and reaching each search round. Shrinkage stabilizes short searches, while intervals replace point estimates when some documents cannot be sampled. We also prove that estimating the average evidence direction of a common pool differs from measuring how a change in search policy alters the evidence read. The latter requires intervention. On questions from the MS2 systematic-review benchmark, CESS reduces mean absolute error against the candidate-pool average by $9.2\%$ and reduces the estimate's change under opposing document rankings by $39.4\%$ relative to averaging the evidence scores of documents read. Across trajectories from a public Open Deep Research agent, the corresponding reductions reach $60.1\%$ and $87.2\%$. A further 4,800 trajectories under paired interventions confirm that correcting a pool estimate and measuring a policy effect are different tasks. CESS therefore audits whether the evidence direction underlying a report reflects the documents available for evaluation, while a separate intervention analysis measures the effect of search decisions.