SearchAtlas maps how AI search agents gather and use evidence
SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs
Computation and Language
Summary
Many AI agents search for answers but only their final answers are checked for correctness, not how they found them. The authors created SearchAtlas, which turns the complicated search steps into easy-to-read graphs showing how pieces of evidence lead to an answer. This helps spot problems in the search process that often cause wrong answers. Their method works well compared to human analysis and shows insights other judges miss.
What this means in practice
- •For ai system developers: Monitor and improve AI search agents by visualizing how they gather and connect evidence to answers during search sessions.
- •For enterprise knowledge management teams: Audit automated Q&A systems to detect when search processes fail, ensuring more reliable information retrieval for business use.
Authors
Jiacheng Sang, Mengyuan Li, Sanxing Chen, Yukun Huang, Yu Feng, Bhuwan Dhingra
Abstract
LLM search agents are often evaluated on final-answer accuracy, overlooking the process. Analyzing a search strategy requires understanding how credible evidence is retrieved to address question constraints. This valuable information is buried in raw search trajectories that are long and difficult to parse. We introduce SearchAtlas, a framework that converts search trajectories into structured graphs whose edges represent how evidence is propagated across the reasoning trace, from the query that retrieves it to the final answer. Our automated parsing pipeline achieves a mean edge F1 of 86.0% against human-annotated graphs and remains consistent across repeated runs. We analyze five search agents on three benchmarks, revealing systematic differences in search scale and evidence aggregation. SearchAtlas exposes fragmented answer support, question constraints that do not reach the answer, and unverified parametric knowledge entering the response. These process failures are strongly associated with incorrect answers, even more so than an LLM judge given either the raw trajectory or the ordered query list, suggesting that the constructed graphs provide useful interpretability. Moreover, an audit of cases in which process-diagnostic scores disagree with final-answer correctness shows that they capture information not reducible to answer accuracy.