Large language models evaluated on long web search tasks with error tracking
LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
Artificial Intelligence
Summary
Finding information across many web pages can be challenging for AI models, especially when answers require combining lots of details. The authors created a new way to test how well these models do at searching the web over long sequences of steps. Their system keeps track of each search interaction and helps find exactly where the model went wrong if it produces a bad answer. They tested many models on real tasks and verified that their evaluation reliably pinpoints errors and supports detailed checks.
What this means in practice
- •For ai developers: Test and compare AI systems on extended web search tasks to improve how models gather and verify information over multiple steps.
- •For enterprise search teams: Analyze where AI search assistants fail during long research sessions to improve reliability and transparency in complex query answering.
Authors
Jialiang Liu, Shusheng Li, Wenjun Tan
Abstract
Long-horizon web search requires LLM-based agents to integrate information across web pages through extended interactions under task-specific constraints. Evaluating this capability is important not only for measuring final answer quality, but also for assessing whether an agent acquires, verifies, and uses evidence reliably over extended interactions. Existing evaluations, however, mainly rely on simulated trajectories or on post-hoc trajectories reconstructed from final answers, rather than on retained real interaction traces, which limits the fidelity of behavioral assessment and weakens support for fine-grained error localization. To address this gap, we introduce LONGSEARCHEVAL, a trajectory-aware agent evaluation framework that combines a unified agent execution protocol with source constrained outcome verification and retrospective failure-node localization. Stage 1 verifies answer groups against agent-cited webpages without human-authored gold answers; Stage 2 audits normalized rounds and links flagged failure nodes to incorrect answer groups. We evaluate 16 models from eight families on 100 tasks, yielding 1,600 retained runs and 6,545 automatically flagged failure nodes. Trajectory collection and evaluation required 1,037 and 318 hours of cumulative runtime, respectively. Human validation yields 94.50% agreement on supplied answer-group labels and 98.11% feature-correspondence precision for audited round-group links. Together, outcome scores and linked trajectory evidence support controlled model comparison and retrospective inspection of answer-related content.