Benchmark measures when extra AI search is worth the cost
SearchHorizonBench: A Benchmark for Search Value in AI Agents
Artificial IntelligenceMachine Learning
Summary
Deciding when an AI should spend more time searching for answers is tricky because extra searches use resources but might improve results. The authors created SearchHorizonBench, a large dataset and tool to help AI learn when extra searching actually helps and by how much. They found that current methods only explain a tiny part of the benefit from extra searching, showing room to improve AI decision-making. This benchmark includes thousands of example questions and AI answer attempts to test and develop smarter search strategies.
What this means in practice
- •For ai system developers: Train and evaluate AI agents to decide when additional search effort improves answers efficiently, reducing wasted computation in complex tasks.
- •For data analysts: Improve data-driven decision tools that need to balance resource cost versus expected gain by modeling search value from past execution experience.
Authors
Jingjie Ning, Xueqi Li, Yibo Kong
Abstract
AI agents must decide when another search is worth its cost. Answer-success probabilities capture question difficulty, while this decision requires estimating the additional benefit of search. We introduce SearchHorizonBench, a benchmark for learning and evaluating search value from execution experience. Paired continuations and repeated executions separate expected search gain from generation noise and connect prediction quality to decision utility. The resource contains 45,632 executions across 3,168 questions. On 1,632 held-out questions, historical calibration reduces Brier error by 16.2%, with the improvement concentrated in success-level information. Search gains have a standard deviation of 21.4 percentage points, and tested historical estimators explain less than 2% of this variance in point estimates. Matched-configuration learning curves probe this distinction across training scales. At the reference token price, free selection observations yield a 2.49-point utility opportunity over the best constant action, driven primarily by gain information. Reusable histories, frozen splits, strong baselines and scoring tools provide a common foundation for developing and evaluating value-aware search agents.