Atlas-bench tests local search agents finding places that fit needs
ATLAS-Bench: An Agentic Tool-Use Benchmark for Local Area Search
Artificial Intelligence
Summary
Finding places that match all your requirements and preferences in local searches is hard for AI agents. The authors created ATLAS-Bench, a set of 1,200 local search queries with clear must-have needs and preferences, plus real reviews and place details. They also built ATLASBox, a tool that lets AI agents try finding places given a location and time, checking their answers for accuracy and usefulness. Tests show even top models struggle to find fully satisfying places consistently, revealing the difficulty of precise local search.
What this means in practice
- •For local search developers: Evaluate and improve AI agents that return place recommendations meeting user needs with evidence from attributes and reviews.
- •For chatbot builders: Test and enhance chatbots that support real-time, location-based queries with mandatory and preferred place criteria.
Authors
Keyun Zhang, Dapeng Yin, Fanfu Wang, Shuang Lang, Zhenyu Zhu, Yulin Huang, Chong Yang, Pengjie Wang
Abstract
Reliable local search is essential for LLM agents to support everyday decisions and longer plans by finding places that meet users' needs at given times and locations. Yet existing benchmarks provide limited coverage of whether search results satisfy mandatory requirements, account for preferences, and ground factual and experiential judgments in place attributes and reviews. We introduce ATLAS-Bench, a benchmark of 1,200 queries across three regions, expanded from real search events using place attributes, reviews, and existing user profiles from the platform. Each query distinguishes mandatory requirements from preferences and factual conditions from subjective judgments, with supporting evidence and criteria for checking each requirement. Queries are validated through iterative assertion checks and cross-model review. We also introduce ATLASBox, a standardized evaluation environment with search tools and a fixed point-of-interest (POI) knowledge base. Given a query, timestamp, and location, an agent searches place attributes and reviews to recommend evidence-supported options or report that no place meets all mandatory requirements. We evaluate answer quality, trajectory quality, and execution cost. Across 14 configurations of 11 models, GPT-6-Luna achieves the highest constraint satisfaction rate among returned recommendations (71.5%) and factual accuracy (95.6%). However, it returns at least one acceptable recommendation for fewer satisfiable queries than GLM-5.3 (67.3% versus 85.3%). These results highlight the challenge of satisfying individual requirements while providing usable recommendations across queries.