Vision language agents benchmarked on interactive geolocation tasks
GeoGuesser Bench: Benchmarking Vision-Language Agents on Interactive Geolocation
Artificial IntelligenceMachine Learning
Summary
Finding your location just from pictures is a tough challenge for AI because it needs both knowledge about the world and the ability to explore its surroundings. The authors created GeoGuesser Bench, a set of tasks where AI models act inside a simulated Street View and try to guess locations or find objects. They tested many models and found that being good at using world knowledge doesn't always mean being good at exploring. Their results show where current AI struggles and provide tools to improve training for better exploration skills.
What this means in practice
- •For navigation system developers: Use benchmarked benchmarks to improve AI that locates places from limited visual input in mapping and localization applications.
- •For robotics engineers: Develop AI agents capable of exploring real-world environments and recognizing landmarks by simulating and testing them with GeoGuesser Bench.
Authors
Huanzhi Mao, Yichuan Wang, Michael Koo, Wenyuan Huang, Huy Doan, Jinghan Xia, Ziyao Wang, Joseph E. Gonzalez
Abstract
Vision-language model (VLM) agents that act in the open world need two abilities: world knowledge to interpret what they see, and exploration to find what they cannot yet see. We study how agents combine the two through interactive visual geolocation. GeoGuesser Bench contains 282 tasks in an offline Street View environment where agents rotate, zoom, and move between real panoramas. Its three task types shift the balance from knowledge to exploration: agents (1) report the coordinates of their starting location, (2) name the enclosing street, city, state, or country, or (3) find, read, and count nearby objects. The coordinate tasks in (1) also include 42 real GeoGuessr World Championship rounds, whose expert guesses provide a human reference. We evaluate 23 VLMs: the best accuracy is 62.5% on place naming and 35.0% on object finding, and the best agent averages 4352.4 GeoGuessr points on the championship rounds, against 4770.3 for the better expert in each round. Beyond these headline numbers, world knowledge and exploration are only weakly correlated across models: agents that localize well do not necessarily explore well. To understand where agents fail, we categorize their errors into vision, navigation, and tool-use failures. Further ablations show that the strongest agents, such as Claude Opus 4.8, still perform well with limited views, while weaker agents degrade; customizing the environment thus adjusts task difficulty. The environment's verifiable rewards and restorable state also support training, and our Planner-Worker-Compressor harness cuts planner tokens by 57-61% by compacting the interaction history. We hope GeoGuesser Bench and its environment serve as a high-quality foundation for simulating real-world exploration at scale.