GeoWorld platform tests AI decisions in changing real maps
LiveGeo: Benchmarking Geographic Agents in Continuously Evolving Worlds
Artificial Intelligence
Summary
Deciding right in a world that keeps changing is hard, even for advanced AIs. The authors built GeoWorld, a platform that mimics real geographic places that evolve over time. They also created LiveGeo, a series of tests where AI agents must gather current facts, check old plans, and update them when things change. Their findings show that current language models often fail to keep decisions correct as the world shifts. This work helps improve AIs that need to act reliably in dynamic geographic settings.
What this means in practice
- •For geospatial software developers: Build AI agents that update plans reliably as geographic data changes over time using the GeoWorld platform and LiveGeo benchmark.
- •For emergency management teams: Use insights from LiveGeo benchmarks to improve AI tools that must track evolving disaster zones and adapt response plans promptly.
Authors
Pengcheng Zhou, Xuanyu Liu, Yanchen Yin, Tianjie Ju, Bobo Li, Shengqiong Wu, Hao Fei, Mong-Li Lee, Wynne Hsu
Abstract
We introduce GeoWorld, a controllable and continuously evolving geographic environment platform for LLM agents, and LiveGeo, a benchmark for evaluating reliable decision-making in real-world geographic environments using GeoWorld. Unlike existing evaluations that largely reason and plan over fixed geographic states, GeoWorld is grounded in real-world geographic structure and allows external world states to evolve along their own temporal processes while preserving verifiable state histories. LiveGeo places geographic tasks within this evolving environment, requiring agents to actively acquire task-relevant information, verify whether the evidence supporting prior decisions remains valid, and revise their plans when necessary. Our results reveal that even frontier LLMs struggle with static tasks, with no model consistently excelling across tasks, while strong static performance does not ensure dynamic reliability. Further analysis identifies a key bottleneck: agents struggle to acquire and translate relevant evidence into valid decisions and maintain their correctness as the world evolves. By releasing LiveGeo together with its GeoWorld platform, we aim to provide the community with a flexible and extensible infrastructure for studying the next generation of agents capable of reliable action in evolving geographic worlds.