GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks
2026-08-07 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageInformation RetrievalMachine Learning
AI summaryⓘ
The authors created GeoBenchLLM, a new benchmark to test how well large language models (LLMs) understand geographic data. They used twelve different public datasets covering various geo-related tasks to see how these models handle space and time information. Their tests showed that both the reasoning ability and the size of the LLMs greatly affect how well they perform. This benchmark helps better understand and compare LLMs on geography-based questions.
Large Language ModelsBenchmarkingGeospatial DataTemporal UnderstandingGeneralizationDatasetReasoningModel Size
Authors
Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno, Lynda Tamine
Abstract
In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities. In this paper, we present \benchName, a comprehensive benchmark for probing LLMs on geo-related tasks. We leverage a careful selection of twelve publicly available datasets from diverse geo-related tasks and domains, and evaluate a set of LLMs on geo-spatial and temporal understanding using our benchmark. Our results show that reasoning and size have a strong impact on overall performance. GeoBenchLLM is publicly available at https://github.com/Rfr2003/GeoBenchLLM.