Vision language models struggle with numbers in city images
USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
Many computer programs can look at pictures and answer questions about cities, but they often have trouble with math and counting tasks related to those images. The authors created a new test called USAI-Quant to check how well these programs handle number-based questions about city features, using satellite images of large U.S. cities. They found that current best models still do poorly with such math questions, and they analyzed where and why these struggles happen.
What this means in practice
- •For urban planners: Use the USAI-Quant benchmark to evaluate and improve AI tools for measuring city features from satellite images.
- •For remote sensing analysts: Assess quantitative question answering skills of vision-language models when analyzing urban satellite data.
Authors
Dongdong Wang, Qingqi Song, Yuzhou Chen, Deepak Balakrishnan, Ravi Shankar Srinivasan, Shenhao Wang
Abstract
Large vision-language models (VLMs) have emerged as a powerful paradigm for urban and spatial AI. However, current state-of-the-art large VLMs still struggle with quantitative reasoning on remote sensing imagery. Existing benchmarks and algorithms are predominantly based on qualitative Visual Question Answering (VQA), providing limited insights into the quantitative reasoning capabilities of VLMs for built environment metrics. To address this gap, we develop Quantitative Urban and Spatial AI benchmark (USAI-Quant), the first benchmark designed to quantitatively evaluate VLM's reasoning capabilities on built environment metrics via remote sensing imagery. USAI-Quant is curated from the 335 largest U.S. cities, aligning high-resolution remote sensing images with quantitative built environment metrics. We then evaluate both general-purpose and remote sensing VLMs (RS-VLMs) by applying VQAs to tens of built environment metrics across three complexity levels. Our results reveal that current state-of-the-art models consistently fall short on numeric reasoning tasks. We further conduct in-depth analyses across models, question types, and geographic locations, uncovering insights into performance variability and task-specific challenges.