OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors noticed that many tests for AI reading maps don’t require real understanding of the pictures since the images can be replaced by text without losing much accuracy. To fix this, they created OmniMapBench, a new test with over 2,000 questions on 1,600 maps that need true visual reasoning skills. They also made a way to measure how much the test depends on visuals called the Visual Dependency Index (VDI), showing OmniMapBench needs more image understanding than older tests. When they tried 25 top AI models, none scored perfectly, showing this test is hard and will help improve AI map reading abilities.
Large Vision-Language Models (LVLMs)Visual GroundingMap DocumentsVisual ReasoningBenchmarkVisual Dependency Index (VDI)Multi-step ReasoningDocument UnderstandingQuestion-Answering
Authors
Yang Chen, Yunwen Li, Yufan Shen, Minghao Liu, Tianyu Zheng, Bin Fu, Qunshu Lin, Zhi Yu, Botian Shi
Abstract
Recent advancements in LVLMs necessitate robust benchmarks for complex, visually grounded reasoning. A critical limitation is identified in many document understanding benchmarks: visual content is often reducible to text, enabling high performance without genuine visual grounding. To address this limitation, OmniMapBench is introduced to foster visual-centric reasoning for map documents. The benchmark comprises 2,096 manually annotated question-answer pairs across 1,603 map documents from nine categories. It is designed to probe a hierarchy of skills, ranging from perception to multi-step visual reasoning. To quantify benchmark properties, a simple yet effective benchmark-level metric is proposed: the Visual Dependency Index (VDI), defined as the accuracy drop when images are replaced with question-agnostic descriptions. OmniMapBench exhibits higher VDI than established benchmarks, which quantitatively validates its focus on irreducible visual reasoning. Comprehensive evaluations of 25 leading LVLMs are conducted on OmniMapBench. A significant performance gap is observed, with the top-performing model achieving only 75.03\% accuracy. This result underscores the challenges posed by OmniMapBench to current LVLMs. This work aims to catalyze progress in visual-centric reasoning for document understanding of LVLMs. The dataset and code are publicly available at https://github.com/SIGMME/OmniMapBench.