Benchmarking ai agents for diagnosing urban problems with multimodal data
DUDA-Bench: Benchmarking LLM Agents on Multimodal Data-Driven Urban Diagnosis
Artificial Intelligence
Summary
Urban diagnosis helps find problems in cities by analyzing many types of data to decide what needs fixing and where. The authors created DUDA-Bench, a set of tasks that tests how well AI agents can perform this complex, multi-step analysis using data from 12 cities. They found that while AI models can handle some individual tasks, they struggle to complete the full diagnosis from start to finish, especially when adapting plans or verifying information. This benchmark helps show where AI still needs improvement to support real-world city planning and management.
What this means in practice
- •For urban planning teams: Test and improve AI workflows that analyze multiple data types to identify and localize city problems for better planning decisions.
- •For environmental monitoring teams: Evaluate AI agents on coordinated analysis across data sources to detect environmental issues and their causes in urban settings.
Authors
Yizhi Song, Hang Ni, Weijia Zhang, Hao Liu
Abstract
Urban diagnosis integrates heterogeneous observations to identify urban problems, localize affected areas, and investigate contributing factors, informing evidence-based urban planning and management. However, its reliance on labor-intensive, case-specific expert workflows limits scalability and reuse, motivating the exploration of agent-based execution. To evaluate this capability, we introduce DUDA-Bench, a hierarchical and interactive benchmark that formalizes data-driven urban diagnosis as a multi-stage agent workflow. It comprises 86 atomic and 22 workflow tasks spanning four analytical stages, grounded in multimodal data from 12 cities covering five urban problem types. Evaluations of seven backbone models and five agent systems reveal a substantial gap between isolated analytical competence and end-to-end diagnosis, with system benefits varying across backbones. Trajectory analysis shows that unresolved evidence gaps propagate across stages, while successful recovery involves revising assumptions and actions using feedback. These findings highlight limitations in coordinating analytical capabilities across stages, particularly adaptive planning, evidence integration, and verification. More broadly, DUDA-Bench provides a framework for translating expert analytical workflows into hierarchical agent tasks and process-aware evaluation, supporting systematic assessment of end-to-end analytical capabilities.