MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence
2026-08-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created MMArch, a new test to check if AI models can understand complex engineering images and apply engineering rules to answer questions. They made sure the test requires combining information from multiple images and thinking about engineering ideas, not just reading text or recognizing pictures. When they tested 18 AI models, both open-source and commercial, the best models scored much lower than human experts. The main problem for AI was using engineering principles and connecting information across different images. This shows there is still a lot of work needed to improve AI in engineering tasks.
Multimodal Large Language ModelsArchitectureCivil EngineeringBenchmarkVisual Evidence IntegrationEngineering PrinciplesShort-answer QuestionsModel EvaluationError AnalysisExpert Review
Authors
Chenxu Du, Kang An, Tengyue Wang, Zhongyu Yang, Xinqi Yang, Yuanchi Zhu, Hebao Zhu, Ziliang Wang, Faqiang Qian, Yunli Yang, Qibing Ren
Abstract
Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion. We introduce MMArch, a benchmark for architecture and civil engineering spanning ten subdomains and built entirely from figures in peer-reviewed papers. Its $1{,}212$ short-answer items are produced by a decoupled planner--writer pipeline and validated through automated screening, a blind adversarial audit, and expert review, so that answering requires perceiving the relevant evidence, identifying the governing principle, and applying it, not exploiting textual or single-figure shortcuts. Evaluating $18$ open-weight and proprietary MLLMs against a domain-expert panel, we find a wide gap: the strongest open-source model attains about $30\%$ and the best proprietary system $52\%$, while human experts reach $95\%$, more than forty points ahead. Our error analysis shows that failures concentrate in applying principles and combining evidence across figures rather than in locating it, pointing to substantial headroom for future research. Code and data are available at https://dcx-swjtu.github.io/MMArch/.