Multimodal models struggle to understand large engineering drawings
LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding
Computer Vision and Pattern Recognition
Summary
Engineering drawings are often very large and full of detailed information, making them hard to understand automatically. The authors created LED-BENCH, a test designed to see how well computer models can understand these big, complex drawings in fields like mechanics, architecture, and transportation. They tested 11 popular AI models and found that even the best only got about half the questions right. The models had more trouble when questions needed combining information from different parts of the drawings. This shows current AI has a hard time with detailed reasoning on complex engineering images.
What this means in practice
- •For engineering software developers: Improve automated tools that read engineering blueprints by identifying weaknesses in current AI understanding of large, detailed drawings.
- •For construction project managers: Use assessment methods from LED-BENCH to test if AI systems can reliably interpret architectural plans before deployment in real projects.
Authors
Ke Li, Tianyi Wang, Enning You, Chaoqun Wan, Zhen Huang, Shuxian Liang, Xian-Sheng Hua
Abstract
Real-world engineering drawings are often large-format and information-dense. Answering practical engineering questions therefore requires both fine-grained understanding of engineering information and complex reasoning over objects, relations, and evidence distributed across different regions or views. Existing benchmarks cover a range of engineering-drawing understanding tasks involving local regions, individual views, specific elements, or more constrained visual settings. However, systematic evaluation of understanding over realistic, large-format engineering drawings remains limited. To address this gap, we introduce LED-BENCH, a multidisciplinary benchmark designed to evaluate fine-grained understanding and complex reasoning over large-format engineering drawings, spanning the mechanical, architectural, and transportation domains. Extensive evaluation of 11 representative multimodal large language models, including 8 proprietary models and 3 open-source models, reveals substantial remaining difficulty: GPT-5.6 Sol achieves 52.65% average accuracy, performance varies across engineering domains, and all evaluated models perform worse on questions requiring broader evidence integration. These results highlight fine-grained understanding, evidence integration, and complex reasoning over large-format engineering drawings as persistent challenges for current multimodal models.