Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
2026-08-12 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created a new test called Diagram-MMU to check how well multimodal large language models (MLLMs) can understand and work with scientific diagrams. Their test includes thousands of diagrams and questions, focusing on turning diagrams into coding commands, editing those codes, and answering questions about diagrams. They found that models find it harder to convert and edit diagrams into code than to answer questions about them. Some models improve with extra guidance while doing coding tasks but get worse at answering questions, except for one model called Claude-4.6 Opus, which did better on all tasks.
Multimodal Large Language ModelsScientific diagramsDiagram parsingLaTeX TikZ codeDiagram-to-code generationDiagram question answeringBenchmark datasetAgentic settingsModel evaluation
Authors
Weihao Bo, Shan Zhang, Yanpeng Sun, Jie Liu, Yongke Yao, Jinhao Du, Wei He, Kai Zou, Zechao Li, Jingdong Wang
Abstract
Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.