Vision language models improve diagram topology extraction accuracy
Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Diagrams are used to show complex systems with many connected parts, but understanding all the parts and their connections fully is hard for machines. The authors created a big set of diagrams called Knossos to help train and test computers on this problem. They also designed a method named Ariadne that breaks down the task into smaller steps, which helps computers do better at identifying all elements and their links. Their approach improved the performance of vision-language models on diagrams across many different types.
What this means in practice
- •For engineering design teams: Extract complete system diagrams to automate understanding and verification of engineering schematics.
- •For technical documentation writers: Automatically generate accurate relational graphs from complex procedural diagrams to improve document clarity and navigation.
Authors
Bangwei Guo, Xujiang Zhao, Shengyu Chen, Yanchi Liu, Wei Cheng, Xi Zhu, Guoning Zhang, Dimitris N. Metaxas, Haifeng Chen
Abstract
Structural diagrams are widely used to represent complex systems and relational information across scientific, engineering, procedural, and spatial domains. Recent vision-language models (VLMs) have become increasingly capable of recognizing diagram elements and reasoning about their content, while complete diagram topology extraction remains comparatively underexplored. In this paper, we study diagram-to-graph topology extraction: extracting all diagram entities and the complete relations among them. To enable large-scale supervised training and systematic evaluation of this task, we introduce Knossos, a benchmark of 19,200 diagrams across six diverse domains, with 245,179 nodes and 439,740 edges. Its symbolic generation process provides exact alignment between rendered diagrams and annotations of complete topology, relation types, and connector geometry. To address the modeling challenge of complete topology extraction, we also present Ariadne, a structured framework that decomposes the task into node inventory extraction and source-conditioned edge prediction. Extensive experiments show that training on Knossos substantially improves complete topology extraction in smaller open-source VLMs. Ariadne further improves over one-step extraction under matched supervision, demonstrating the additional benefit of structured decomposition. It achieves the highest average Edge F1 among the evaluated methods on Knossos, while both backbone variants also improve over their unadapted counterparts on the real-world external benchmark. Code and benchmark are available at https://github.com/bangwayne/knossos_Ariadne_Public.