System recovers editable graphics code from diagram images
Back2Struct: Making Structured Images Editable Again
Computer Vision and Pattern Recognition
Summary
Images of diagrams and charts are often saved as pictures that cannot be easily changed without redrawing them. The authors introduce Back2Struct, a tool that turns these pictures back into editable graphic code. Instead of just tracing lines, it understands the parts like text and shapes, so the output can be edited in common software like PowerPoint. They also trained the system to make sure the output code is correct, clear, and looks like the original image.
What this means in practice
- •For graphic designers: Edit and reuse diagrams or charts from images directly inside popular design tools by converting them back into vector code.
- •For presentation creators: Import images of structured graphics into presentation software with full editability of shapes and texts without manual redrawing.
Authors
Pengyu Yan, Yixin Wu, Yunjie Tian, David Doermann
Abstract
Structured images, such as diagrams, charts, and flowcharts, are inherently symbolic and can be compactly represented in an editable format, yet in practice, they are often rendered as images, and therefore not graphically editable. This mismatch presents a significant challenge for researchers, engineers, and designers who wish to incorporate modified versions of existing graphic content into new materials without manually reconstructing it. In this study, we presentBack2Struct, which "makes structured images editable again" by directly recovering vector graphics code (SVG / XML) from image representations. Given an image of a structured graphic, Back2Struct predicts semantically object-level SVG / XML code that explicitly encodes text, shapes, topology, and layout, rather than performing low-level pixel vectorization. The generated code can be seamlessly imported into tools such as PowerPoint, allowing users to edit, refine, restyle, and reuse graphic content while preserving structural fidelity. Beyond supervised fine-tuning on ground-truth SVG token sequences, we further optimize Back2Struct with reward-based learning to better match deployment-time requirements: the output should be syntactically valid, properly concise, and visually faithful to the input diagram. Specifically, we design a composite reward that jointly encourages SVG / XML compilability, length consistency with the reference code, and structural or semantic similarity between the generated and ground-truth graphics. These complementary signals guide the model to produce SVGs that are not only closer to the training distribution, but also more complete, editable, and renderable in practice. Experiments show that Back2Struct improves accuracy, editability, validity, and user alignment over baselines. Dataset and code are available at: pengyu965.github.io/Back2Struct.github.io