Vector graphics help vision language models think with images better
Multimodal Thinking with Renderable Programs
Computer Vision and Pattern RecognitionComputation and Language
Summary
Vision-language models can understand pictures and text but struggle to use images when solving problems step-by-step. The authors created a new system called SVGLM that uses vector graphics (SVG) to represent images as simple shapes and instructions. This approach helps the model generate and edit images while reasoning, making the whole process easier to understand and more efficient. They tested SVGLM on math problems and showed it can think with images and create graphics better than before.
What this means in practice
- •For software engineers: Build applications that combine image creation and reasoning using interpretable vector graphic instructions within language models.
- •For graphic designers: Develop tools that allow interactive editing and generation of vector images guided by textual commands integrated into AI reasoning workflows.
Authors
Sunli Chen, Ding Zhong, Ziqiao Ma, Jiaxin Liu, Zeyuan Yang, Hao Zhang, Lie Lu, Joyce Chai, Chuang Gan
Abstract
Current vision-language models (VLMs) excel at visual content understanding and text-based reasoning, yet their structure limits the advancement of incorporating images into the reasoning chain. Though Omnimodal models have made efforts in unifying text and image generation, they focus on visual tasks in the open-domain, lacking tractability due to rasterized or latent representations of images. We introduce SVGLM, a framework that uses scalable vector graphics (SVG) primitives to connect text and image in reasoning tasks. We exploit the duality of SVG as both image description and text instructions, yielding a more compact, interpretable solution to equip general VLMs with the capability of generating images within the reasoning process. We provide a large curated dataset of SVG-based image editing dataset, as well as the paradigm to tune open-source VLMs. Experiments on a mathematical reasoning benchmark demonstrate that SVGLM achieves strong SVG generation power as well as think-with-image intelligence. Our results highlight SVG as a suitable medium for building more robust digital domain agents, bridging the gap between text-based thinking and pixel-based images.