Iterative visual feedback improves plot to code generation quality
Simple Test-Time Refinement for Plot-to-Code Generation via Visual-Code Diagnostics
Computation and LanguageComputer Vision and Pattern Recognition
Summary
Turning pictures of plots into computer code that recreates them is tricky and often has mistakes. The authors introduce a way to catch and fix errors by looking at both the code and the pictures it produces repeatedly, similar to how people debug programs. They train small models to spot four main types of errors based on how the pictures look compared to the code. By using these visual clues to guide fixes step-by-step, their method improves the accuracy of generating code from plot images.
What this means in practice
- •For software developers: Improve automated code generation tools for creating visual plots with fewer errors by refining code based on visual differences identified at runtime.
- •For data visualization engineers: Enhance plot creation pipelines by integrating iterative correction mechanisms that align generated plot code more closely with intended visual output.
Authors
Peiming Guo, Jinpeng Huang, Yangbin Sun, Gaoxiong Cao, Ensheng Shi, Yuchi Ma, Yu Zhang, Meishan Zhang, Baotian Hu, Min Zhang
Abstract
Plot-to-code generation has been greatly advanced by recent multimodal large language models (MLLMs). However, existing studies mainly focus on one-pass generation or iterative sampling guided by verifier-based filtering, which does not align with the typical human programming workflow of generate first, then iteratively error-location and fix. This naturally motivates a test-time refinement paradigm for plot-to-code generation. In this work, we present the first study on plot-to-code generation via iterative test-time refinement. We propose a Visual-Code Diagnostics framework that enables refinement through visual feedback from source code. Specifically, we first categorize generation errors into four coarse-grained types, and then train lightweight discriminators to recognize and localize these errors based on the rendered visual outputs and corresponding code. Unlike traditional compiler- or debugger-based verification, our approach performs iterative refinement guided by one diagnostics system, enabling more effective semantic and visual correction during test time. Extensive experiments demonstrate that our method consistently improves generation quality across diverse generators and significantly outperforms strong test-time scaling baselines.