Reinforcement learning improves image to code with intermediate feedback
Reinforcement Learning from Intermediate Renders for Image-to-Code Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Generating code to recreate images is hard because feedback is usually only given at the end, making it unclear which parts of the code are good or bad. The authors found that partial versions of the code can be run early to see how well they match parts of the picture. They created a method that provides ongoing feedback during code generation by comparing these partial renderings to the target image. This approach helps the model learn better and produces more accurate image-to-code results.
What this means in practice
- •For web developers: Generate precise SVG code from images for scalable graphics in websites more efficiently using intermediate render feedback.
- •For technical document designers: Create accurate TikZ code from figure images to automate drawing complex diagrams in technical documents.
Authors
Omri Kaduri, Kate Feingold, Phillip Isola, Tali Dekel
Abstract
Reinforcement learning is increasingly used to post-train vision-language models for image-to-code generation, such as generating SVG code from a reference image, by optimizing rewards computed from the final rendered output. However, relying on a single terminal reward provides sparse feedback that is poorly aligned with the contribution of individual tokens. A generated program may contain operations that accurately reproduce some parts of the target image alongside others that introduce errors, yet all tokens are trained from the same final outcome. We observe that many intermediate code prefixes are not only executable, but already produce meaningful partial renders that reflect progress toward the target. This property provides a natural source of denser supervision during generation. Based on this observation, we introduce IR4RL, an RL framework with a token-level render-progress reward that turns changes between intermediate renders into localized feedback for the generated sequence. We evaluate our approach on Image-to-SVG and Image-to-TikZ generation. Across both tasks, our method improves over supervised fine-tuning and standard GRPO, yielding new state-of-the-art open-source models. This shows that intermediate rendering provides a simple and effective source of process supervision for RL post-training of image-to-code models.