Coding agents construct images with precise structure and internal scene understanding
How Far Can Coding Agents Go in Visual Generation?
Computer Vision and Pattern RecognitionMachine Learning
Summary
Modern image generators usually create pictures by tweaking model parameters invisibly, but this paper looks at coding agents that build pictures by writing programs you can run. The authors find these agents are very good at drawing images with accurate shapes and clear layouts, even if the pictures aren't very realistic. They also show that these coding programs keep enough visual information to answer questions about the image better than looking at the pictures directly. Finally, the agents seem to have an internal way of imagining scenes that stays consistent even without showing images yet.
What this means in practice
- •For graphic designers: Generate images with exact structures and visual constraints using programmatic drawing methods.
- •For computer vision engineers: Use reconstructed visual code representations to improve accuracy in answering questions about images.
Authors
Hengguang Zhou, Yuanhao Ban, Jiaqi Feng, Xiaohuan Pei, Justin Cui, Cho-Jui Hsieh
Abstract
Modern image generation largely relies on specialized generative models that leave the process of image construction implicit in learned parameters. We explore an alternative: general-purpose coding agents that construct images through executable programs. Our study of zero-shot drawing on AgentDrawBench, a 200-prompt benchmark, reveals a distinctive capability profile. Agents excel at preserving structure and satisfying precise visual constraints, achieving competitive results in domains where compositional accuracy matters more than photorealistic appearance, while remaining limited in perceptual richness. Their visual expressiveness varies with the drawing interface, and a qualitative study on reconstruction shows how procedural skills extracted from human demonstrations support different artistic media. We then examine code as an external visual language of thought: a structured, executable representation that agents can construct and inspect. We test it in visual question answering task and find reconstructed visual code retains substantial information for question answering and, in some domains, supports higher accuracy than the original image. These abilities raise a further question: do agents draw on a shared internal representation of scene content beyond a particular drawing program? We explore an internal visual language-of-thought hypothesis, in which a shared, compositionally structured representation supports visual operations across drawing languages. Cross-harness drawings and images assembled from independent single-pixel queries exhibit related layouts and tones. Agents also maintain self-generated room arrangements across viewpoints after visible drawings are removed. These results demonstrate that the coding agent would maintain a consistent and coherent world inside its imagine even before the harness gets involved and the generation starts.