TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics
2026-06-02 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceComputers and Society
AI summaryⓘ
The authors created TurtleAI, a test with 823 real-world visual programming tasks involving Turtle Graphics to check how well vision-language models (VLMs) can understand and write code that draws patterns. They tested over 20 models, including advanced ones like GPT-5, and found most did poorly, with success less than 30%. To improve, they made a way to generate training data from just a few examples and fine-tuned one model, Qwen2-VL-72B, which then did about 20% better. They also discovered that some models struggle to understand spatial layouts and recreate images accurately, while their fine-tuning helped link the reasoning better to the actual code.
Vision-language modelsVisual programmingTurtle GraphicsSpatial reasoningSynthetic data generationFine-tuningCode synthesisBenchmark datasetGPT-4oQwen2-VL-72B
Authors
Chao Wen, Jacqueline Staub, Adish Singla
Abstract
Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors limit their performance. To bridge this gap, we introduce TurtleAI, a benchmark containing 823 tasks curated based on real-world visual programming tasks in the Turtle Graphics domain. Solving these tasks requires models to perceive geometric patterns, reason about spatial relationships, and synthesize Python code that faithfully reproduces geometric patterns. We evaluate 20+ VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, and find that they struggle significantly, with most achieving success rates below 30%. To address these limitations, we propose a data generation technique that requires only a small set of seed samples. Fine-tuning Qwen2-VL-72B on the resulting synthetic data yields an improvement of about 20% on real-world tasks. Our failure analysis reveals that GPT-4o struggles with spatial reasoning and precise visual replication, whereas fine-tuning primarily improves the alignment between visual reasoning and code implementation.