Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models
2026-08-31 • Artificial Intelligence
Artificial IntelligenceComputer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how language models that only use text and code create pictures by writing programs. They made a test called Autoregressive Mosaics to see if models really understand 2D layouts or just follow instructions to write code. Their results show that while all models can follow exact instructions well, their ability to design layouts from vague descriptions varies and depends on how the model outputs the picture (code vs. SVG). They also found that models plan their layouts based on prompts but adjust as they generate the image. This means spatial understanding in these models is influenced by both the model itself and the way the image is created, not just by code-writing skills.
large language models2D spatial layoutprogram generationAutoregressive MosaicsSVGcode generationimage compositionmodel promptingactivations probingoutput medium
Authors
Ashwin Nedungadi, Stefan Oehmcke, Stefan Lüdtke
Abstract
Large language models (LLMs) trained only on text and code can sometimes generate programs that draw recognizable images. However, it is unclear whether this reflects an internal representation of 2D spatial layout or simply the ability to translate spatial descriptions into code. We introduce Autoregressive Mosaics (AM-Bench), a benchmark that separates these factors: First, a translation task gives a model a fully specified geometry of a picture in words as a prompt and asks for the code that produces it. Second, a layout task requires the model to compose an image from an underspecified prompt. Across eight open-weight text-and-code-only models, all models reliably translate specified geometry into code, but their open-ended layout performance differs substantially, indicating that these differences are not explained by code-generation ability alone. An output-medium ablation further shows that the interface or medium of expression that the model uses matters: replacing procedural code with raw SVG improves layout scores across all models. Finally, probing model activations shows that a coarse layout plan is present before generation, but reflects only the layout implied by the prompt. During generation, models track the evolving geometric state instead of executing an initially fixed plan. Overall, these results show that 2D spatial performance in text-only LLMs depends on both the model and the output medium, and is not explained by code-generation ability alone.