Large language models generate and interpret images pixel by pixel
PixelBench: Can LLMs Understand and Generate Images Pixel by Pixel?
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
Most language models create images using descriptions of shapes and lines, but this paper explores whether they can handle images by controlling each tiny dot or pixel directly. The authors made PixelBench, a way to test if language models can generate or understand images at the pixel level using only their usual text tools. They found that the best models can make simple pictures from text and do better than some standard image processing methods when simplifying images. They also improved a model by training it to reconstruct images pixel by pixel, which helped it perform better on various visual tasks.
What this means in practice
- •For mobile app developers: Create text-driven apps that generate simple images directly from pixel data without relying on complex image processing pipelines.
- •For graphic design teams: Automate the pixel-level simplification of images better than some traditional pixelization tools, aiding low-resolution design workflows.
Authors
Haoquan Zhang, Feiyang Liu, Juanxi Tian, Wanhua Li, Haodong Duan, Weiyang Liu
Abstract
Large language models (LLMs) can generate visual assets through text-based formats such as SVG and CAD, but these symbolic representations describe images using high-level primitives rather than explicit raster values. We ask whether LLMs can instead generate pixel images directly through their existing vocabulary, without a learned image tokenizer or decoder. We introduce PixelBench, a benchmark for text-to-pixel (T2P) and image-to-pixel (I2P) generation across multiple resolutions, together with a complementary pixel-understanding evaluation. Models generate an RGB color palette followed by a grid of indices specifying the palette color of each output pixel. In T2P, the strongest models produce recognizable images directly from text. In I2P, the strongest models outperform palette-matched downsampling and specialized pixelization algorithms at low resolutions. Pixel-understanding performance correlates strongly with generation performance. A resolution-scaling study shows that pixel-generation performance often improves initially before declining at high resolutions. Finally, we introduce Pixel Reconstruction Tuning (PRT), which augments visual instruction tuning with caption-palette-grid trajectories. We post-train Molmo2-4B from its pretraining checkpoint by mixing general instruction data with our curated PRT data, improving performance across 16 visual-centric benchmarks, indicating that explicit pixel reconstruction can provide useful auxiliary supervision.