Visual language models use general heads to read and describe images
Using OCR Heads to Verbalize Image Semantics
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Visual language models (VLMs) turn pictures into words, but how they do this is not fully known. This paper looks closely at how these models recognize words printed in images using special parts called OCR heads. The authors found these heads are actually general tools that help identify meaningful features in images, not just text. By understanding and changing these heads, it's possible to both read words in images and even edit image content in meaningful ways.
What this means in practice
- •For computer vision engineers: Improve image captioning models by targeting specific attention heads that reveal semantic features across images.
- •For graphic design tool developers: Build tools that let users edit objects in images by modifying interpretable latent features alongside text recognition capabilities.$Commercial implications: Supports new image editing products that integrate semantic understanding and object replacement for designers.
Authors
Sheridan Feucht, Benno Krojer, Sarah Wang, Henry Abrahamsen, Byron C. Wallace, David Bau
Abstract
How do VLMs map from pixels to semantics? To understand this general question, we focus on a narrow one: studying how VLMs perform optical character recognition (OCR). Across four models, we identify attention heads causally necessary for OCR, and discover that these are in fact general-purpose heads that output interpretable semantic features across all image tokens. For example, pointing these heads at an image token containing the word "bike" causes Qwen3-VL-8B to output "bike," but pointing them at a bird wing causes the model to output the token "feathers." We collapse these heads' attention weights into a single verbalization lens transformation that reveals interpretable semantic features in hidden states across all layers. When combined with projection to vocabulary space, we can obtain interpretable labels starting from layer 0, showing that image representations are in fact aligned with language in early layers. We find that we can also use the inverse of this transformation to edit non-word concepts, e.g., replacing a tractor with a revolver in a naturalistic image, providing causal evidence that this subspace is useful for more than just OCR. Our results are an example of how the study of specific mechanisms can shed light on broader interpretability problems.