Vision language models reuse reading parts to specialize in OCR tasks
From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists
Computer Vision and Pattern Recognition
Summary
Reading text in images is a tricky task for computers, and it wasn’t clear if general vision language models had to build new special parts to get good at it. The authors found that these models actually reuse certain components made for copying and retrieving text to become experts at OCR, which is reading out full text sequences. They also saw that as the models specialize, they keep the same parts but change how strong or important each part is for reading. This means OCR in such models works like repeatedly copying visual clues and pasting them in the right order.
What this means in practice
- •For application developers: Improve OCR features in apps by integrating vision language models that reuse text-copy mechanisms to read full document images more accurately.
- •For mobile device makers: Embed specialized vision language models in mobile devices to boost text recognition performance without adding new complex modules.$Commercial implications: This paper enables better OCR components for smartphones and tablets by adapting existing vision language models, improving user document scanning features.
Authors
Yuanxiang Huangfu, Hanmeng Zhong, Linqing Chen, Jeffrey Tiong Jee Hui
Abstract
Does a general vision--language model acquire specialized OCR ability by developing a new reading circuit or by reusing an existing mechanism? We address this question in the setting of full-sequence OCR, rather than local-answer retrieval. Using an evidence-grounded protocol with held-out causal interventions, we identify sparse and stable OCR-head sets in GLM-OCR, MinerU2.5, and PaddleOCR-VL-1.6. We then investigate the mechanistic origin of these OCR heads by comparing them with independently identified textual retrieval/copy heads in general VLMs. Across two general VLMs, visual OCR heads strongly overlap independently identified textual retrieval/copy heads, yielding untuned top-20 intersections of 73.3% and all-head Spearman correlations of 0.677-0.886. The overlap and causal interventions suggest that full-sequence OCR operates as dense sequential multimodal copy-and-paste, repeatedly retrieving visual evidence and routing it to the current output position. Finally, we examine how this shared circuit changes as a general VLM becomes an OCR specialist. Matched base-to-specialized comparisons show that OCR specialization largely preserves head identity, retaining 17-20 of the top 20 heads per task with all-head rank correlations of 0.874-0.942, while redistributing their functional and causal strengths.