Vision language models struggle to read overlapping text layers accurately
Sorry Robot, Happy Human: Vision-Language Models Read Only One of Two Legible Typographic Layers
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Vision-language models can read text in images but have trouble when two layers of text overlap. The authors created a dataset with images containing two different texts layered on each other—one with sharp lines and one with soft shading—and tested several AI models on it. While humans could read both layers easily, the models mostly only recognized the sharp-lined text at high resolution. At low resolution, the models could read the shaded text but not the sharp lines. This shows these AI systems have difficulty understanding images with complex text layering.
What this means in practice
- •For image processing developers: Improve text extraction algorithms to handle multiple overlapping text layers by identifying model limitations highlighted in this study.
- •For document digitization teams: Design workflows that consider vision-language models’ inability to accurately read layered text, improving OCR reliability on complex documents.
Authors
Mert İncidelen, Yamen Kashkash, Asya Berker, Murat Aydoğan
Abstract
Vision-language models (VLMs), despite their success in optical character recognition (OCR) tasks, are vulnerable to typographic attacks and have a fragile structure for images with multiple text layers. In this study, the DecoyBench dataset was created using the Decoy Font method. The dataset consists of 300 images, each containing text with sharp contour lines superimposed on another text with soft shading. Six recent closed-source models from three different model families were evaluated using this dataset under two different prompting conditions (naive and guided) and at two different resolutions ($512\times512$ and $64\times64$). A validation study showed that human participants could read both text layers with high accuracy. In contrast, the models, with most variants and both prompting methods, read the contour text with near-human accuracy at high resolution, but almost never fully extracted the shading text. At low resolution, the contour text could not be read by either the models or humans, while the shading text could be extracted with high accuracy. The findings indicate that the evaluated VLMs exhibit a consistent behavioral limitation when processing typographic structures containing multiple spatial frequency layers.