Vision language models lose accuracy when image layout changes
Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs
Computer Vision and Pattern Recognition
Summary
Vision-language models can answer questions about images well, but small changes in how the image is arranged can cause these models to give wrong answers. The authors found that even when the model reads the correct answer in the image, it sometimes fails to use that information correctly because the way the image is shown has changed. They showed that by helping the model focus on the right parts of the image or text, most of these errors can be fixed. This means changing the image layout affects how well the model uses the information it can actually see.
What this means in practice
- •For ai application developers: Improve visual question answering by controlling image arrangement to reduce answer errors from layout changes.
- •For automated document processing teams: Enhance information extraction accuracy by guiding models with layout cues and transcription corrections when image layouts vary.
Authors
Dingyang Lin, Yingfeng Luo, Chenglong Wang, Chenwei Zhu, Anxiang Ma, Jingbo Zhu, Tong Xiao
Abstract
Vision-language models (VLMs) have achieved strong performance on tasks such as visual question answering, yet small image resizes can turn correct answers into errors. We investigate whether changes in visual configuration, such as image tiling and token arrangement, contribute to this instability. Across seven checkpoints and four benchmarks, equally small resizes cause more correctness flips when they switch configurations. Surprisingly, in over half of these cases, models answer the question incorrectly but can still read the correct answer when told what to read. Furthermore, attention interventions in LLaVA-NeXT suggest that configuration changes can weaken the use of readable information during answering. We therefore guide models using field cues and their own transcriptions. With annotation assistance, these forms of guidance together correct 97.2% of errors with readable information. These findings show that configuration changes can affect how models use information they can still read.