Text reading and reasoning remain strong despite poor image quality
Beyond OCR Accuracy: Text-Centric VQA Under Image Degradation with Modular and End-to-End
Computer Vision and Pattern Recognition
Summary
Reading text in images and answering questions about it is harder when the images are blurry, compressed, or low-resolution. The authors compared two ways to do this: one using step-by-step text recognition and another that tries to do everything together. They found that the step-by-step method works better when images are degraded, especially after some fine-tuning for the specific type of image. Also, usual measures of text recognition errors don't predict how well the system answers questions because understanding the meaning can overcome some reading mistakes.
What this means in practice
- •For mobile app developers: Build apps that extract and interpret text from photos taken in poor lighting or motion conditions to answer user questions reliably.
- •For document digitization teams: Use fine-tuned modular OCR pipelines to improve text extraction accuracy and reasoning on degraded scanned documents.
Authors
Ritali Vatsi, Rachapudi Jagadeesh, Shruti Singh Baghel, Himani Sharma, Amit Shukla, Pawan Goyal
Abstract
Text-centric Visual Question Answering (VQA) requires reading and reasoning over text embedded in images, a task made substantially harder when images suffer from real-world degradation such as motion blur, low resolution, or compression artifacts. While modular OCR-based pipelines and end-to-end vision-language models are both widely used for this task, their comparative robustness under degraded conditions remains underexplored. We present an empirical study comparing two modular pipelines with SA-DBNet, a custom detector architecture combining ResNet-18 with self-attention spatial modeling and deformable convolutions against an end-to-end vision-language baseline, evaluated on 4013 degraded images with 7000 question-answer pairs. Fine-tuned modular pipelines achieve up to 57.50% exact-match accuracy versus 38.00% for the end-to-end baseline, with domain-specific fine-tuning yielding a gain of up to 29.50 percentage points. Critically, we find that conventional OCR error metrics like Character Error Rate and Word Error Rate are unreliable predictors of downstream VQA performance, as semantic reasoning can compensate for recognition failures when contextual cues are present. These findings highlight the importance of task-aware evaluation for text-centric VQA systems under realistic visual conditions. Codes are available here