Page images improve document QA accuracy but increase latency and cost
Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA
Artificial Intelligence
Summary
When computers answer questions about documents, they can use either the words extracted from the pages, images of the pages themselves, or both. This paper shows that using page images generally helps computers answer more accurately but takes more time and computing power as documents get longer. Interestingly, some questions are answered correctly only by looking at images or only by looking at text, so using both can be better. The authors also built a simple system that chooses between text or image input based on the question, improving speed and accuracy.
What this means in practice
- •For document ai developers: Optimize document question answering systems by routing queries to text or image inputs for better accuracy and reduced processing time.
- •For enterprise software engineers: Improve internal document search tools by incorporating multimodal input representations selectively to balance accuracy and operational costs.
Authors
Nikhil Reddy Pottanigari, Sepideh Kharaghani, Saverio Vadacchino, Alejandro Posada, Ying Zhang
Abstract
Every document QA system begins with a choice that is rarely studied on its own: whether to feed the model page images, extracted text, or both. We isolate this choice, holding the prompt, judge, and scoring pipeline fixed, across four commercial model endpoints, two corpora, and two context regimes (gold evidence pages and the full document). On documents that fit the image budget, page images lead on accuracy at every document length on both corpora, but this advantage carries a growing latency and cost premium: text latency stays roughly flat as documents lengthen while image latency rises steadily. Text and images also fail on different questions, with exactly one representation correct on 19--25% of items across the reported cells, so neither subsumes the other. Exploiting this complementarity, a lightweight TF-IDF router that reads only the question text gains 2.6 points over always-text while cutting median latency 30% relative to always-vision, on a document-disjoint held-out split.