Visual token reduction speeds up image question answering effectively
Adaptive Visual Token Reduction for Accelerated Image Understanding
Computer Vision and Pattern Recognition
Summary
Looking at very detailed images to answer questions takes a lot of computer work. The authors developed a way called ReFIT that smartly reduces the amount of image data the computer needs to focus on by reshaping image areas based on instructions and removing unneeded parts. This method keeps important visual patterns like long lines of text and helps the computer answer questions faster and more accurately. They tested it on several benchmarks and found it works well.
What this means in practice
- •For mobile app developers: Build apps that answer questions about images more quickly by reducing image data in an instruction-aware way.
- •For document digitization teams: Improve processing speed and accuracy of reading images with text that is aligned horizontally or vertically.
Authors
Seyoung Jeong, Jong Pil Yun, Sang Jun Lee
Abstract
Large Vision-Language Models achieve strong VQA performance, but processing high-resolution, information-rich images requires substantial computation, motivating visual token reduction. However, existing methods often prune individual tokens or rely on fixed-size cropping, limiting their ability to preserve spatially structured information such as horizontally or vertically elongated text. To address this limitation, we propose ReFIT, an instruction-guided visual token reduction framework for efficient LVLM inference. ReFIT consists of Relevance-Guided Window Reshaping (RWR) and Instruction-Guided Token Refinement (ITR), where RWR captures instruction-relevant regions by adapting to their spatial characteristics, while ITR further removes unnecessary visual tokens. Experiments on four VQA benchmarks demonstrate that ReFIT improves answer accuracy while reducing computational cost, and qualitative results demonstrate its effectiveness in localizing relevant regions and removing unnecessary visual information.