Vision transformers focus on important image parts to improve detail recognition
EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception
Computer Vision and Pattern Recognition
Summary
Fine-grained visual tasks need models to tell apart tiny differences in images and explain their answers by pointing to exact details. The authors introduce EviViT, a small add-on that helps existing vision transformers decide where to look more closely in a big image. It uses human eye movement data to guide this focus and connects zoomed-in parts with the overall picture so the model understands details in context. This approach improves accuracy on many fine-grained tasks while using fewer resources.
What this means in practice
- •For computer vision engineers: Improve image classification by focusing model attention on critical regions within high-resolution scenes without losing global context.
- •For robotics system developers: Enhance robot perception modules by enabling efficient detailed examination of scene parts relevant to specific questions or tasks.
Authors
Yaoxin Niu, Zhangquan Chen, Yang Zhang, Xiang An, Zhumei Wang, Chih-Ting Liao, Hongkun Cao, Ruqi Huang
Abstract
Fine-grained visual perception enables vision-language models to distinguish subtle attributes and ground their answers in visual evidence. In high-resolution scenes, processing the whole image at greater resolution spends visual tokens on irrelevant content, while isolated crops can lose the context needed to interpret the selected evidence. We introduce EviViT, a lightweight attachment that learns where a pretrained vision transformer should acquire detail. Human visual-search traces supervise a question-conditioned evidence density, which guides regional re-reading from the original pixels and the allocation of visual tokens. A sparse, coordinate-aware bridge then connects the regional features to the global scene, allowing the host to interpret precise evidence in context. Learned with the host backbone frozen, the attachment serves both the base model and compatible post-trained descendants without refitting. Experiments across nine hosts show consistent gains in average fine-grained accuracy. Matched-budget comparisons further show that EviViT outperforms global-only processing at every tested token ceiling while using fewer visual tokens.