Self-Improving Small Object Grounding in LVLMs
2026-06-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors studied how attention patterns inside Large Vision Language Models (LVLMs) can help locate small objects in images without extra training. They found that the model's attention maps contain useful information for predicting how well boxes fit objects. Using this, they created a method called ACS that selects the best boxes, improving object localization. They also developed a simpler version, ACS-Free, which works without any training by focusing on certain attention heads, performing well on benchmarks like COCO.
Large Vision Language Modelsattention mapsobject groundingIoU regressortransformer layersattention entropyACS frameworkCOCO datasetObjects365 datasetsmall object localization
Authors
Tianze Yang, Yucheng Shi, Ruitong Sun, Ninghao Liu, Jin Sun
Abstract
Can internal attention patterns in Large Vision Language Models (LVLMs) identify reliable small-object boxes without fine-tuning? In this work, we provide an affirmative answer. Attention structure in LVLMs encodes grounding quality-a lightweight IoU regressor trained solely on attention maps achieves strong IoU prediction (Pearson r > 0.67). This regressor powers the regressor-based variant of our Attention-based Candidate Selection (ACS) framework, called ACS-Learned, which selects the best box from multiple sampled candidates to improve object grounding. By analyzing what the regressor learns, we reveal which transformer layers and heads are most critical and derive ACS-Free: a training-free selector that ranks candidates by attention entropy on these discriminative heads, with no learned component at inference. Experiments on COCO and Objects365 demonstrate up to 19% self-improvement on small object localization, with ACS-Free ranking best among all training-free methods, demonstrating that useful attention structure improves both localization reliability and interpretability in LVLMs.