Vision language models count items with finger pointing but vary widely
Give the Model a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionMachine LearningRobotics
Summary
Vision-language models try to count objects in images like children learning to count, sometimes exactly for small numbers but estimating for larger sets. The authors gave these models simple pointing abilities to mark items, similar to how children use fingers, and studied how well the models count through this process. They found that different models have very different counting abilities, and while pointing doesn’t always improve counting accuracy, it creates a trace that helps understand how models count. This approach also revealed how counting strategies differ across models and how counting challenges appear in dense or complex scenes.
What this means in practice
- •For computer vision developers: Use pointing-based audits to interpret and diagnose how vision-language models count objects in images.
- •For human-computer interaction designers: Design user interfaces that employ pointing or marking to facilitate step-by-step image understanding by AI assistants.
Authors
Yingxin Jia, Hui Geng, Wenxin Yang, Mohan Zhang
Abstract
Vision-language models (VLMs) are exact on small sets and drift into estimation on larger ones, a profile reminiscent of children before they master counting. Children cross the barrier with epistemic actions: pointing at each item while reciting number words, moving counted items aside, partitioning the array. We give VLMs the same actions as dumb image tools with no visual intelligence of their own in a training-free agent loop. Our central claim is about measurement, not intervention: a model that counts by marking leaves a normally hidden serial procedure on the image, and the developmental principles of counting turn those marks into an interpretable audit of serial visual reasoning: every tag is classified against point-level ground truth as on-target, duplicate, phantom, or omitted, the answer is checked against the number of tags, and order irrelevance is tested by forcing opposite traversal orders. Across 9 models, 682 images, and 7 harness conditions we find that (i) the exact-counting range is a model-specific signature spanning nearly two orders of magnitude (R80 from 3 to 150 dots); (ii) as an intervention the finger mostly fails: tags help 4 of 9 models on small-count photographs, hurt imprecise pointers, and rarely beat a verbal routine or a blind point list; what the loop buys is an auditable trace, not accuracy; (iii) as measurement it succeeds: a tag-perturbation probe shows that precise pointers read their answers off the tags, pointing precision tracks which models benefit, dense scenes fail by omission, and models rarely revise after seeing their own tags; and (iv) the capacities dissociate: models follow an instructed traversal order yet often change their answer when it reverses, and children's regular-over-scattered advantage appears only under the routine and survives lattices that defeat a rows-times-columns shortcut.