Visual counting errors reveal gaps in AI perception and reasoning
DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Counting objects in images sounds simple, but AI models often get it wrong. The authors created DecoyCount, a test using tricky images with fake items designed to confuse AI. This test shows that AI sometimes understands the counting rules if given hints, but gets confused when left on its own. Surprisingly, smarter AI models don't always do better here, and adding extra clues doesn't help much. This means AI still struggles to see and reason about the world in the same way humans do when counting.
What this means in practice
- •For computer vision developers: Improve image understanding systems by testing and reducing vision-reasoning misalignment in object counting tasks.
- •For ai system integrators: Evaluate and refine vision-language AI models' counting reliability in complex, cluttered real-world images before deployment.
Authors
Qiuyi, Zhang, Zhen Xu, Jing Gu, Seojin Bang, Arnav Mohanty Das, Yaning Hu, Gautam Kumar, Jared Lichtarge, Le Xue, Bettina Catoto, Akul Datta, Daniel Luo, Farha Khan, Marcella Valentine, Yinfei Yang, Andrew M. Dai
Abstract
Visual counting benchmarks report whether vision-language models produce the correct number, but not whether errors arise from perception or reasoning. We introduce DecoyCount, a benchmark built from 1,660 real-world images with designed decoys across seven mechanisms, along with paired prompts: a neutral question, a matched alignment cue, and a novel multi-hop, multi-image question that situates the same counting problem in a realistic scenario. Using this design, we find that counting failures are induced partly by visual misalignment: models can execute the intended counting process when cue-guided but fail to select it unaided, and we term this difference in accuracy the reasoning-alignment gap. Despite being highly capable on general reasoning, frontier models remain below 50\% accuracy on our benchmark, while alignment gaps range from 11.8-22.1 percentage points; strikingly, we observe that more capable models, greater thinking budgets, and agentic processes often induce larger gaps. Because humans may bring context that models lack, we test whether textual underspecification contributes to misalignment using a scenario-based multi-hop condition without explicit hinting, but this added context does not improve aggregate performance. Our analysis concludes that stronger general reasoning does not by itself generalize to aligned visual reasoning, leaving misalignment with human visual expectations as a substantial component of the remaining deficit.