Video language models struggle with accurate object counting
CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Counting objects in videos is a hard problem for video-language models, which combine text and video understanding. The authors found that existing benchmarks mostly test counting of small numbers of objects, so they created a new dataset with videos showing from 5 to 50 objects. Testing many models showed none could consistently count well, and each model preferred certain numbers, making their accuracy uneven. This means better benchmarks like theirs are needed to truly understand and improve counting in videos.
What this means in practice
- •For video ai developers: Evaluate and improve object counting accuracy of video-language models using a balanced, diagnostic dataset spanning wide count ranges.
- •For visual analytics teams: Detect and reduce counting biases in video analysis tools by testing against a controlled synthetic benchmark with varied object counts.
Authors
Marc Chapus, Carlos Crispim-Junior, Véronique Eglin, Atilla Baskurt
Abstract
Object counting in videos remains a poorly characterized capability of multimodal large language models. Counting questions are included in many existing VideoQA benchmarks, but the ground-truth count distributions in these benchmarks are heavily skewed toward small values. Across the nine benchmarks we survey, the median ground-truth count is at most five, and the three most frequent values together account for between 45% and 93% of the videos. The resulting mean accuracy is then dominated by the most frequent counts; a value-dependent bias at a rare count contributes negligibly to the average and remains invisible. We introduce CountVideoBench, a benchmark of 2,600 synthetic videos covering 26 cardinalities in the interval [5, 50] (100 videos each), grouped into six clusters of consecutive counts that span the range while enabling near-neighbor comparisons. Each video contains a single class of static objects under a controlled camera motion, removing object motion and reducing recognition as confounds. We evaluate sixteen video-language models from nine architecture families in a zero-shot setting. No model exceeds 27.6% Exact Match. The dependence of accuracy on cardinality varies across models, ranging from monotonic decline to a non-monotonic peak-and-collapse pattern, which rules out increasing difficulty with cardinality as the dominant explanation of counting failure. Every model further concentrates its predictions on its own set of preferred integers, with accuracy peaks coinciding with these values and collapsing at adjacent counts. These findings highlight the need for balanced, diagnostic counting benchmarks to reliably assess this fundamental visual capability.