NumBench: Diagnosing Counting Failures in Text-to-Image Models
Computer Vision and Pattern RecognitionDatabases
Summary
The authors created NumBench, a large test set of 640,000 prompts to study why AI models often generate images with the wrong number of objects. They designed a new model explaining how objects compete for limited space, leading to errors especially when many objects are requested. They also developed a new scoring method to better measure counting accuracy. Testing various AI systems showed they struggle with counts above 50, and layout and object arrangement affect accuracy. A human study confirmed their automated evaluations are reliable up to 50 objects.
Authors
Sandeep Wadhwa, Mayank Vatsa, Richa Singh, Parrva Chirag Shah, Prakhar Galriya
Abstract
Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We introduce \textbf{NumBench}, a benchmark of 640{,}000 prompts spanning 1{,}600 categories and counts from 1 to 100. Its factorial design varies object composition, spatial guidance, and appearance conditions while balancing counts and category exposure. We also develop a process model in which requested instances compete for a finite set of resolvable image regions. The model predicts a near-quadratic collision deficit at low occupancy and shows how coordinated placement reduces it. For scalable evaluation, we propose the Confidence-Weighted Numeric Precision Score (\cwnps), which aggregates three calibrated detectors and discounts uncertain proposals. Across five commercial systems, two open models, and two specialized counting methods, performance declines sharply with requested count; all evaluated methods are weak above 50 objects. Count range has the largest measured effect, followed by layout and composition. Grid guidance is strongest among guided layouts, consistent with the coordination prediction, although the analysis does not establish collision as the sole cause. A 14{,}400-image human study supports automated evaluation through count 50, while results on 243 natural-language prompts show transfer beyond NumBench templates.