Visualworldbench tests fine-grained visual knowledge in large language models
VisualWorldBench: A Fine-Grained Multi-Task Benchmark for Evaluating Visual World Knowledge in MLLMs
Computer Vision and Pattern Recognition
Summary
Understanding detailed and specific visual information is challenging for many AI models. The authors created VisualWorldBench, a large and detailed test that checks how well AI models recognize and differentiate between many closely related visual categories. It includes various tasks to see if models really understand the fine details, not just general guesses. They found current top models still make significant mistakes, showing there's room for improvement. They also built a big training dataset that helps models get better at these detailed visual tasks.
What this means in practice
- •For multimodal ai developers: Evaluate and improve AI models’ abilities to understand and distinguish fine-grained visual categories using a broad, challenging benchmark.
- •For image recognition system engineers: Enhance fine-grained object detection and classification by fine-tuning models with a large-scale, diverse training set covering tens of thousands of categories.
Authors
Chen Chen, Zheming Liang, Jiaqi Wang
Abstract
Visual world knowledge is a core capability of multimodal large language models (MLLMs), underpinning knowledge-intensive visual question answering and fine-grained visual reasoning. However, existing benchmarks remain limited in two key respects. First, their category coverage is narrow: most benchmarks span only a small number of categories across a few domains, falling far short of the open-ended scale of real-world visual entities. Second, their task construction is often insufficiently rigorous: models can answer correctly using coarse-grained cues, vague associations, or elimination strategies without truly mastering precise concepts, making it difficult to accurately assess their knowledge of fine-grained entities. We introduce VisualWorldBench, a taxonomy-grounded multi-task benchmark built on a hierarchical taxonomy of 21,291 leaf-node classes across 8 visual domains, with 32,541 evaluation instances whose ground-truth labels cover 15,479 categories across four complementary tasks: closed-set recognition, contrastive selection, precise localization, and open-world naming, covering different directions of knowledge retrieval and varying degrees of answer openness to comprehensively evaluate visual world knowledge. Most questions are constructed from taxonomically proximate categories, forcing precise discrimination among fine-grained entities. Experiments reveal that the strongest closed-source model achieves only 80.4% average accuracy, exposing capability gaps hidden under conventional benchmarks. We further construct VisualWorldBench-50K, a training resource covering 50K+ categories. Fine-tuning on VisualWorldBench-50K yields over 21 percentage points improvement on VisualWorldBench, with consistent transfer to public fine-grained recognition and visual knowledge benchmarks.