Visionq introduces image quality judging based on specific visual criteria
VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Computer vision papers often compare images, but judges usually pick favorites without clear reasons. The authors created VisionQ, a way to judge images based on specific visual features named by the original paper authors. This allows more meaningful comparisons by matching judges’ choices to declared important qualities. VisionQ also includes a large set of real comparison images, a detailed list of visual criteria, and a trained judging model that improves accuracy.
What this means in practice
- •For computer vision engineers: Evaluate image outputs with judges that score based on explicit visual qualities instead of overall preference, improving model assessment and development.
- •For machine learning platform teams: Incorporate VisionQ benchmarks to test and improve vision-language model judges that review qualitative visual content reliably across criteria.
Authors
Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, Hoang D. Nguyen
Abstract
Qualitative comparison figures are central evidence in computer vision papers, and vision-language models (VLMs) are increasingly used to judge them. Yet existing benchmarks score only scalar quality or overall preference, so a judge can be rewarded for picking the preferred image for the wrong visual reason. We introduce VisionQ, the first benchmark built from peer-reviewed CV comparison figures that grounds every judgment in a named visual criterion: each question states the criterion, and a judge is credited only when it selects the output the authors identify as best on that criterion. We call this task criterion-conditioned visual discrimination. VisionQ comprises (1) a corpus of 1,409 CVPR and ICCV papers with 1,800+ validated comparison figures and 3,911 hand-annotated data points linking method crops to author-stated visual claims; (2) a six-axis, 51-leaf taxonomy of the visual criteria behind qualitative judgment; (3) a criterion-conditioned evaluation protocol that hides method names, captions, and paper identity and reports accuracy per criterion; and (4) VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge trained on symmetric evidence pairs, which reduces last-option predictions by 7.0pp and improves accuracy by 2.5pp on a held-out test set. Evaluating 20 open- and closed-source VLM judges, we find that the strongest reach only 63.1% accuracy (chance 32.2%) and that reliability varies sharply across criteria. Code: https://github.com/ReML-AI/visionq. Data: https://huggingface.co/datasets/visionq-anon-2026/VisionQ-1k.