Vision language models judge images better using rubric dimensions
Feed the Panel Dimensions, Not Verdicts: Rubric-Decomposed Fusion of Vision-Language Aesthetic Judges
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
Judging how good or beautiful an image looks is tricky for AI models. The authors found that combining overall image scores from several models doesn’t improve results much. Instead, having each model score images on specific qualities like color or composition, based on a human-designed checklist, and then combining those scores leads to better judgments. This approach works better across different sets of images than just using whole-image scores.
What this means in practice
- •For image content platforms: Improve automated aesthetic quality assessments by combining model scores on image qualities rather than overall judgments.
- •For creative software developers: Enhance photo editing tools with more reliable AI evaluations of specific image attributes using multi-model rubric-based fusion.
Authors
Amit Jadhav, Shaurya Beriwala, Beomjin Kim
Abstract
Vision-language models (VLMs) are deployed as zero-shot judges of image aesthetics, and panels of several models are recommended, on thin evidence, as the way to make such judges reliable. On two human-rated datasets, EVA and PARA, we find that a panel of holistic judges never significantly beats its best member, whether the verdicts are averaged or fused by a learned combiner. What a panel is worth depends on what it is fed. We therefore have each model score each image on the five dimensions of a frozen, human-written rubric and fuse those scores, alongside each model's verdict, across model families with an out-of-fold combiner. The dimension scores measure what their labels claim: with the overall human score partialled out, a dimension prompt carries more attribute-specific information than the holistic prompt in 28 of 30 model-attribute cells. Fused, they beat the best single VLM in all ten three-family panels on EVA (against that best single model, +0.07 Spearman rho for the strongest trio and +0.10 for the pre-declared one, and +0.06 and +0.07 when averaged over twenty fold partitions; against the panel mean, the primary test gives +0.118 on its EVA design set), and on PARA they reach parity under Spearman rho and a small, non-significant loss under Kendall tau-b, where one model already captures 85% of the human noise ceiling. It is not a feature-count artefact: giving the same combiner an equal number of pure holistic columns, split from the same repetitions, does not reproduce it. The gain costs a few hundred labels, which do not transfer between datasets, and 4.8x the API calls on EVA; we report it with paired bootstraps and Kendall tau-b, alongside a failed pre-registration and the configurations that lost.