Large multimodal models improve evaluation of AI generated images
Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment
Computer Vision and Pattern Recognition
Summary
As tools that turn text into images get better, it's harder to judge how good those images really are. Traditional methods miss details like how well the image matches the text or whether it looks artificially made. The authors created a special test called SQUARE-Bench that checks these image evaluators on many detailed aspects, including if the image is responsibly made. They found some large AI models can judge images even better than expert humans, but they still struggle in certain areas. They also showed these models can help improve images step-by-step by giving smart feedback.
What this means in practice
- •For ai product teams: Evaluate and improve text-to-image generation systems with detailed quality and responsibility feedback.
- •For content moderation teams: Use enhanced AI evaluations to detect potentially problematic or misaligned AI-generated images in user content.
Authors
Yu Zhao, Jiarui Wang, Huiyu Duan, Ye Zhao, Jutao Tang, Juntong Wang, Guangtao Zhai, Xiongkuo Min
Abstract
With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture fine-grained alignment and generative artifacts. While large multimodal models (LMMs) are increasingly adopted as evaluators, existing benchmarks typically study semantic understanding, quality perception, and authenticity identification in isolation, while largely neglecting responsibility detection. This leaves a gap in unified and comprehensive validation. To bridge this gap, we introduce SQUARE-Bench, a comprehensive benchmark that systematically evaluates LMM capabilities as evaluators of AI-generated images across four aspects: Semantics, Quality, Authenticity, and Responsibility. SQUARE-Bench introduces a granular taxonomy of 38 sub-dimensions to evaluate nearly 10K AI-generated images sampled from 22 diverse models, ranging from legacy to state-of-the-art generators, complemented by over 3K real-world images. The images are annotated with curated question-answering pairs. Extensive experiments on 23 LMMs reveal that top proprietary models, such as Gemini-3-Pro, already outperform the individual human expert baseline. However, the performance gap between models remains significant, exhibiting notable disparities in fine-grained inference and domain-specific robustness. Beyond benchmarking, we conduct a proof-of-concept study of LMM-guided iterative editing, in which dimension-specific LMMs provide diagnostic feedback to fixed image editors. The resulting guided system yields selective improvements in semantics, authenticity, and responsibility, while exhibiting a consistent visual-quality trade-off. SQUARE-Bench can serve as both a diagnostic tool for characterizing LMM evaluator capabilities and studying their use in T2I generation refinement. The benchmark and dataset will be released upon publication.