New multi-agent system scores image quality more reliably on open-ended traits
An Evolutionary Agentic Approach for Open-ended Image Quality Perception
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Assessing the quality of images involves more than just measuring sharpness or fidelity—it includes things like how natural or correct text in images looks. Existing methods struggle when asked to judge new kinds of quality because they rely on fixed ideas and lots of training. The authors developed PACE, a method where multiple AI agents work together to create clear, question-based tests instead of relying on broad impressions. This leads to better alignment with how people judge image quality on many different traits, even ones not seen before.
What this means in practice
- •For image generation developers: Improve evaluation of generated images by reliably scoring new quality aspects like physical realism or text correctness without retraining models.
- •For digital content quality assurance teams: Create adaptable testing protocols for assessing diverse image quality dimensions with minimal human labeling effort.
Authors
Zhenchen Tang, Bo Peng, Zichuan Wang, Songlin Yang, Leilei Cao, Fengjie Zhu, Jing Dong
Abstract
Generative models are rapidly expanding image quality assessment (IQA) beyond traditional fidelity factors to emerging dimensions such as physical plausibility and text-rendering correctness. However, existing IQA models rely on fixed definitions and heavy supervision, making them difficult to extend to open-ended perceptual dimensions. We identify holistic bias as an important limitation: when scoring an unseen dimension, models reuse generic quality priors, leading to scoring errors and rank inversion. To address this, we propose PACE (Perceptual Agentic Collaborative Evolution), a training-free multi-agent framework that formulates open-ended IQA as explicit protocol construction. Given a target dimension, PACE uses collaborative agents to construct an evaluation protocol composed of verifiable Visual Question Answering (VQA) probes, grounding evaluation in concrete visual evidence rather than holistic impressions. The resulting protocol is calibrated using only four human-annotated images per dimension, while a dual-track scoring mechanism aligns model perception with human scoring scales. Across traditional IQA, structural fidelity, context-aware aesthetics, and newly defined open-ended dimensions, PACE consistently improves its MLLM backbone, achieving competitive performance across diverse IQA settings, and reduces the Holistic Override Rate (HOR) from 44.4\% to 8.6\%.