Multimodal AI judges found biased by irrelevant image editing cues
Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
Computer Vision and Pattern RecognitionMachine Learning
Summary
It can be hard to tell if AI systems that judge image edits are fair because changes to images might affect their quality. The authors created a special test set where they kept the quality of edits the same but added distracting clues. They found that several AI judges were influenced by these unrelated cues, changing their ratings even though the edit was still good. This means these AI judges are not fully reliable because they get biased by things that shouldn’t matter.
What this means in practice
- •For image editing software teams: Improve automatic evaluation tools by identifying biases in AI judges affected by irrelevant visual cues during image editing assessment.
- •For ai quality assurance teams: Develop better testing protocols for AI models assessing image edits to ensure robustness against visual artifacts and ordering biases.
Authors
Yuan Huang, Zirui Song, Xiuying Chen
Abstract
Multimodal large language models (MLLMs) are increasingly used as automated judges for instruction-based image editing and as reward signals for model training. However, systematically auditing whether these judges are influenced by cues irrelevant to editing quality is challenging because visual interventions may themselves alter the quality being evaluated. A judgment shift can therefore be attributed to bias only when the intervention is verified to preserve the underlying editing quality. To address this challenge, we introduce EditJudgeBias, a counterfactual benchmark with verified quality preservation, comprising 1,196 real editing samples and 13 cues injected across four evaluation sites. We verify quality preservation for the requested edit using calibrated multimodal validators, controls, and human inspection. We then audit five MLLM judges along three complementary dimensions: invariance to quality-preserving cues, agreement with human judgments, and stability of pairwise preferences. Importantly, observed shifts are evaluated against each judge's own zero-dose and re-query noise floors rather than against zero. Experiments show that quality-preserving cues move every judge beyond its own noise. Fabricated majority opinions increase ratings, irrelevant visual elements cause larger shifts than whole-image manipulations, and swapping candidate order reverses up to 60.9% of pairwise decisions. Edit-region cues also tend to reduce human agreement. The three measures characterize judges differently, showing that robustness cannot be captured by a single metric.