Depict measures text image match using answer agreement scoring
DEPICT: Scoring Text-to-Image Alignment by Answer Agreement
Computer Vision and Pattern Recognition
Summary
Matching images to their captions is a tricky but important problem in computer vision. The authors propose a new method called DEPICT that checks if answers about details in the image match those from the caption, rather than relying on fixed correct answers. This approach improves the detection of tricky cases like negations and swapped attributes. DEPICT combines this detailed checking with an overall holistic assessment and outperforms other similar methods without needing extra training.
What this means in practice
- •For caption evaluation teams: Automatically score how well generated image captions match images, improving detection of errors like missing objects or false attributes.
- •For data curation teams: Better filter and curate datasets by identifying image-text pairs with inconsistencies such as incorrect counts or ignored negations.
- •For content moderation teams: Detect hallucinated or misleading content in text-to-image generation outputs by verifying detailed alignment of generated images and descriptions.$Commercial implications: Supports moderation tools that ensure generated visual content matches text precisely, reducing harmful or nonsensical outputs.
Authors
Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins
Abstract
Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations. Recent work addresses this by fine-tuning evaluators on preference data or by prompting a vision-language model, either holistically with the caption or with decomposed verification questions. However, existing approaches fall short: fine-tuned metrics remain bound to one backbone and training distribution; holistic metrics miss fine-grained details; and decomposed metrics rely on a fixed-YES assumption that penalizes faithful images whenever that assumption fails. In contrast, we propose DEPICT, a training-free metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighting questions by how decisively the caption determines them. By replacing fixed references, our agreement rule increases negation accuracy from 19% to 88%. To recover the context lost during decomposition, DEPICT merges this agreement score with a holistic score. We evaluate DEPICT on five benchmarks and eleven backbones from three model families and find that it surpasses all training-free metrics and exceeds fine-tuned evaluators on two out of three human-correlation benchmarks.