Vl-dedup improves deduplication of image and text data in clusters
VL-Dedup: Graded Multimodal Entity Resolution for Image-Text Data Deduplication
Computer Vision and Pattern Recognition
Summary
Large collections of image and caption pairs often contain duplicates or mismatched pairs, such as the same image with different captions or captions reused for different images. The authors created VL-Dedup, a method that groups images visually into clusters, controls cluster size, and uses specialized models to decide which captions to keep or flag. Unlike traditional methods that judge pairs independently, VL-Dedup looks at the entire cluster to make smarter decisions about duplicates and caption quality. They also built a new benchmark dataset to test their approach, showing it reduces information loss while cleaning data.
What this means in practice
- •For image dataset curators: Improve cleaning of large image-caption datasets by grouping similar visuals and selectively keeping diverse, relevant captions with fewer errors.
- •For content moderation teams: Identify and handle image-caption mismatches or duplicates more precisely to maintain data quality in multimedia platforms.
Authors
Zhenxi Liu, Guanze Chen, Dehui Yu, Xingyu Wang, Bingxu Hou, Chengwei Li, Tao Ma, Ke Wang, Gongsheng Yuan
Abstract
Image-text corpora contain copies of one image, captions that describe one image in different ways, captions reused for different images, and captions that do not match their images. Each case calls for a different curation action. Deduplication and multimodal entity resolution (ER) only decide whether records are the same, and they are evaluated on labeled pairs. Yet pair scores do not tell which records end up in the same cluster or which useful captions get deleted. We therefore formulate image-text deduplication as graded multimodal ER, which decides visual identity, caption handling, and mismatches for each pair, and we evaluate it at three levels: pairs, clusters, and the retained set. Our method, VL-Dedup, groups records of the same visual entity into clusters and caps the size of each cluster. Within each cluster, a vision-language model (VLM) and CLIPScore decide which captions to keep and which to flag. We also build VL-DedupBench, with 50,000 COCO and SOP records and 66,788 labeled pairs. On it, a classifier that selects no false edge among the labeled pairs still joins millions of record pairs from different entities. Spending VLM calls on the pairs inside clusters, rather than on pairs with uncertain scores, gives better caption decisions from the same model. Even when almost all clusters are correct, keeping one record per cluster loses facts in many deletions, while the deletions of VL-Dedup lose much less information.