Multimodal model finds misleading captions mismatched with real images
MiD-VLM: Detecting and Grounding Global Semantic Misinformation in Image-Caption Pairs
Computer Vision and Pattern Recognition
Summary
Sometimes an image is real, but a caption next to it changes the story and misleads people. The authors focus on detecting this kind of semantic misinformation where the picture itself isn't altered, only the caption is misleading. They built a computer program that looks at both the image and caption together to decide if they fit or if the caption is false. Their program also finds which words in the caption are misleading. Tests on a special dataset showed their program works very well to spot these mismatches.
What this means in practice
- •For social media platforms: Detect misleading captions paired with authentic images to reduce spread of false narratives.$Commercial implications: It enables content moderation tools to flag semantic misinformation for safer user interactions by combining vision and language analysis.
- •For news verification teams: Identify and highlight misleading phrases in captions that distort genuine visual news content.
Tested on one dataset.
Authors
Tarunya Prasad, Bharatesh Chakravarthi
Abstract
Semantic misinformation can arise when an authentic image is paired with a caption that changes or distorts its narrative. Because the image itself may contain no visible signs of manipulation, methods designed to detect pixel-level artifacts can miss these misleading image-caption pairs. We study this problem using DGM4+, a dataset designed to evaluate the detection of global scene and semantic inconsistencies. We propose a multimodal model that combines image-text representations from SigLIP with visual scene features from DINOv2 through a projection MLP, and adapts Qwen2-VL-7B using LoRA. The model jointly predicts whether an image-caption pair is authentic, identifies the manipulation type, and grounds the misleading caption tokens. On the DGM4+ validation split, the model achieves an F1 score of $0.9679$ for binary authenticity classification. These results suggest that parameter-efficient adaptation of vision-language models is promising for detecting semantic inconsistencies, while also motivating further work on precise token-level grounding.