Vision language models struggle to align explanations with images
EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Some AI systems that describe images and answer questions can give explanations that sound right but don’t actually match what’s in the picture. The authors created a test called Explanation-Driven Counterfactual Testing (EDCT) to check if the AI’s answers and explanations change correctly when the images are slightly edited. They used this test to find that many vision-language models make inconsistent explanations. Their work also shows these tests can help improve AI by fine-tuning with the edited examples.
What this means in practice
- •For ai system developers: Check and improve the consistency of model explanations with visual evidence when developing vision-language systems.
- •For autonomous vehicle teams: Use counterfactual testing to verify that driving-related AI models provide explanations consistent with visual changes in road scenes.
Authors
Sihao Ding, Santosh Vasa, Aditi Ramadwar, Thomas Monninger
Abstract
Vision-Language Models (VLMs) can produce Natural Language Explanations (NLEs) that sound plausible yet remain inconsistent with the visual evidence they cite. We present Explanation-Driven Counterfactual Testing (EDCT), an intervention-based protocol that extracts visual concepts cited in a model's explanation, applies verified minimal edits to them, and tests whether the resulting answer and explanation remain consistent with the edited image. Using this protocol, we create EDCT-Bench, a comprehensive benchmark spanning three complementary domains: knowledge-intensive visual question answering (OK-VQA), safety-critical driving (DriveLM), and 3D spatial reasoning (3DSRBench). Across the evaluated VLMs, EDCT reveals substantial faithfulness gaps, with models frequently producing responses inconsistent with verified visual changes. Finally, our fine-tuning study suggests that EDCT-generated counterfactuals provide high-impact training signals.