Visual question answering improved with causal reasoning and contrastive learning
Counterfactual Reasoning for Robust Visual Question Answering
Computer Vision and Pattern Recognition
Summary
Visual question answering models sometimes rely on shortcuts in training data, which makes them struggle with new or different data. The authors developed a training method that helps the model focus on truly relevant visual clues by using a special kind of learning called counterfactual contrastive learning. They introduced new techniques to better distinguish important features and align visual evidence with answers. Their method performs well both on standard tests and on tests designed to expose shortcuts, showing improved reliability.
What this means in practice
- •For machine learning engineers: Improve AI models that answer questions about images by reducing errors caused by language shortcuts in training data.
- •For computer vision developers: Create more reliable visual understanding systems by better aligning visual features with question answers using new contrastive training methods.
Authors
Truong-Binh Duong, Thanh-Ngan Tran, Ngoc-Thao Nguyen, Bac Le
Abstract
Modern Visual Question Answering (VQA) models often exploit spurious correlations in training data, leading to poor out-of-distribution (OOD) generalization due to language bias. Although counterfactual learning has shown promise, existing methods can be improved to better guide attention toward causal evidence and strengthen feature discrimination. To address this, we propose a novel training framework that enhances counterfactual contrastive learning for VQA. Our framework introduces three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding. Our model achieves a competitive accuracy of 61.64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62.80% on the standard VQA v2 dataset, yielding a small generalization gap of 1.16%. This demonstrates a strong balance between OOD robustness and in-distribution performance.