Vision language models reduce language bias for better test results
MAGIC: Margin-gated Contrastive Debiasing with Evidence Restoration for VLM Test-time Robustness Scaling
Computer Vision and Pattern RecognitionMachine Learning
Summary
Vision-language models can sometimes rely too much on language patterns, making mistakes when interpreting images and text together. The authors present a method called MAGIC that improves these models by checking how confident they are about different parts of their answers. For uncertain cases, MAGIC adds extra checks by changing text and images a bit to reduce bias and bring back important clues that were lost. This approach helps the models make clearer and less biased decisions when tested on new data.
What this means in practice
- •For ai system engineers: Improve inference reliability in vision-language AI systems by integrating margin-gated contrastive debiasing to reduce language bias at test time.
- •For machine learning platform builders: Develop efficient model inference pipelines that prune ineffective perturbation branches, lowering computational costs while maintaining robustness.
Authors
Shunshun Gu, Enze Zhang, Ming Pei, Hang Zhou, Xiao Luo
Abstract
This paper studies the problem of test-time robustness scaling for large vision-language models (LVLMs), which aims to mitigate language bias during inference. Existing approaches typically adopt test-time scaling techniques such as contrastive decoding to improve inference performance. Despite the progress, the performance of these approaches is far from satisfactory due to intentional evidence removal during perturbations and redundant inference cost. Towards this end, we propose a novel approach named Margin-gated Contrastive Debiasing with Evidence Restoration (MAGIC). The core of our MAGIC is to evaluate the decision margins of LVLMs to selectively activate textual and visual perturbation branches for efficient debiased inference. In particular, our MAGIC first measures the prediction margins of crucial tokens, and applies a gating function to identify low-confidence responses that are susceptible to language biases. For selected samples, we integrate the token logits from multiple textual and image perturbation branches with the original ones for debiased decision. More importantly, we selectively restore the most informative corrupted views with sharp prediction distributions for ambiguous samples after debiasing, enabling evidence recovery for discriminative decision-making. We further utilize a calibration dataset to actively identify and prune ineffective branches during contrastive debiasing for cost reduction. Extensive experiments on benchmark datasets validate the effectiveness of the proposed MAGIC in comparison with extensive baselines.