Vision language models can be misled by biased helper text
When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
When AI systems are asked questions about pictures using extra helper text, that helper text can sometimes trick the AI into choosing wrong answers. The authors found that the most harmful helper text is not always clearly wrong, but text that matches the question yet conflicts with the image and pushes the AI toward a wrong answer. They created a method to test how this misleading text affects AI models, and they also proposed a way to reduce these errors without retraining the models. Their work shows that understanding the reliability of helper text requires looking at how it influences final answers, not just whether the text is factually correct.
What this means in practice
- •For ai system engineers: Improve robustness of vision-language question answering by detecting and neutralizing biased auxiliary text that misleads model decisions.
- •For machine learning deployment teams: Deploy inference-time fixes that reduce errors from misleading text without needing model retraining in multi-modal AI systems.
Authors
Tam Le Thi Thanh, Hoang Tran Van, Hong-Hanh Nguyen-Le, Thanh Duc Ngo
Abstract
In tri-modal visual question answering (VQA), auxiliary text is commonly used to complement visual and textual inputs, yet its reliability is often uncontrolled. While prior work studies modality conflicts in general, it remains unclear how different types of unreliable auxiliary text affect answer selection under fixed image-question-option contexts. In this work, we show that the most harmful auxiliary text is not necessarily the most factually incorrect, but the one that aligns with the question while contradicting the image and favoring a specific distractor, leading to systematic redirection of model predictions. To isolate this effect, we introduce the Textual Reliability Ladder, a controlled diagnostic protocol that decomposes auxiliary text along three axes: image consistency, question relevance, and option support. Across multiple datasets (ScienceQA, VCR, A-OKVQA, Causal-VidQA) and recent VLMs, we find that such distractor-supporting text induces the largest accuracy drops (up to 53.1%) and concentrates errors on specific incorrect options. To mitigate this failure mode, we propose a training-free inference-time intervention that explicitly counteracts this redirection effect via noise-stability steering and dynamic grounding, reducing redirected errors while largely preserving performance under faithful text. Our results highlight that auxiliary-text reliability must be understood at the decision level, rather than solely through factual correctness, and provide a practical pathway toward more robust tri-modal reasoning.