Multimodal reasoning improves by learning from evidence-ablated negatives
Grounding Multimodal Reasoning with Evidence-Ablated Negatives
Computation and Language
Summary
Multimodal reasoning AI systems often cheat by relying too much on language patterns instead of actually understanding the images they see. The authors present EVAN, a new method that teaches these AI models to avoid this shortcut by creating tricky, altered images with important parts hidden. This forces the model to learn to rely on the real visual evidence rather than just guessing from language clues. Their experiments show that EVAN helps AI better balance information from text and images and improves reasoning performance.
What this means in practice
- •For multimodal ai developers: Improve training of AI systems that reason over images and text by reducing reliance on language shortcuts with evidence-ablated negative samples.
- •For computer vision teams: Develop more robust visual question answering systems by incorporating negative training examples that mask key image regions.
Authors
Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Heng Huang
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become a powerful tool for improving multimodal reasoning. However, standard outcome supervision suffers from credit assignment ambiguity, as a model may receive positive rewards by exploiting language priors or dataset shortcuts rather than grounding its reasoning in the requisite visual evidence. This makes high-quality negative responses critical, since effective negatives should remain plausible under the current policy while exposing the shortcut that should be suppressed. In this paper, we propose EVAN, a multimodal reinforcement learning framework utilizing EVidence-Ablated Negatives to train LMMs against their own language-prior failures. EVAN first identifies highly visually dependent examples using a scaled image information gain metric and preserves data diversity with a mixed-ranking strategy. It then constructs evidence-ablated images by masking the most discriminative image patches, eliciting plausible but visually ungrounded responses from the current policy. During policy optimization, these negative responses are incorporated as bounded auxiliary signals under the clean image, penalizing shortcut learning while maintaining RL stability. Experiments across challenging multimodal reasoning benchmarks demonstrate that EVAN mitigates modality imbalance and improves over strong RLVR baselines. Our code is available at https://github.com/DawnLIU35/EVAN.git.