AI summaryⓘ
The authors address challenges in training Vision-Language Models (VLMs) to reason well by using reinforcement learning (RL). They found that applying RL directly can cause problems because the model may ignore images and rely too much on language shortcuts. To fix this, the authors created a warm-up step where the model learns from carefully selected examples that clearly link images and language, forming a dataset called FaithfulQA. This helps the model understand how visual information and language connect before using RL, leading to better and more stable reasoning based on images. Their experiments show this approach improves accuracy and reduces mistakes where the model isn’t grounded in the visuals.
Reinforcement LearningVision-Language ModelsMultimodal ReasoningVisual FaithfulnessFaithfulQA DatasetCausal ConsistencyVisual Question AnsweringSparse RewardsModel Warm-Start
Authors
Peng, Lee, Yin Zhang, Yanglin Zhang, Haonan Wu, Zishan Liu, Ruoxi Zang, Xin Zhu, Jiayin Zheng, Jian Yao, Zefeng Ji, Fei Ma
Abstract
Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.