CORA: Analyzing and bridging thinking-answer gap in Multimodal RLVR via Consistency-Oriented Reasoning Alignment
2026-06-12 • Computation and Language
Computation and Language
AI summaryⓘ
The authors explore a problem in teaching large vision-language models to reason correctly, where the reasoning steps don't always match the final answer. They find that this mismatch persists during training and when the model is used. To fix this, they introduce a method called CORA that rewards the model for making its reasoning and answers consistent. Their experiments show that CORA helps models give better answers with more reliable reasoning explanations. This work focuses on improving how these models think and answer together rather than just improving visual understanding.
reinforcement learninglarge language modelsvision-language modelsreasoning tracessemantic consistencypolicy optimizationreward modelmultimodal reasoningvisual hallucinations
Authors
Jiayue Cao, Zhicong Lu, Xuehan Sun, Wei Jia, Hongling Zheng, Changyuan Tian, Zichuan Lin, Wenqian Lv, Nayu Liu
Abstract
Reinforcement learning with verifiable rewards (RLVR) has successfully elicited the reasoning capabilities of large language models, motivating its extension to multimodal scenarios. Existing methods primarily focus on improving the visual coverage of reasoning traces and mitigating visual hallucinations, but underestimate the semantic inconsistency between the reasoning process and the final answer. In this paper, we delve into thinking-answer inconsistency in RLVR for large vision-language models (LVLMs), showing thorough analyses of rollouts collected throughout Group Relative Policy Optimization (GRPO) training process and post-RLVR evaluation outputs that this issue persists during training and remains present during inference. Motivated by the analysis, we propose Consistency-Oriented Reasoning Alignment (CORA), which introduces thinking-answer semantic consistency into RLVR through a lightweight plug-and-play consistency reward model, and further incorporates Hybrid Reward Advantage Splitting (HRAS) to stably coordinate task and consistency optimization. Extensive experiments across representative multimodal reasoning benchmarks and mainstream LVLMs show that CORA improves task performance while effectively mitigating thinking-answer inconsistency, leading to more faithful reasoning traces.