Contrastive modeling improves reasoning in multimodal language models
Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learning
Artificial Intelligence
Summary
Multimodal large language models (MLLMs) can learn from examples to perform tasks involving images and text, but they often just copy example answers without understanding the reasoning behind them. The authors introduce a new method that teaches these models by contrasting bad and better answers alongside explanations of the reasoning process. This helps the models follow the correct thought process instead of just copying surface answers. Their approach also picks the most relevant examples for refining the model’s answers and knows when more improvement is needed, leading to better results especially in visual question answering.
What this means in practice
- •For machine learning engineers: Improve multimodal AI systems by integrating reasoning-aware example selection and refinement to boost visual question answering accuracy.
- •For ai-powered customer support teams: Enhance multimodal chatbots that interpret images and text by guiding models to follow reasoning steps for better response quality.
Authors
Mingbo Yang, Wenqiang Wang, Zhaolu Kang, Peng Chen, Yannan Chen, Sunshang Wang, Yan Xiao
Abstract
In-context learning (ICL) is widely used in multimodal large language models (MLLMs) and achieves strong performance across a wide range of multimodal tasks. However, existing multimodal ICL methods often rely on surface level imitation of in-context demonstrations, making it difficult for MLLMs to align their responses with the reasoning path required by the given multimodal input. This limitation becomes more pronounced in complex multimodal tasks, thereby restricting further improvements in MLLM performance. To address this issue, we propose a new multimodal ICL framework that combines contrastive demonstration modeling with the self-refinement capability of MLLMs. Specifically, our framework reformulates each demonstration by explicitly contrasting a suboptimal response with a better response under the same input, together with a reasoning path that reveals how the response should be refined. This contrastive formulation makes the reasoning path toward the desired response more explicit and guides the MLLM beyond superficial imitation. Furthermore, because effective refinement depends on the current response, we introduce a response-conditioned retrieval mechanism to select demonstrations whose reasoning paths are more relevant to the current response. In addition, we use a lightweight alignment controller to predict response quality and determine whether further refinement is needed. Experiments on three types of multimodal tasks show that the proposed framework consistently improves MLLM performance, with particularly notable gains on visual question answering (VQA).