Visual grounding improves multimodal in-context learning accuracy
Visual Grounding First, Multimodal In-context Learning Follows
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Multimodal large language models can struggle to correctly understand images when learning from examples, often paying more attention to text and making mistakes. The authors find that the key problem is the model not focusing enough on the important visual parts in example images. They create a method called ProCoRe that trains the model to better match text descriptions to images using contrastive reinforcement learning. This approach improves how well the model learns from both images and text, even without special example datasets. They also provide a new test to check if models truly understand images instead of guessing from previous knowledge.
What this means in practice
- •For multimodal ai developers: Improve accuracy of AI systems that interpret images and text by enhancing visual attention during example-based learning.
- •For ai assistant builders: Build assistants that better understand visual inputs with fewer examples by refining how visual data is processed and grounded.
Authors
Minhyuk Seo, Minjae Lee, Chaeeun Lee, Wei Lin, Muhammad Jehanzeb Mirza, Tinne Tuytelaars, Jonghyun Choi
Abstract
LLMs have shown strong in-context learning (ICL) capability, but extending it to Multimodal LLMs (MLLMs) remains challenging. Prior multimodal ICL methods often rely on ICL-specific datasets for additional training or task vector extraction, which can improve performance on the same ICL benchmark used for adaptation. However, they do not generalize well to other benchmarks and often induce forgetting in previously well-solved tasks. For this reason, we identify 'visual grounding' as the key bottleneck in multimodal ICL; MLLMs often fail to attend to task-relevant visual evidence in demonstrations, instead over-relying on textual cues and producing hallucinated outputs. To address this, we propose ProCoRe to enhance visual grounding through contrastive reinforcement learning on multimodal contrastive data for improving multimodal ICL. ProCoRe generates captions for contrastive image pairs, decomposes them into propositions, and optimizes contrastive rewards that encourage alignment with corresponding images while discouraging mismatched ones. Despite never seeing ICL-formatted training examples, ProCoRe improves average multimodal ICL classification accuracy by 5.6% and captioning ROUGE-L by 13.8% over the off-the-shelf Qwen-VL-3-8B model, outperforming ICL-data-dependent state of the arts across benchmarks. Finally, we introduce PICL, a personalized multimodal ICL captioning benchmark that tests whether MLLMs genuinely understand visual demonstrations rather than relying on pre-trained semantic priors.