Multimodal reinforcement learning improves visual target discovery skills
Same Reward, Different Skills: When Multimodal RL Learns to Look
Machine Learning
Summary
This work looks at how reinforcement learning (RL) models understand images when they’re trained mostly without seeing them. The authors found that just having images at test time doesn’t fully teach the model to use visual information. They designed a task that forces the model to actually use the image to find a hidden target, and showed the model got much better at this skill, even on new scenes it didn’t see before. This shows that what a model learns depends a lot on what the reward requires.
What this means in practice
- •For multimodal ai developers: Design training setups that make visual evidence necessary for correct answers to improve models’ ability to ground language in images.
- •For interactive virtual assistant builders: Improve assistants’ ability to accurately interpret and respond to questions about unseen visual scenes by refining reward signals during training.
Authors
Haocun Ye, Xinlong Jiang, Qile Chen, Bingyu Wang, Teng Zhang, Shubai Chen, Tingyu Wu, Zhenkun Zheng, Yiqiang Chen
Abstract
Reinforcement learning with verifiable rewards (RLVR) improves vision-language benchmark scores even without visual information during training. With images at test, blind-trained models recover roughly half of the real-image gain at 3B and nearly four fifths at 7B. Prolonged real-image training can erode grounding while benchmark gains persist. Both findings expose the same gap: an image in the prompt is not an image in the learning signal. Our design rule, visual resolvability, asks that visual evidence be necessary for a correct answer and that the task remain learnable. We test it on counterfactual coordinate scenes in which the question stays fixed and the target is never named, so a correct answer requires finding the target in the image. With standard GRPO and correctness-and-format rewards, a 7B model raises its accuracy at finding the target (discovery) from 0.425 to 0.875 on held-out scenes denser than any it trained on, and it improves on question types it never trained on. Two controls locate the source of the gain. Replacing test images with gray canvases drops discovery to zero; training on gray canvases instead, at matched step 30 and in each of four seeds, yields essentially none of the gain even when the model is then tested with real images. The learned skill carries over to grounding tasks built independently of the training corpus. A caption that answers the training question, added to the same images, reward and budget, cuts the gain by nearly two thirds. Changing what reward requires changes what RL learns.