Multimodal reward modeling without human labels improves hallucination scores
Unsupervised Multimodal Reward Modeling
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Training reward models usually needs human judgments telling which responses are better, which is expensive and sometimes inconsistent. The authors show you can train a reward model without any labeled human preferences by making the model score original captions higher than small, automatically created changes of those captions. They pick the best internal model layer to do this, which helps a lot, and their approach beats or ties top systems on several tests while needing much less computing. Their method can even reduce hallucinations in image captioning when retrained with its own verified data, demonstrating a cheaper and human-free way to teach AI to judge outputs.
What this means in practice
- •For image generation teams: Improve the reliability of image captions by training reward models without costly human annotations using automatic caption perturbations.
- •For software engineers in ai services: Reduce computing and data costs for multimodal reward evaluation by using frozen vision-language models and small linear probes trained on unlabeled data.
Authors
Yueying Li, Jingxuan Fan, Kianté Brantley, Sham M. Kakade, Hanlin Zhang
Abstract
Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer. The layer is the decisive choice, worth 5.8 points over the last layer, and one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks at 32B, 6.6 points ahead on VL-RewardBench, from a few thousand parameters at about a thousandth of fine-tuning compute. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. Because the backbone is frozen, the objective is convex, and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6.4 points in hallucination judging. Retraining the head on the policy's own verifier-labeled candidates cuts caption hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark. Together, an intermediate-layer probe, free same-image perturbations, and a verifier-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank.