Vision language models improve image descriptions with fine tuned rewards
Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Large vision-language models sometimes make mistakes by imagining objects in images that aren’t really there, known as hallucinations. The authors found that common training methods can make models avoid these mistakes by simply saying less, which isn’t helpful. To fix this, they created a detailed dataset that clearly marks which objects are and aren’t in each image, giving better feedback during training. They also developed a way to judge parts of a sentence separately so the model can fix mistakes without losing good information. This approach helps the model give more accurate and detailed descriptions of images.
What this means in practice
- •For software engineers: Create image captioning features that produce more accurate and detailed descriptions by improving training with fine-grained rewards.
- •For computer vision teams: Use faithful image descriptions as new context to improve performance in visual recognition and discrimination tasks.
Authors
Xingming Long, Jie Zhang, Yuecong Min, Shiguang Shan, Xilin Chen
Abstract
Object hallucination remains a major challenge for large vision-language models. While off-policy preference optimization proves to be an effective solution, on-policy reinforcement learning provides a more promising direction as it directly targets a model's current failure modes. However, we find that without fine-grained reward formulation and allocation, on-policy optimization often falls into an easy shortcut: reducing hallucinations merely by saying less---making fewer valid claims. To comprehensively resolve this, we propose a fine-grained alignment framework that couples dense reward signals at the data level with precise credit assignment at the algorithmic level. Specifically, we first construct the Dense Object Presence and Absence (DOPA) dataset to address sparse annotations that prevent valid object claims from being verified and rewarded. DOPA exhaustively annotates the deterministic presence and absence of every concept across an expanded vocabulary, significantly increasing the density of reliable reward signals during on-policy rollouts. Second, we propose Subsentence-level Credit Assignment for on-Policy Optimization (SCAPO) to prevent response-level shared advantages from allowing local hallucinations to compromise all other valid outputs within the same response. By assigning credit to each subsentence independently based on its object claims, SCAPO can precisely reinforce faithful generations and penalize hallucinations. Furthermore, we leverage the resulting faithful image descriptions as auxiliary context to transfer generative gains to discriminative tasks. Experiments demonstrate that our method produces highly informative, faithful descriptions in generative tasks while yielding clear performance gains on discriminative evaluation.