Models fail in middle and end of long image captions
Where Do Long Captions Fail? Position-Aware Diagnosis and Reinforcement Learning for Detailed Image Captioning
Computer Vision and Pattern Recognition
Summary
When computer models write long descriptions of images, they sometimes make mistakes by adding wrong details in the middle and becoming vague at the end. The authors studied exactly where and how these mistakes happen by breaking down captions into small visual facts. They then created a new training method that rewards the model for being accurate in different parts of the caption, improving its ability to describe images in a detailed and correct way. This helps make long image captions more trustworthy and precise.
What this means in practice
- •For image captioning developers: Improve image captioning systems by reducing inaccurate or vague details in longer generated captions using position-specific training rewards.
- •For content moderation teams: Better detect and control hallucinated or unsupported descriptions in automatically generated long image captions to ensure content reliability.
Authors
Yuanze Hu, Zhichao Yang, Junwei Jing, Xin Yang, Wenxuan Liu, Tinghai Zhang, Jia Fu, Hongzhi Zhang, Xiangyu Wu, Ruiming Tang, Tingting Gao, Han Li
Abstract
Detailed image captioning is usually framed as an amount problem: a model should mention more visual facts while avoiding hallucination. However, autoregressive VLMs generate captions as ordered sequences, and later generations may be produced under weaker or noisier visual grounding. We therefore argue that long-form captioning should be studied not only by what facts are mentioned, but also by where they appear. We introduce a position-aware rubric representation that decomposes dense human references into atomic visual claims, verifies generated captions against those claims, and induces a quartile-wise composition of supported, hallucinated, and generic content. This lens reveals two qualitatively different failures in long captions: models tend to produce unsupported specificity in the middle, while the final portion often escapes into vague, non-evidential language rather than simply accumulating more hallucination. A same-reference human control does not exhibit these patterns; additional human audits examine the visual validity of the induced labels. Motivated by this diagnosis, we propose PAGER (Position-Aware Grounded Evidence Reinforcement), a GRPO training objective with separate signals for supported fine-grained middle evidence, evidence-bearing tail continuation, claim coverage, and content-aware length control. The result is a closed loop: the same ordered claim representation identifies where long captions fail and supplies the position-specific reward needed to repair those failures. Experiments on detailed-caption benchmarks show that PAGER improves position-wise grounding and pairwise caption preference while preserving competitive global caption quality.