Pigeons improves long video captioning by tracking subjects and story
PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary
Computer Vision and Pattern Recognition
Summary
Videos that last a long time are hard to describe with captions because current tools look at only short clips and miss connections. The researchers created PIGEONS, a method that first finds and remembers important people or things in the whole video, then writes captions for each clip while keeping track of these subjects and the overall story so far. This helps the captions stay consistent and tell a smoother story over time. Their tests show PIGEONS works better on long videos and also does well on shorter ones.
What this means in practice
- •For video platform developers: Create automatic captions that keep characters and story consistent across long videos to improve viewer accessibility and engagement.$Commercial implications: Enables better subtitles and video summaries for streaming platforms, enhancing user experience and accessibility for millions of viewers.
- •For media content editors: Generate more coherent and accurate descriptive captions for long news or documentary videos to aid editing and summarization workflows.
Authors
Xinlong Chen, Weihong Lin, Yuanxing Zhang, Xinyu Liu, Yue Ding, Bozhou Li, Bohan Zeng, Yang Shi, Qiang Liu, Pengfei Wan, Liang Wang, Tieniu Tan
Abstract
Detailed video captioning has become an essential semantic interface for multi-modal understanding and generation. While existing methods mainly target short videos, extending them to long videos remains challenging, primarily due to limited context windows of current video understanding models. A naive solution is to adopt segment-wise captioning followed by concatenation, yet this paradigm often suffers from inconsistent subject references and fragmented long-range narratives. To mitigate these issues, we introduce PIGEONS, an incremental long-video captioning model built upon a Persistent Identity Gallery and an Evolving Overarching Narrative Summary. Before caption generation, PIGEONS scans the entire video to construct an identity gallery that captures key subjects along with their visual references, thereby providing a stable referential anchor. The model then performs clip-by-clip captioning conditioned on the persistent gallery and an evolving overarching narrative summary, which compactly maintains and updates essential long-range contextual information after each captioning step. To enable these capabilities, we develop a three-stage training strategy that integrates supervised fine-tuning and reinforcement learning. Extensive experiments demonstrate that PIGEONS markedly improves long-video captioning performance, achieving superior accuracy, completeness, and subject referential consistency over existing methods, while maintaining competitive performance on short-video captioning.