PercepCap: Video Captioner with Structured Spatio-Temporal Perception
2026-07-22 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created PercepCap, a method to improve video captioning by making the model first explain what objects and events it sees before writing a caption. This helps spot errors in understanding the video, since the model shows the objects' locations and timings it detected. They train PercepCap in two steps: first teaching it to describe perceptions, then refining it with feedback that rewards both good perception and caption quality. They also built special data linking captions to specific video parts so the model learns clear connections between text and video. Their approach outperforms a strong baseline in quality of captions and related question answering.
Video CaptioningSpatio-temporal UnderstandingMultimodal Large Language Models (MLLMs)Perception TraceReinforcement LearningFine-tuningObject TrajectoriesTemporal EventsCaption-Anchored DataQwen3-VL
Authors
Yifan Xu, Zihao Wang, Zhixiao Wang, Jiaming Zhang, Yichun Yang, Desen Meng, Yuanxing Zhang, Pengfei Wan, Limin Wang
Abstract
Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.