EVA-Cap improves video captions with event-focused alignment
EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment
Computer Vision and Pattern RecognitionComputation and Language
Summary
Video captioning tries to describe what happens in a video using text. Current methods often write captions without paying close attention to the meaningful parts or events in the video, which can make the descriptions less accurate. The authors created EVA-Cap, a new way to break down captions into smaller event-based pieces and align them closely with the video’s sound and visuals. They trained EVA-Cap on a special dataset that focuses on events and showed that it describes videos more precisely while still doing well on general tasks.
What this means in practice
- •For video streaming services: Generate more accurate and detailed captions by aligning text with specific audiovisual events in video content for better accessibility and search.$Commercial implications: Provides enhanced captioning products for video platforms that improve user experience and compliance with accessibility standards.
- •For media content production teams: Improve automated video summarization by breaking down and aligning captions with event-based segments of audiovisual content during editing workflows.
Authors
Jiapeng Shi, Qiuxia Hou, Jiaqi Leng, Junke Wang, Yanhao Zhang, Haonan Lu, Ziyi Ye, Zuxuan Wu
Abstract
Recent advances in Omni Language Models (OLMs) provide a unified solution for audiovisual video captioning. Nevertheless, existing approaches predominantly treat captioning as an unstructured text generation process, neglecting the inherent semantic structure of videos and leading to suboptimal fine-grained audiovisual alignment. To bridge this gap, we propose EVA-Cap, a novel framework that decomposes captions into atomic audiovisual events and transforms unstructured text supervision into event-centric alignment. We train EVA-Cap via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) on a self-curated dataset, EVA-Data, which is built with event-centric quality control. During the GRPO phase, the model is explicitly optimized against three event-centric alignment targets: global event completeness, intra-event attribution, and inter-event synchronization. Extensive experiments demonstrate that EVA-Cap significantly outperforms existing methods in event-centric evaluation (e.g., +9.0% average gain on ChronusAV), while achieving competitive performance on general audiovisual captioning benchmarks (e.g., +3.3% on WorldSense).