Audio visual captions improved with bidirectional event alignment
AVTime: Reinforcing Dense Audio-Visual Captions with Bidirectional Alignment
Computer Vision and Pattern Recognition
Summary
Understanding videos with sound involves knowing what happens and when it happens. The authors created a large new dataset with detailed video captions tied tightly to specific moments, helping models learn better. They built a model that rewards matching both the timing and meaning of events in videos. They also designed a new way to measure how well captions describe events and their timing. Their work aims to make video descriptions that are more accurate and useful.
What this means in practice
- •For video platform developers: Improve automatic video captioning tools by providing temporally precise and semantically accurate descriptions for longer videos with sound.$Commercial implications: Enables video platforms to sell enhanced captioning features improving accessibility and search.
- •For ai content moderation teams: Use the new dataset and model to better detect and summarize events in multimedia content with accurate timing and meaning.
Authors
Mingyang Wu, Kaituo Feng, Bohao Li, Kaixiong Gong, Zihao Yin, Xiangyu Yue
Abstract
Dense temporal audio-visual captioning is important for multimodal video understanding and generation. However, prior work is constrained by three main limitations: (1) the lack of a dataset combining high temporal resolution, longer videos, and bidirectional alignment between time and events; (2) the reliance of most existing RL methods on checklists; and (3) the lack of a benchmark that bidirectionally evaluates event semantic correctness and temporal accuracy. To address these challenges, we propose: (1) AVTime-50K, a high-quality 50K-scale dataset with temporally precise, semantically accurate captions and longer videos; (2) AVTime, a state-of-the-art model optimized with a novel Bidirectional Temporal-Semantic Alignment (BiTSA) reward that imposes explicit bidirectional constraints between time and events; and (3) AVTime-Bench, a specialized benchmark that introduces a novel bidirectional evaluation of temporal accuracy and semantic correctness. Our code, models, and datasets are available at https://huggingface.co/collections/Apryle/avtime.