Audio visual model improves spotting multiple events in videos
TiTok: Audio-Visual LLM for Multi-Segment Temporal Grounding
Computer Vision and Pattern Recognition
Summary
Finding multiple moments in a video that match a question is hard, especially when sounds matter too. The authors made TiTok, a model that uses both video and sound to find many events accurately. TiTok adds special time markers to better guess event start and end times, and it uses rewards that focus on different aspects like how many events it finds and how precise the timings are. They also created new tests and scores to check how well models count the number of events. TiTok outperforms earlier methods on these tests.
What this means in practice
- •For video content analysis teams: Automatically identify and segment multiple events in long videos using both sound and visuals for better tagging or indexing.
- •For security monitoring teams: Detect and time multiple specific events in surveillance footage leveraging audio and visual cues more precisely.
Authors
Eunji Shin, Dahyun Choi, Seungyeon Jo, Yejin Hong, Jiyoung Lee
Abstract
Audio-visual multi-segment grounding (AV-MSG) in untrimmed videos, reasoning over audio-visual evidence and predicting multiple segments for a query, is a fundamental problem but remains challenging. Visual-only models overlook complementary acoustic cues, while audio-visual models often fail to calibrate the number of events - a phenomenon we refer to as count miscalibration. We present TiTok, an audio-visual large language model (AV-LLM) that localizes an arbitrary number of temporal event segments for each query. For precise boundary prediction, we introduce the Time Token Interleaving (TTI) method, which explicitly injects special time tokens into the audio-visual stream to align input-side temporal perception with output-side temporal prediction. We further propose decoupled, multi-segment-oriented rewards for reinforcement learning, consisting of global, local, count, precision, and format rewards, optimized with Group reward-Decoupled Normalization Policy Optimization (GDPO). To assess the performance on AV-MSG, we establish a new UnAV-100-based evaluation protocol, and propose the CountF1 metric for quantifying count miscalibration that overlap metrics fail to capture. TiTok reaches 65.7 mIoU and 0.58 CountF1, achieving state-of-the-art performance. Our code is available at this link.