Video highlight detection that handles missing visual audio or text streams
TSMD: Temporal-Stream Modality Dropout for Robust Video Highlight Detection
Computer Vision and Pattern Recognition
Summary
This work deals with making video highlight detection more reliable when parts of the video data are missing. Videos often have visual, sound, and text parts, but these can be incomplete or missing entirely. The researchers found that previous methods didn’t handle these problems well and their standard evaluation was also misaligned with how highlights actually appear. To fix this, they created a new method called TSMD that simulates missing data and uses a better way to train the system. This approach improves highlight detection even when half the video frames or entire data streams are missing.
What this means in practice
- •For video streaming services: Improve highlight detection reliability when videos have missing or corrupted audio, visual, or text streams during playback.$Commercial implications: Enables more accurate automatic highlight clips for streaming platforms despite imperfect video data, improving user engagement.
- •For digital media editing teams: Use robust highlight detection for creating summaries from incomplete or partially corrupted multimedia footage to save editing time.
Authors
Bo-Yuan Cheng, Kuan-Yu Chen, Po-Han Huang, Jeng-Lin Li, Jian-Jiun Ding
Abstract
Existing multimodal video highlight detectors typically assume that visual, audio, and textual streams are continuously available. In practice, however, inputs may suffer from localized frame missingness or complete-stream outage. We formulate this robustness challenge along two dimensions: temporal missingness, where frames are missing independently in each modality, and stream-level missingness, where one modality is unavailable throughout a video. Moreover, we find that the mean squared error (MSE) loss is misaligned with both the evaluation metrics and the peak-driven nature of highlights. Therefore, we propose Temporal-Stream Modality Dropout (TSMD), which combines structured missingness simulation with a joint objective comprising pointwise MSE, per-video Pearson correlation, and peak-oriented RankNet loss terms. TSMD has three variants: temporal, stream-level, and mixed dropout. On the MoSu and Mr. HiSum datasets, TSMD-Temporal improves mAP@15 by 7.06 and 3.41 points over TripleSumm under 50% independent temporal removal, whereas TSMD-Stream performs the best under complete-stream removal. TSMD-Mix retains most of these complementary benefits and ranks the best or the second-best across the evaluated temporal and stream-level conditions.