Video saliency modeling improves with dynamic content dataset
Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters
Computer Vision and Pattern Recognition
Summary
Predicting where people look in videos is harder than in still images because videos have motion and changes over time. The authors show that many existing video saliency models don’t use motion effectively since they perform similarly to models designed for static images. They created a new dataset called SalTempto with more dynamic video clips to better test how well models capture motion information. On this new dataset, models that consider motion do much better, but there is still plenty of room to improve in predicting human attention in videos.
What this means in practice
- •For video content creators: Develop tools that better predict viewer attention on dynamic video scenes to optimize editing and engagement.
- •For advertising agencies: Enhance video ad placement and design by using attention models tuned on dynamic content to capture real viewer gaze behavior.$Commercial implications: Improves targeting video ads based on better prediction of where viewers look in dynamic scenes.
Authors
Susmit Agrawal, Rebecca Wanner, Juliane Verwiebe, Matthias Tangemann, Matthias Bethge, Matthias Kümmerer
Abstract
Video saliency prediction is inherently harder to model than static image saliency due to the additional temporal dimension. Video saliency benchmarks rest on the premise that predicting gaze on video requires utilizing temporal activity distributed across frames. Prior work has challenged this, showing that static baselines recover a significant fraction of the explainable gaze information on LEDOV, a popular video saliency dataset, and that video saliency models fail in the same places as this static baseline. We verify that this diagnosis still stands: under a more capable gold standard than the original analysis, and an updated panel of recent architectures, the strongest temporal architecture in the panel still does not substantially improve over a fine-tuned static baseline. However, it remains unclear whether the marginal gain reflects limitations of current temporal architectures or a lack of temporal patterns in the benchmark itself. We introduce SalTempto, a video saliency benchmark with greater dynamism: 224 clips of highly dynamic content, sourced from the HACS-Segments dataset so that each clip contains an event together with its lead-up and aftermath, with gaze recordings from up to 16 subjects and a training split for adapting pretrained models. On SalTempto, the static baseline recovers only about 13\% of the headroom above the centerbias, against more than half on LEDOV. A fine-tuned temporal architecture shows a substantial gain in performance over the static baseline, indicating that it does capture meaningfully more temporal information, which LEDOV fails to measure. Yet, even this SoTA model still leaves nearly half of SalTempto's headroom unexplained, indicating room for improvement in video saliency modelling. Examination of SalTempto also lets us describe human tendencies that models miss. SalTempto link: https://huggingface.co/datasets/bethgelab/video_saliency.