Vision language models reveal key moments for video action recognition
Meaningful Moments: A Corpus of VLM-Derived Temporal Importance Labels for Video Action Recognition
Computer Vision and Pattern Recognition
Summary
Not all parts of a video are equally helpful for understanding what action is happening. The authors show that big multimodal AI models, which understand both images and language, can guess which short moments in a video are most important for recognizing actions. They created a huge dataset with importance scores for millions of short video segments across many action classes. Using these scores to pick video parts for analysis can improve action recognition accuracy, especially on certain datasets with clear key moments.
What this means in practice
- •For video analytics developers: Select the most informative short video clips for faster and more accurate automated action recognition based on VLM-derived importance labels.
- •For video content moderation teams: Prioritize review of key video segments identified by vision-language models to improve efficiency in spotting important actions.
Authors
Cameron Keith, SouYoung Jin
Abstract
Not every moment in a video is equally useful for recognizing an action. A few decisive moments often carry the evidence: the instant an object moves, the peak of a dive, or one object placed on another. Existing video datasets rarely provide large-scale, action-conditioned labels for which short segments carry that evidence. This work asks whether vision-language models can identify meaningful moments at scale and whether those labels improve action recognition.We introduce Meaningful Moments (MM), a corpus of about 4.58 million VLM-generated per-segment importance scores for nearly 500,000 class-labeled videos spanning Something-Something v2 (SSv2), Kinetics-400 (K400), and Diving-48 across 622 action classes. We freeze the action classifier and vary only temporal sampling. An alpha-parameterized comparison of importance-led selection against an importance-inverted control separates gains from the VLM signal from gains caused by variable-density sampling alone. Across all three datasets, VLM-derived importance is informative: at the hard-cut endpoint, keeping VLM-important segments outperforms keeping VLM-designated filler. Usefulness depends on class structure. On Diving-48, importance-led fast-forwarding preserves full-video recognition while the inverted control collapses by roughly 30 percentage points. On K400, hard-cut selection beats uniform sampling by 1.45 percentage points, rising to 2.05 under threshold selection. On SSv2, the labels carry signal but do not recover full-video performance. Direct scoring matches greedy removal within its noise floor at roughly 25 times fewer oracle calls, and a four-VLM cross-oracle study shows consistent downstream trends. We release the importance labels and evaluation protocol under CC-BY-4.0; SSv2 class strings remain under CC-BY-NC-4.0.