Codec dynamics improve unsupervised localization of actions in video
CoDy: Codec Dynamics as Complementary Priors for Unsupervised Temporal Action Localization
Computer Vision and Pattern Recognition
Summary
Finding where actions start and end in videos without labeled examples is very hard. The authors noticed that data inside video codecs contains clues about motion and changes during actions. They created a method called CoDy that uses this codec data along with traditional video features to better detect actions and their boundaries. Their experiments showed CoDy works better than previous approaches on popular benchmark datasets. This means video compression info can help understand actions without any manual labeling.
What this means in practice
- •For video platform engineers: Enhance automated identification and timing of actions in large-scale unlabeled video collections using codec-based cues.
- •For sports video analysts: Improve detection and clipping of key action moments in sports footage without needing manual annotations by using codec dynamics as complementary signals.
Authors
Tianyu Yan, Qingyi Yang, Qin Yang, Xufei Guo, Yuan Tang, Rui Wang, Yixue Hao, Long Hu
Abstract
Unsupervised temporal action localization (UTAL) aims to identify action categories and localize their temporal boundaries without annotation supervision. Existing methods typically derive supervision for both tasks solely from RGB and optical-flow representations, often resulting in noisy category assignments and inaccurate temporal boundaries. In this work, we reveal an overlooked property of video codecs: group-of-pictures quantization patterns preserve category-discriminative dynamics, while motion-vector and residual variations exhibit pronounced responses around action transitions. Based on this insight, we propose CoDy, a codec dynamics-guided framework that exploits the codec as complementary information to the visual view for UTAL. CoDy integrates a codec-enhanced view with RGB and flow through confidence-aware consensus clustering, producing reliable soft pseudo-labels. It further converts codec dynamics into a class-agnostic boundary prior, which supervises visual boundary learning and guides proposal initialization and iterative refinement. The codec branches are only used during the training stage and introduce no extra codec-side computation at inference. Extensive experiments on THUMOS14 and ActivityNet v1.3 demonstrate that CoDy consistently outperforms existing UTAL methods and achieves state-of-the-art localization performance. These results establish codec signals as effective annotation-free priors for jointly inferring which actions occur and when they occur.