Boundary voting network improves action segmentation in videos
Boundary Voting Network for Ambiguity-Aware Timestamp-Supervised Action Segmentation
Computer Vision and Pattern Recognition
Summary
Videos often show activities that need to be divided into clear steps, but marking exactly where one step ends and another begins is tricky when only a few frames are labeled. The authors designed a method called boundary voting network to better locate these boundaries by using clues from the whole video, helping the computer guess more accurately. This new approach helps the model learn from uncertain regions and improves how well actions are segmented. They tested it on several video datasets and found it works better than previous methods.
What this means in practice
- •For video analytics developers: Enhance training of action segmentation models with sparse timestamp labels to generate more accurate video action boundaries.
- •For sports video editors: Assist in automatically segmenting sports videos into distinct actions using minimal manual annotation.
Authors
Runzhong Zhang, Yueqi Duan, Yang Chen, Weipeng Hu, Chen Cai, Suchen Wang, Yap-Peng Tan
Abstract
Timestamp-supervised action segmentation aims to segment and classify actions in untrimmed videos with a random frame annotated per action. Precisely localizing action boundaries from timestamp annotations is crucial for this setting, as it enables generating framewise pseudo-labels and applying the well-explored fully-supervised training. However, prevailing methods struggle with intrinsic uncertainty in boundary localization due to less discriminative features in action-transiting regions. This imprecise boundary estimation significantly reduces the stability and reliability of the generated pseudo-labels in ambiguous action-transiting regions, consequently resulting in performance deterioration of the trained segmentation models. In our paper, we introduce the boundary voting network that mitigates feature ambiguity by hierarchically propagating video-level global prior knowledge into local action-transiting regions. By generating key action representations as votes throughout the video and targeting action-transiting regions, all votes collaboratively contribute to action-transiting feature enhancement and boundary localization refinement. Extensive experiments demonstrate the effectiveness of our method on GTEA, 50Salads, and Breakfast datasets.