Efficient temporal action detection using native sparse convolutions
SV-TAD: Native Sparse Convs for Efficient Temporal Action Detection
Computer Vision and Pattern Recognition
Summary
Videos often contain lots of repetitive or unneeded parts, making it slow to analyze them for important actions. The authors created a new method that allows computer programs to skip these less useful parts while still efficiently looking for actions. They designed a way for certain network parts, called convolutional adapters, to work directly on the important parts without slowing down, speeding up the process significantly. Their method speeds up video action detection without losing accuracy.
What this means in practice
- •For video analysis teams: Improve speed and reduce cost of detecting actions in long videos by integrating native sparse convolutions with Vision Transformers.
- •For automated surveillance developers: Develop faster real-time systems for recognizing activities by efficiently handling large video streams with token pruning and sparse convolutions.
Authors
Ricardo Pizarro, Roberto Valle, José M. Buenaposada, Luis M. Bergasa, Luis Baumela
Abstract
To adapt billion-parameter Vision Transformers for long-video understanding, recent methods freeze the backbone and train lightweight convolutional modules. While effective for parameter-efficient training, existing adapters do not reduce inference-time computation, leaving scalability with respect to video length largely unaddressed. Token selection can reduce attention cost by pruning redundant tokens, but it breaks the spatial grid structure required by convolutional adapters. This forces an expensive dense reconstruction, nullifying much of the potential speedup. We address this by introducing native sparse 2D convolutions, a primitive that allows these adapters, for the first time, to operate directly and efficiently on dynamically pruned token sets. We integrate this primitive into SV-TAD, an adapter framework for temporal action detection, reducing VideoMAEv2-L computation by up to 64% and achieving 2.2x faster inference, while maintaining state-of-the-art accuracy on THUMOS-14 and ActivityNet-1.3. When scaled to InternVideoNext-L, our approach surpasses the previous state of the art at roughly half its computational cost. Moreover, the sparse formulation naturally supports auxiliary task tokens, which improves fine-grained assembly detection on ATTACH.