Video recognition gains speed and accuracy with efficient spatio-temporal interactions
Spatio-Temporal Multi-Order Interaction for Video Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Video recognition systems often need to analyze how things change over time and space, but existing methods can be slow and biased towards certain patterns. The authors introduce a new technique combining efficient time-based processing with balanced spatial and temporal interactions. This approach makes video analysis faster and captures both local and global details more effectively. Their method matches or beats current top systems on popular video action datasets while using less computational power.
What this means in practice
- •For video streaming developers: Improve real-time action detection in video streams by reducing computational costs without sacrificing accuracy.
- •For sports analytics teams: Enhance automated video analysis of player actions using more balanced spatial and temporal modeling techniques.
Authors
Kevin D. Nguetche, Lilian Hollard, Eric Granger, Marco Pedersoli
Abstract
Many temporal approaches for video recognition rely on explicit cross-channel aggregation projections, which have quadratic complexity in the number of channels, introducing significant computational overhead. In addition, the spatio-temporal representation used for video recognition is biased either toward local order interactions (for CNNs) or toward global order interactions (for transformers). Starting from the multi-order spatial representation of MogaNet, our method introduces two complementary contributions to tackle these problems. First, for the temporal modelling, we introduce parallel multi-order processing with depthwise-only convolutions and implicit gating-based aggregation, which is linear in the number of channels, inducing a considerable computational reduction. Second, we propose a dual interaction between spatial and temporal dimensions, which allows a better spatio-temporal representation without increasing the computational cost. By combining these two contributions, our proposed VideoMogaNet achieves performance on par with or better than state-of-the-art action recognition approaches across multiple video benchmarks (UCF101, HMDB51, Kinetics-400, Diving48), while considerably reducing the cost of the temporal branch.