Video action detection improves with new temporal and text attention methods
Multi-Scale Latent Temporal Compression Attention and Video-Text Cross Attention Fusion for Video Action Detection
Computer Vision and Pattern RecognitionMultimedia
Summary
Understanding actions in videos requires figuring out how things change over time. Previous computer programs often get confused by too many similar video frames or struggle to combine the overall scene with small details. The authors created a new technique that looks at video frames over different time lengths to keep important information without repetition. They also designed a way for the program to better combine video visuals with written descriptions to understand actions more clearly. Tests show these ideas help computers recognize video actions more accurately.
What this means in practice
- •For video analytics teams: Improve detection of human actions in continuous video streams using enhanced temporal and text-based attention methods.
- •For multimedia content platform developers: Enhance video captioning or tagging systems by better aligning visual content with textual descriptions through fine-grained feature fusion.
Authors
Yicheng Qiu, Keiji Yanai
Abstract
In video action detection, modeling the temporal dependencies in continuous video streams is crucial. Previous methods based on deep neural networks can perform frame-level video recognition tasks. However, these methods struggle to model dynamic temporal patterns, facing challenges such as redundant frame information, difficulty balancing global and local features, and insufficient fusion of visual and textual features. To address these issues, we propose a video temporal attention module based on the Multi-Scale Latent Temporal Compression Attention, which aggregates continuous temporal sequences into feature representations across diverse temporal scales while efficiently reducing redundancy. We also introduce a Video-Text Cross Attention Fusion module that leverages cross-attention at both latent and standard feature levels, achieving fine-grained fusion between video and textual features to enhance semantic alignment and feature interaction. Experimental results across multiple video recognition benchmarks demonstrate that our proposed methods achieve superior performance compared to existing methods, validating the effectiveness and generalization capability.