PAM-Mamba speeds up dense action detection in long videos
PAM-Mamba - Efficient Dense Action Detection using Mamba
Computer Vision and Pattern Recognition
Summary
Detecting many overlapping actions in long videos is hard and slow for many current AI models. The researchers developed PAM-Mamba, a new AI method that processes video sequences much faster while keeping good accuracy. It uses clever ways to understand time in videos and mixes information from both past and future frames efficiently. It also chooses training examples smartly to improve learning. Their approach reduces the computing needed by over half compared to similar methods.
What this means in practice
- •For video analysis engineers: Implement real-time systems that localize multiple overlapping actions efficiently in extended video footage using PAM-Mamba’s scalable architecture.
- •For sports analytics teams: Analyze long sports videos to detect and time overlapping player actions more quickly and with fewer computational resources.
Authors
Dhruvraj Singh Rawat, Faegheh Sardari, Armin Mustafa
Abstract
Dense action detection requires precisely localising multiple overlapping actions in long untrimmed videos. Transformer-based methods achieve strong accuracy but their $O(T^2)$ self-attention cost limits scalability. We present PAM-Mamba, a linear-time $O(T)$ architecture built on Mamba with a multi-branch temporal encoder. Gated Learnable Positional Encoding (GLPE) controls positional signal injection via a scalar gate initialised to zero, starting training without added positional signal. A Hub-and-Spoke Funnel extends the non-hierarchical multi-scale design of PAT with a direct residual from the original full-resolution input to each coarse branch, providing each coarse branch with a projection of the input features. Bi-TemBa (Bidirectional Temporal Mamba) first extracts local motion cues with a shallow Temporal Convolution Module, then applies a cascaded bidirectional Mamba scan; the backward pass refines the forward-scanned representation from the opposite direction, giving each temporal position access to context from both directions in linear time. Uncertainty-Based Mixup uses prediction entropy to guide the selection of mixup partners. Using the PAT evaluation implementation, we obtain 26.82 (Charades I3D), 34.59 (Charades CLIP), 46.56 (MultiTHUMOS I3D), and 46.92 mAP (MultiTHUMOS CLIP), with 26.1-27.9M parameters, 61-63% fewer than PAT.