Multi-frame method improves event camera action recognition accuracy
Event-based Action Recognition by Pre-Training on Multi-Frame Representations
Computer Vision and Pattern Recognition
Summary
Recognizing actions from event cameras, which capture fast movements and protect privacy better than regular cameras, is hard, especially when actions last a long time or are complex. The authors developed a new way to combine multiple event frames to better understand these actions. They trained a smart computer model called a Vision Transformer to focus on important parts of the event data, improving its ability to recognize actions. Their approach works particularly well on long or complicated action videos, outperforming previous methods on standard test datasets.
What this means in practice
- •For robotics developers: Improve robot perception by using multi-frame event-based action recognition for better understanding of fast and complex human motions.
- •For security system designers: Enhance surveillance systems by integrating improved event camera recognition methods to detect complex activities over longer durations.
Authors
Emma Norrby, Tomas Wilkinson, Per Nyblom, David Gustafsson
Abstract
Action recognition is a crucial task in domains like robotics, patient monitoring, or surveillance. Recently, event-based action recognition has attracted much attention due to the event camera's ability to capture fast motions and its improved privacy over conventional vision cameras. While frame-based event representations work well for short event sequences, their discriminative ability declines sharply with longer and more complex event sequences. In this work, we introduce a multi-frame representation that greatly improves recognition performance over its single-frame counterpart. We adopt a masked auto-encoding framework to pre-train a Vision Transformer using a sparsity-aware masking algorithm and token selection mechanism to improve downstream performance and control computational complexity. We also introduce multiple mask tokens for the decoder with a position-aware attention mechanism, leading to further improved performance. Our proposed method shows impressive performance for challenging long-duration and complex event sequences, yielding state-of-the-art performance on THU-EACT-50 CHL and DailyDVS-200.