Improving subtle human action recognition with hierarchical soft fusion
SoftRerank: Hierarchical Soft Fusion with Candidate-Label Reranking for Long-Tailed Micro-Action Recognition
Computer Vision and Pattern RecognitionMultimedia
Summary
Recognizing tiny and brief human actions like subtle gestures or facial expressions is tough because they look very similar and change very little. The authors improve this by fully training a video model and adding new ways to better distinguish similar actions, especially when some actions happen much less often than others. They also use a special method to recheck difficult cases and choose the best label. Their approach performs well on a challenging dataset of micro-actions and won a competition.
What this means in practice
- •For video analytics developers: Enhance video systems to detect subtle human gestures and expressions more accurately, particularly when data has many rare action types.
- •For human-computer interaction engineers: Build interaction models that better interpret brief, low-intensity human signals for improved user experience in responsive systems.
Authors
Yichi Zhang, Zhichao Xia, Yanjun Chi, Lingsi Zhu, Yuefeng Zou, Jun Yu, Qingsong Liu, Jianqing Sun, Shengping Liu
Abstract
Micro-actions are subtle, low-intensity non-verbal behaviors that provide cues to fine-grained human states, including emotions and intentions. Recognizing them remains difficult because they are brief, contain weak visual changes, and often exhibit similar motion patterns across categories. This paper addresses these challenges with a fine-grained micro-action recognition method that combines full fine-tuning of InternVideo2.5, hierarchical soft fusion, and a lightweight candidate-label reranker. For the long-tailed label distribution in MA-52, we use class-balanced sampling and inverse-frequency reweighting to reduce the effect of frequent classes during training. We fine-tune InternVideo2.5 end to end and attach coarse and group-conditional fine-grained classification heads to the shared video representation, improving the consistency between coarse and fine predictions. For ambiguous samples, the candidate-label reranker uses hard samples and video-label matching to focus on easily confused fine-grained actions. Experiments validate the proposed method, which achieves a 79.99% F1-mean on MA-52 and ranks first in the 3rd Micro-Action Analysis Grand Challenge at ACM Multimedia 2026.