Micro action recognition improves using soft fusion and reranking techniques
SoftRerank: Hierarchical Soft Fusion with Candidate-Label Reranking for Long-Tailed Micro-Action Recognition
Computer Vision and Pattern RecognitionMultimedia
Summary
Recognizing small and subtle human actions, like slight facial movements or gestures, is hard because they are quick and look similar across different categories. The researchers developed a new method that improves recognition accuracy by fine-tuning a video model and using a two-step system that first guesses broad categories and then narrows down to specific actions. They also use a smart way to balance data from common and rare action types during training. Their approach helps the system better tell apart tricky, similar actions and sets a new top performance in a recent competition.
micro-actionsfine-grained recognitionlong-tailed distributionvideo classificationInternVideo2.5class-balanced samplinginverse-frequency reweightinghierarchical soft fusioncandidate-label reranking
Authors
Yichi Zhang, Zhichao Xia, Yanjun Chi, Lingsi Zhu, Yuefeng Zou, Jun Yu, Qingsong Liu, Jianqing Sun, Shengping Liu
Abstract
Micro-actions are subtle, low-intensity non-verbal behaviors that provide cues to fine-grained human states, including emotions and intentions. Recognizing them remains difficult because they are brief, contain weak visual changes, and often exhibit similar motion patterns across categories. This paper addresses these challenges with a fine-grained micro-action recognition method that combines full fine-tuning of InternVideo2.5, hierarchical soft fusion, and a lightweight candidate-label reranker. For the long-tailed label distribution in MA-52, we use class-balanced sampling and inverse-frequency reweighting to reduce the effect of frequent classes during training. We fine-tune InternVideo2.5 end to end and attach coarse and group-conditional fine-grained classification heads to the shared video representation, improving the consistency between coarse and fine predictions. For ambiguous samples, the candidate-label reranker uses hard samples and video-label matching to focus on easily confused fine-grained actions. Experiments validate the proposed method, which achieves a 79.99% F1-mean on MA-52 and ranks first in the 3rd Micro-Action Analysis Grand Challenge at ACM Multimedia 2026.