GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of recognizing new human actions from skeleton data without prior examples by connecting skeleton patterns to text descriptions. They note existing methods miss physical and geometric details, creating a gap between meaning and movement. To fix this, they use a pre-trained Text-to-Motion model to generate realistic motion features and combine them carefully with text features, improving recognition accuracy. Their method also refines class prototypes by identifying reliable unseen examples, leading to better performance on standard benchmarks.
Zero-shot learningSkeleton-based action recognitionText-to-Motion modelsGenerative priorsFeature fusionPrototype refinementKinematic featuresNTU-60 datasetGeneralized zero-shot learningSemantic-kinematic gap
Authors
Jidong Kuang, Hongsong Wang, Jie Gui
Abstract
Zero-shot skeleton-based action recognition (ZSAR) aims to recognize unseen action categories by aligning skeleton features with textual semantics. However, existing methods rely on text-derived prototypes that inherently lack geometric structure and physical constraints, resulting in a pronounced \textit{semantic-kinematic gap}. To bridge this gap, we propose \textbf{GenPrior}, the first framework to exploit generative priors from pre-trained Text-to-Motion (T2M) models for ZSAR. Specifically, we introduce Dispersion-Gated Feature Fusion, which distills kinematic prototypes and intra-class dispersion from generative motion sequences and employs a learned gating network to adaptively inject reliable structural cues into textual embeddings while suppressing synthetic artifacts. Furthermore, we propose Generative Prototype Refinement, which leverages these generation-enhanced prototypes as anchors to mine high-confidence unseen samples, calibrating class prototypes toward the true distribution and thereby unleashing strong performance gains. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate that GenPrior achieves state-of-the-art performance under both zero-shot and generalized zero-shot settings. Code is available at https://github.com/jidongkuang/GenPrior.