Pose refined model improves zero shot recognition of social bodily behaviors
PRIMA: Pose Refined Inductive Motion Adapter for Zero Shot Social Bodily Behavior Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Recognizing small body movements like grooming or nodding helps understand social interactions, but these behaviors change depending on the situation. The authors created PRIMA, a lightweight module that guides existing vision-language models using human pose information during training, without needing pose input later. PRIMA helps models better understand body shapes and motions, improving their ability to recognize different social behaviors without retraining for each new category. This works even on new datasets and across different model types, showing better predictions that match human impressions of social signals like rapport.
What this means in practice
- •For social robotics developers: Enable robots to recognize subtle human social gestures without retraining for every new behavior set by using pose-guided modules.
- •For video surveillance teams: Improve automatic detection of nuanced human activities related to social signals in indoor environments using RGB-only cameras.
Authors
Mingcong Ding, Babette Bühler, Tim Fütterer, Ulrich Trautwein, Enkelejda Kasneci
Abstract
Recognizing fine-grained bodily behaviors, such as self-grooming, postural shifts, and backchanneling, is a useful intermediate step for computational social signal analysis. But the behaviors that matter vary across social contexts, and retraining a model for each new taxonomy is expensive. Vision-language models (VLMs) can adapt to new categories through text prompts, yet they lack sensitivity to body structure and often confuse subtle postural distinctions. We propose PRIMA (Pose Refined Inductive Motion Adapter), a lightweight module that injects a body-structure prior into frozen VLMs using human pose as a training-time supervisory signal, requiring no pose input at inference. PRIMA separates a model-specific projection head from a shared core that encodes transferable body-configuration knowledge; adapting to a new VLM requires retraining only the head. We train PRIMA on a micro-action dataset and evaluate entirely through zero-shot prompting. The adapter improves recognition on its training domain, transfers to additional VLM backbones at different scales with the core frozen, and generalizes to two unseen target datasets from social psychology and daily activity recognition. In a supporting downstream study, the improved behavior predictions also yield higher correlations with human ratings of rapport and dominance in a collaboration context. Within the scope of the tested models and indoor datasets, the results indicate that pose-guided training can produce body-structure priors that generalize across unseen behavior vocabularies and can be partially reused across VLM backbones while preserving RGB-only inference.