Objective metrics better predict human judgments for co speech gestures
Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation
Computer Vision and Pattern Recognition
Summary
Evaluating computer-generated gestures that accompany speech is tricky because existing measurements don't always match what people actually perceive. The authors created a new way to judge these gestures by combining various computer-based measurements with feedback from real users. They introduced a new metric that tracks how well the meaning of gestures is kept in generated movements. Their tests show that some new combined metrics better match what humans think, especially in timing and content accuracy. Overall, they emphasize the importance of checking automatic scores against human opinions.
What this means in practice
- •For virtual avatar developers: Measure and improve the naturalness and semantic accuracy of gestures produced by avatars synchronized with speech.
- •For multimedia content creators: Assess and refine gesture animations in video presentations to enhance viewer engagement through better speech-gesture alignment.
Authors
Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak
Abstract
Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify. We present a perceptually grounded and semantics-aware benchmark that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. We first curate a list of 13 objective metrics covering different aspects, including distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness. For the semantic-appropriateness category, we propose a new metric, Semantic Gesture Preservation (SGP), which measures how far semantic gestures in the ground truth are preserved in the generated gestures. For this, we augment the BEAT2 dataset's annotations using a multi-modal LLM. We then conduct a perceptual study where 101 participants score generated gestures among five dimensions, including human-likeness, motion diversity, absence of animation errors, speech timing and content match. We systematically analyze objective metric--subjective score correlations. Unlike Semantic Score (SC), which shows no significant association with the evaluated perceptual dimensions, SGP is selectively aligned with speech-aware human judgments. We construct five target-specific composite metrics aligned with the subjective dimensions. These composites improve perceptual alignment across all five dimensions, with the largest gains for absence of animation errors and content match, indicating that complementary objective signals can better approximate human judgments than individual metrics alone. Overall, our results show that objective metrics require validation against subjective evaluations.