Motion language evaluators struggle to recognize structure and mirror actions
Can Motion-Language Models Ground Structure? STRIDE for Evaluating the Evaluators
Computer Vision and Pattern Recognition
Summary
Motion-language models are tools that match actions with descriptions, but the systems used to judge how well they do this often miss important details. The authors created a new test called STRIDE that checks if these judges can tell the right order of movements, recognize when an action is mirrored, and identify the action itself. They found that these evaluators often fail to notice mirrored actions and don't properly understand the structure of the descriptions. They also discovered that many current tests can be passed without truly understanding the action, and showed a way to improve models by training them with harder examples.
What this means in practice
- •For motion recognition developers: Improve motion-language evaluator training by using STRIDE benchmark to better detect temporal and mirrored action errors.
- •For ai system trainers: Enhance contrastive learning methods for language-motion models to increase sensitivity to ordering and reflections using structural hard negatives.
Authors
Lixing Tan, Qing Xia, Yuting Guo, Shuai Li, Aimin Hao
Abstract
Motion-language models are typically scored by motion-language evaluators, but how well these evaluators ground language structure remains unclear. Here, we introduce the Structure grounding via Temporal-order, Reflection, and Identity Diagnostic Evaluation (STRIDE) benchmark to systematically evaluate the ability of evaluators to track temporal order, mirror reflection, and action identity. STRIDE comprises $5{,}869$ triples, each consisting of a motion, its original caption, and a perturbed caption, spanning both short and long descriptions. We likelihood-balance caption pairs to reduce text-only bias and estimate each evaluator's caption preference under unrelated motions to measure the discrimination gain from matched motions relative to this baseline. Our experiments reveal weak structural grounding and severe deficits in mirror sensitivity among the audited evaluators, which commonly used evaluation protocols fail to expose. To understand why these limitations go undetected in standard tests, we examine the evaluators more closely. We find that text-only priors alone can solve naive perturbation tests on existing datasets, while common retrieval and distributional metrics barely respond to structural corruption introduced by mirroring ground-truth motions. These findings suggest a natural intervention: structural hard negatives. Our experiments show that a simple modification to contrastive learning substantially improves performance on temporal order and mirror reflection. The benchmark and code will be released.