Language aligned model improves gait severity estimates across clinical sites

3rd Place Solution to Human Motion Challenges in Real-World and Clinical Settings (MoCha) @ECCV2026: Language-Aligned Motion Representations for Domain-Generalizable UPDRS-Gait Severity Estimation

Computer Vision and Pattern Recognition

Summary

Measuring how severe a person's gait problems are, especially for diseases like Parkinson's, is important but hard to do well in many different clinics. The authors created a new method that uses computer analysis of motion combined with language descriptions to better understand and compare walking patterns. They trained a model that learns from text descriptions of movements and adapts to different clinic settings, improving its accuracy across sites it hasn't seen before. This approach ranked third in a major challenge focused on real-world clinical gait assessment.

What this means in practice

Authors

Soojie Kim, Muhammad Munsif, Minkyung Kim, Seungryul Baek

Abstract

In this work, we introduce language-aligned motion representations for domain-generalizable UPDRS-Gait severity estimation, aiming to learn semantically structured motion features that generalize across heterogeneous clinical domains. We first learn motion representations using a Bi-GRU backbone that captures the temporal dynamics of SMPL sequences. Prior to model training, motion captions are generated offline using Qwen2.5-7B-Instruct. The backbone is then trained with both classification and text-alignment objectives to learn discriminative and semantically structured motion representations while accounting for the class imbalance present in the training data. We subsequently adapt the learned backbone independently to each source domain so that the model can capture domain-specific motion characteristics. The resulting source-specific models are then merged at the parameter level to consolidate complementary knowledge across source domains into a single domain-generalized model. To further mitigate class imbalance, we perform GPT-5.5-based pseudo labeling, and our final merged models for each site do not use any class-prior correction during inference. The resulting model is evaluated under the unseen-site setting of the MoCha Challenge, using Macro F1 as the primary evaluation metric. Our method achieves a macro-F1 of 0.57 on the hidden test set with only 637K active parameters at inference, ranking 3rd among 58 leaderboard entries in the MoCha 2026 Challenge. The challenge attracted 1,669 submissions from 112 participants and offered monetary prizes sponsored by Machine Medicine Technologies.
View PDFOpen arXiv