Kinematics improve continuous sign language and text alignment accuracy
Kinematics-Centric Continuous Sign Language Retrieval with Gloss-Guided Boundary-Aware Alignment
Computer Vision and Pattern Recognition
Summary
Aligning sign language videos with text is difficult because traditional methods get confused by how people look rather than how they move. The authors introduce a new approach that focuses on the movements of the body using 3D motion models to better match signs with words. They also use hints about when words appear in the video to break the signing into meaningful parts. This method learns from video imagery during training but relies only on body motion for predictions, making the process more precise. Their approach works better than earlier ones on standard tests involving sign language.
What this means in practice
- •For sign language translation developers: Improve phrase-by-phrase matching between continuous sign videos and text to enhance translation accuracy.
- •For human-computer interaction engineers: Use body movement-based models to create interfaces that better understand and respond to sign language input.
Authors
Chang Liu, Ke Han, Davide Talon, Elisa Ricci, Nicu Sebe
Abstract
Sign language-text alignment remains a fundamental challenge for text-driven sign language understanding. Existing methods predominantly rely on appearance-heavy RGB representations, which entangle motion semantics with visual variations and lead to ambiguous motion-language grounding. In this paper, we reformulate sign language-text alignment in a structured kinematic space and propose a kinematics-centric framework that adopts 3D SMPL-X motion as the primary representation. By explicitly modeling the kinematic dynamics of signing in a unified motion space, our approach reduces reliance on appearance signals and yields more semantically consistent representations. To capture the compositional nature of sign language, we introduce a gloss-guided local alignment mechanism that leverages gloss temporal spans as weak supervision to decompose continuous motion into coherent segments and establish fine-grained motion-text correspondences, thereby reducing ambiguity in localizing word-level semantics in continuous signing. Furthermore, we develop a visual distillation strategy, where RGB signals serve as privileged supervision during training to provide complementary contextual cues, while being completely removed at inference time. Extensive experiments on standard benchmarks demonstrate that our method achieves state-of-the-art bidirectional retrieval performance on CSL-Daily and competitive results on PHOENIX-2014T. These results highlight the effectiveness of kinematic representations and explicit local grounding for sign language-text alignment.