Sign language translation improves using pose motion features and T5 models
Investigating Temporal Motion Features for Pose-to-Text Indian Sign Language Translation
Computer Vision and Pattern RecognitionComputation and Language
Summary
Translating Indian Sign Language into English text can be tricky because it relies on understanding body movements. The authors tested how well different sizes of a language model called T5 worked when combined with pose data. They also added extra information about how the poses change over time, called motion features. This addition made the translations better, especially with the smallest model. Their approach ranked 5th in a recent competition and the code is available for others to use.
What this means in practice
- •For speech recognition teams: Build sign language translation tools that convert pose sequences into English text with improved accuracy using motion-augmented models.
- •For mobile app developers: Develop apps that translate Indian Sign Language videos to English by integrating lightweight pose encoders and motion features for better real-time results.$Commercial implications: Enables sale of consumer-facing sign language translation apps with improved accuracy using novel motion feature integration.
Authors
Manav Dhamecha, Praveen Kumar Chandaliya, Pruthwik Mishra
Abstract
We investigate the effect of pretrained T5 model scale and explicit motion features on pose-to-text Indian Sign Language Translation (SLT) for the WSLP 2026 Shared Task. Pose sequences are projected into the embedding space of T5 through a lightweight pose encoder, with the complete model fine-tuned to generate English text. The shared task data used for this work consists of a test set with 5,334 examples and a validation set with 5,257 examples. We compare T5-small, T5-base, and T5-large, and additionally introduce a motion-augmented variant, T5-small + Motion, that adds explicit frame-to-frame pose differences to the input representation. T5-small achieves the best BLEU and ROUGE scores among the spatial-only models, while T5-large obtains the highest chrF score. Augmenting T5-small with motion features yields the largest single improvement observed in our study, substantially improving BLEU over the spatial-only baseline and making it the strongest model overall on this metric. Our submitted system ranked 5th on the official WSLP 2026 SLT testing leaderboard. The source code and trained models are publicly available on GitHub and HuggingFace.