Hybrid model improves recognition of Indian sign language 12 words
A Hybrid Kinematic GCN and Elastic Transformer for Efficient Indian Sign Language Recognition
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Recognizing Indian Sign Language is hard because signs involve both hands and vary in speed. The authors created a new method that smartly picks the most important video frames and uses special techniques to keep track of timing and hand movements. They built a dataset of many ISL signs to test their method. Their system recognizes signs with nearly 90% accuracy, showing promise for better sign language technology.
What this means in practice
- •For sign language app developers: Develop ISL recognition features that handle variable sign speeds and both-hand gestures with improved accuracy and efficiency.$Commercial implications: This enables building consumer apps that translate Indian Sign Language in real time for deaf or hard-of-hearing users.
- •For video analysis software teams: Enhance video-based gesture recognition tools to better sample and interpret irregularly timed hand and body movements.
Authors
Pushp Garg, Himangshu Sarma, Mrinmoy Ghorai
Abstract
Sign Language Recognition (SLR) presents complex challenges due to variable signing speeds and spatial intricacies. Specifically, Indian Sign Language (ISL) requires precise modeling of its inherently bimanual nature and is bottlenecked by the scarcity of large-vocabulary benchmarks and the inability of standard models to handle irregular signing velocities. To address this, a structurally-aware hybrid framework for efficient, gloss ISL recognition is proposed. To minimize computational redundancy, an entropy-driven SVD sampler that dynamically extracts informative keyframes is used. Because this subsampling creates irregular physical time gaps, the proposed architecture employs a Continuous Dual Positional Encoding paired with a distance-aware Elastic Transformer to preserve natural signing rhythm. Spatially, a Kinematically-Conditioned Adaptive GCN explicitly models bimanual geometry using unified bone-joint features. To validate our framework, a 120-class, 9,453-video native ISL dataset is created. Augmented by Physical Node Dropout and ArcFace optimization to mitigate subject-specific memorization, our framework achieves a competitive 89.8% accuracy, establishing a strong baseline for complex sign language modeling.