Sign language handshape recognition works across languages with zero training
Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes
Computation and LanguageComputer Vision and Pattern Recognition
Summary
Recognizing handshapes in sign languages usually needs a lot of examples from that specific language. The authors show it’s possible to recognize handshapes in Catalan Sign Language without any training videos from that language by transferring knowledge from American Sign Language. They break down handshapes into smaller features shared across languages and use those to identify handshapes with good accuracy. This approach can help develop sign language technology for many low-resource sign languages that lack annotated data.
What this means in practice
- •For sign language technology developers: Create handshape recognition tools for sign languages that lack annotated video data using transfer from better-resourced languages like ASL.
- •For computer vision engineers: Improve multimodal recognition systems by incorporating phonological features as intermediate representations for cross-domain generalization.
Authors
Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro, Coloma Ballester, Ricardo Marques
Abstract
Sign language processing advances rapidly for high-resource languages such as American Sign Language (ASL), yet most of the world's sign languages lack the phonological annotations new methods require. We present the first zero-shot cross-lingual framework for handshape recognition, transferring from ASL to Catalan Sign Language (LSC). Our approach leverages the decomposition of handshapes into five phonological features -- selected fingers, flexion, spread, thumb position, and thumb contact -- shared across both languages, to decode LSC handshapes from predicted features via a composite phonological distance metric. We evaluate three architectures (MLP, SL-GCN, SHuBERT) trained on two ASL corpora (PopSign, Sem-Lex) against a 37-handshape, single-signer LSC benchmark. Zero-shot transfer proves viable once recording-format disparities are harmonized, reaching 80.0% phonological feature accuracy and 54.5% expected handshape accuracy. Phonological decomposition thus offers a bridge for extending sign language technologies to low-resource languages without any target-language video training labels.