Russian fingerspelling dataset supports new sign language recognition models
Znaki: A Russian Fingerspelling Recognition Dataset
Computer Vision and Pattern Recognition
Summary
Fingerspelling is spelling out words letter by letter using hand signs, vital in sign languages. The authors created Znaki, the first public dataset of Russian Sign Language fingerspelling videos, with thousands of recordings from native signers. This helps researchers build better recognition systems that can understand fingerspelled words in Russian. They also tested two recognition methods to provide a starting point for future work.
What this means in practice
- •For sign language technology developers: Build Russian Sign Language fingerspelling recognition tools using a large, annotated video dataset.
- •For assistive device makers: Develop interpreting devices that understand Russian fingerspelling to improve communication for Deaf users.$Commercial implications: Enables creation of products that translate Russian fingerspelled signs into text or speech for Deaf and hard-of-hearing users.
Authors
Petr Surovtsev, Pavel Korotaev, Alexander Nagaev, Alexander Kapitanov
Abstract
Fingerspelling - the sequential signing of letters that compose proper names, technical terms, and out-of-vocabulary words - is an integral component of any sign language. Despite Russian Sign Language (RSL) being used by more than 150,000 native signers, no public fingerspelling dataset for RSL exists, in contrast to American, British, German, and several other sign languages. We introduce Znaki the first open RSL fingerspelling dataset: 37,252 video recordings (35.4 hours) of 1,593 unique sequences - proper names sampled from 16 domain categories - recorded by 68 native RSL signers (37 Deaf, 29 hard-of-hearing, and 2 certified interpreters) through a crowdsourcing pipeline with a two-stage signer qualification procedure and expert validation. The dataset covers all 33 Cyrillic handshapes, and its letter-frequency distribution matches the Russian National Corpus. As reference points for future research, we establish two baselines adapted from American Sign Language fingerspelling recognition: a Squeezeformer-based encoder-decoder, reaching a character-level accuracy ($1-\mathrm{CER}$) of 0.932 and a sequence accuracy of 0.596, and Fingerspelling-PoseNet, reaching 0.886 and 0.545, respectively.