South African sign language dataset created with minimal human effort
Maswao: An automatically collected, prepared, and benchmarked dataset for the South African Sign language
Computation and LanguageArtificial Intelligence
Summary
Many sign languages around the world do not have large datasets to help computers understand them. The authors created a new dataset for South African Sign Language by automatically collecting video from public sources like news broadcasts and converting spoken words into text using speech recognition. They also designed a method to find sentence boundaries and crop the videos to focus on the signed content. They tested the dataset with different machine learning models and found that Transformer-based models work better than others, but there is still room for improvement.
What this means in practice
- •For sign language technology developers: Access a new automatically collected SASL dataset to build and test sign-to-text translation systems with weakly aligned video and text data.
- •For language resource builders: Use the framework for automated collection and preparation of sign language datasets for under-resourced sign languages beyond SASL.
Authors
Thapelo Sindane, Unarine Netshifhefhe, Vukosi Marivate, Abiodun Modupe, Herkulaas Combrink, Ketro Sithole
Abstract
Recent advancements in sign language processing can be attributed to developments in both large-scale dataset collection and algorithmic complexity in areas such as pose estimation techniques, signal processing through Transformer models, and model optimization. However, only a handful of sign languages, such as the American, British, and German, from over 200 globally recognized sign languages, reap the benefits of such advancements, while the rest lag behind. In this study, we aim to address this technological disparity by proposing a framework for the automatic collection and preparation of sign language datasets, with very minimal human-labour intervention, using the South African Sign Language (SASL) as a testbed. Specifically, we automatically collect SASL video content using public resources such as news broadcasts. Additionally, we convert their accompanying audio streams into text using available Automatic Speech Recognition (ASR) tools. Furthermore, we crop the sign language area in the videos and use gradient descent to locate valleys of non-motion to determine end-of-sentence points within long videos to create our weakly sentence-aligned parallel corpora -- the Maswao (Signs) dataset. Finally, we benchmark the weakly aligned dataset on the task of Sign-to-text Translation using Recurrent Neural Networks and Transformer models to compare their individual intricacies. Our results show that Transformer models proposed in the literature outperform recurrent baselines, with the best-performing Transformer achieving BLEU-1 to BLEU-4 scores of 13.78, 5.21, 2.52, and 1.54, respectively, as well as ChrF = 18.09 and ROUGE-L = 10.51. These results indicate that the task remains promising and warrants further investigations.