Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models
Machine Learning
Summary
The authors worked on improving speech recognition for Assamese, which is a language hard to process due to limited data. They fine-tuned an existing speech model called Whisper using a specific Assamese dataset. Their new version performed much better than the original model on various error and accuracy measures. They also optimized the training process to run efficiently on limited hardware. Overall, their improvements help make automatic speech recognition more accurate and faster for Assamese.
Authors
Ganapati Das, Dwipen Laskar, Hasin Afzal Ahmed, Sanjib Kr Kalita, Kshirod Sarmah, Hem Chandra Das, Manjula Kalita
Abstract
Developing Automatic Speech Recognition (ASR) for morphologically rich, low-resource languages such as Assamese is challenging due to insufficient annotated speech data. The pretrained Whisper model performs poorly on Assamese speech recognition tasks. This paper presents a controlled, fine-tuned Whisper-based Assamese ASR system trained on the Mozilla Common Voice 24.0-Assamese corpus. A hardware-aware optimized training pipeline is implemented for resource-constrained environments, employing mixed-precision training and gradient accumulation on Tesla 4 Graphics Processing Units (T4 GPUs). The proposed fine-tuned model significantly outperformed the Zero-shot baseline, yielding Word Error Rate (WER), Character Error Rate (CER), Match Error Rate (MER), and Word Infomation Loss (WIL) of 43.17\%, 13.18\%, 43\%, and 64.81\%, respectively, achieving significant relative improvements of 78.26\%, 93.10\%, 57.0\%, and 35.19\% over the baseline. Semantic evaluation of the fine-tuned model also demonstrates notable improvement over a zero baseline, attaining Bilingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit ORdering (METEOR) scores of 30.81 and 0.5262, respectively. Additionally, the predicted hallucination rate and Real-Time Factor (RTF) are substantially improved by 96.70\% and 32.38\%, compared to the zero-shot baseline.