Whisper tools improve speech transcripts from videos in seven languages
Whisper-Based Speech Transcription from Videos Across Multiple Languages for Cross-Cultural Understanding
Computation and Language
Summary
Understanding people from different cultures often needs tools that can turn spoken words in videos into text in many languages. The authors looked at how well a speech recognition system called Whisper works on videos in seven languages and found it makes quite a few mistakes at first. But by using a little bit of extra training data, the system gets better and creates clearer transcripts. They also share the data and recordings they used so others can try to make the system even better.
What this means in practice
- •For machine learning engineers: Create improved multilingual speech transcription models for video data using modest fine-tuning with public Whisper tools and datasets.
- •For cross-cultural communication tool builders: Develop better automated systems for translating and understanding speech in diverse languages from online videos without deep speech engineering knowledge.
- •For media localization teams: Produce more accurate transcripts from multilingual video content to support subtitles and translations across global video platforms.$Commercial implications: Enables enhanced subtitle services for worldwide video content by reducing transcription errors using fine-tuned automated tools.
Authors
Michael Picheny
Abstract
Cross-cultural understanding has become increasingly important in today's highly connected, cross-national world. The success of LLM-based technologies is now driving the development of automated tools to aid understanding for nonnative people trying to succeed in cross-cultural environments. Building such automated tools is often done by leveraging in-thewild text, audio, and video data. This paper presents techniques for improving speech recognition-based transcript creation in multiple languages from videos to better train these automated tools. The focus is on processes and speech tools that can easily be used by cross-cultural tool builders without requiring deep speech processing expertise. Using publicly available videos from YouTube and Whisper-based tools, average transcription error rate across seven languages (Spanish, Japanese, Korean, Mandarin, Turkish, Russian, and Hebrew) of 30% are observed. With a modest amount of fine-tuning data, the average error rate can be reduced to 20% making such output much more usable for downstream processing. Speech and metadata associated with these videos that can be used by the community to further refine these experiments are released as well.