Omni large language models struggle with multi speaker long videos
Omnilingual-MsASR: Benchmarking Multimodal, Multilingual, Long-Form Multi-Speaker ASR of Omni Large Language Models
Computation and Language
Summary
Videos with many speakers talking for a long time are hard for current advanced AI models to understand. The authors created a new test called Omnilingual-MsASR to check how well these models recognize different speakers and their words across six languages. They found that even the best models often make big mistakes, especially when figuring out who said what and when. This shows that recognizing multiple voices in long videos and using both sound and visuals together remain big challenges.
What this means in practice
- •For speech recognition engineers: Evaluate ASR models on long multilingual videos with many speakers using a new dataset combining audio and video.
- •For video platform developers: Improve speaker identification and transcript accuracy for long-form multilingual content by testing AI models against a benchmark measuring cross-modal understanding.
Authors
Yuan Ge, Pei Zhang, Xize Cheng, Lingzhou Mu, Mingyang Wang, Jin Xu, Baosong Yang, Shengxiang Gao, Jingbo Zhu, Zhengtao Yu, Tong Xiao
Abstract
Omni-modal large language models (LLMs) have achieved remarkable progress in audio and video understanding. However, existing models struggle with long-form, multi-speaker videos, especially in understanding the entire video and recognizing who says what and when. To characterize the capability limit of current omni LLMs and guide future model development, we introduce Omnilingual-MsASR, a multilingual, multimodal benchmark for long-form Multi-speaker Automatic Speech Recognition (MsASR) covering six languages, recordings up to 30 minutes, and scenarios with up to 20 speakers. Omnilingual-MsASR provides parallel audio and video inputs to evaluate models' cross-modal consistency and their ability to exploit complementary cues across modalities. Extensive evaluations of leading omni-modal LLMs reveal substantial performance bottlenecks, while their tcpWERs frequently exceed 100%. These findings highlight fine-grained temporal localization, speaker disambiguation, and cross-modal synergy as key challenges for future omni-modal foundation models. Our work is available at https://github.com/QwenLM/Omnilingua-Bench/tree/main/Omnilingua-MSpeaker.