Audio visual speech recognition benchmark captures natural dialogues
Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition
Computer Vision and Pattern RecognitionMultimedia
Summary
Speech recognition systems often struggle with natural conversations because most tests use clean, scripted speech. The authors created Candor-LR, a large dataset of real video conversations that include overlapping speech and spontaneous dialogue. They found that recognizing speech from audio alone becomes harder in these real scenarios, but adding visual lip-reading helps a lot. Training models with this dataset also makes them work better across different noisy conditions.
What this means in practice
- •For speech technology developers: Improve speech recognition systems to handle natural, overlapping, and noisy conversational speech better using the Candor-LR dataset.
- •For video conferencing platform engineers: Enhance live transcription performance during real conversations by integrating audio-visual speech models trained on realistic dialogue data.
Authors
Rishabh Jain, Aristeidis Papadopoulos, Zhaofeng Lin, Naomi Harte
Abstract
Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the complexity of natural conversation, which involves overlapping speech, spontaneous turn-taking, unscripted vocabulary and variable acoustic conditions. To shift the field toward realistic dialogue, we introduce Candor-LR, a conversational benchmark derived from the CANDOR corpus of 1,656 natural dyadic videoconferences. Our custom data preparation pipeline yields 713.5, 10.1, and 60.1 hours of training, validation, and test data, respectively. Evaluating pretrained AVSR models on Candor-LR reveals that audio-only accuracy drops sharply compared to LRS3, but visual cues compensate effectively, driving much larger performance gains on Candor-LR than on LRS3. Furthermore, training on this corpus significantly improves cross-domain robustness under both clean and noisy conditions, as its realistic conversational data captures broader audio-video features. We open-source our pipeline to ensure reproducibility, establishing Candor-LR as a challenging benchmark for conversational AVSR.