Segment evidence aware system improves multilingual speech question accuracy
SEAR: Segment-Evidence-Aware Routing for Weak-to-Strong Multilingual Speech MCQ
Computation and LanguageSound
Summary
Understanding questions about conversations in many languages is difficult, especially when answers depend on audio details. The authors developed a system that breaks audio into meaningful parts and creates multiple-choice questions (MCQs) about both text and sound. They train their model first on easier text-based questions, then on harder audio-based ones, improving its ability to handle difficult audio tasks. This approach achieved nearly 91% accuracy on a multilingual speech test.
What this means in practice
- •For voice assistant developers: Enhance multilingual voice assistants to better understand complex spoken questions by improving handling of audio context and language variations.
- •For call center automation teams: Improve automated systems to interpret customer speech accurately across languages and accents by using segment-aware question answering.
Authors
Huy Hoang Le, Long-Bao Nguyen, Minh Tri Dao
Abstract
This paper describes our system for Task~2 of the second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge. We adapt Qwen3-Omni-30B-A3B-Instruct with a segment-evidence-aware data and post-training pipeline. A language model converts timestamped ASR into coherent event spans, which are expanded by a boundary margin and cropped from the original recording. We then synthesize complementary semantic MCQs with Qwen3.6-27B and acoustic MCQs with Gemini~3.1 Flash-Lite, followed by structural, grounding, answer-consistency, and target-model trainability checks, yielding 359,825 verified MCQs across 21 language and accent variants. A text-only probe partitions the data into weak, text-answerable items used for supervised fine-tuning and strong, audio-dependent items used for reinforcement learning with Group Sequence Policy Optimization (GSPO), stabilized by debiased advantages, sequence-level importance correction, and dynamic filtering. Our system obtains 90.92% accuracy on the final official evaluation set.