Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026
2026-06-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors describe their method for improving language models that follow instructions in multiple tasks and languages. They created a large dataset by combining short texts into longer ones, generating labels with large language models, and translating between languages. They found a common ranking method worked well for speech recognition but caused problems in understanding tasks with meaning, which they fixed by using a more balanced decoding approach. Their work aims to handle both known and surprise tasks better in instruction-based language models.
Large Language ModelsInstruction FollowingData AugmentationSegment ConcatenationCross-lingual TranslationLikelihood Re-rankingMinimum Bayes Risk DecodingAutomatic Speech RecognitionSemantic Tasks
Authors
Enes Yavuz Ugan, Maike Züfle, Yuka Ko, Supriti Sinhamahapatra, Fabian Retkowski, Seymanur Akti, Jan Niehues, Alexander Waibel
Abstract
With the advent of Large Language Models, single-task and token-based multi-task models have evolved into instruction-based systems that infer task and target language implicitly from natural language prompts. This trend is reflected in IWSLT's Instruction Following Track, which this year introduced new tasks including an unknown surprise task, posing a genuine challenge against overfitting to known tasks. We present KIT's submission to the Long and Short Instruction Following tracks in the unconstrained setting. Our approach combines a general data augmentation pipeline that converts short-form corpora into long-form training data through segment concatenation, LLM-based label generation, and cross-lingual translation, yielding over 1M instances across six tasks and four languages. We further show that likelihood-based re-ranking, while highly effective for ASR, systematically degrades semantic tasks by spuriously selecting candidates generated from segmented audio processing rather than holistic long-form inference, a failure mode resolved by combining likelihood with Minimum Bayes Risk decoding.