Multilingual speech systems adapt new languages while keeping old ones
Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR
Computation and Language
Summary
Expanding speech recognition systems to support more languages is hard because changes can make the system forget languages it already knows. The authors studied how to update parts of a large speech-to-text model that connects spoken words to language understanding, without needing to retrain everything from scratch. They tested different ways to add new languages while avoiding loss of previous language skills. They found some update methods that balance learning new languages and preserving the old ones better than others, especially when adding multiple languages.
What this means in practice
- •For speech recognition engineers: Improve multilingual speech systems by updating connector layers to add new languages without losing accuracy on existing ones.
- •For machine translation developers: Maintain multilingual text-processing modules that adapt to additional target languages while preserving original language capabilities.
Authors
Yuchen Zhang, Haralambos Mouratidis, Ravi Shekhar
Abstract
Expanding multilingual ASR systems to new languages is challenging when adaptation must preserve previously supported languages without replaying source-language training data. We study this problem in connector-based Speech LLMs, where a trainable connector maps speech representations into a frozen large language model. We evaluate target-update construction and merging strategies for adapting the connector while preserving source-language performance. In single-target expansion, target-only fine-tuning and standard TIES merging learn new languages but cause severe source-language forgetting, while SSU-Wanda+TIES gives the best source-preserving trade-off. In multi-target expansion, update merging provides stable source retention, whereas recursive expansion can improve target adaptation under favorable orders but is highly order-sensitive. These results show that multilingual ASR expansion requires balancing target acquisition, source retention, and interactions among target-language updates.