Multilingual speech model improves speaker clustering without labels
BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challenge
SoundComputation and Language
Summary
This paper tackles the challenge of understanding speech from many languages without using any labeled data. The authors trained a speech model that predicts hidden labels created by grouping similar sounds and also tries to identify languages from audio. Their model outperforms several others at grouping speakers correctly but still struggles to match the accuracy of models trained with labeled data when identifying languages or recognizing words. They also found that some evaluation methods used during training do not always predict real-world performance perfectly.
What this means in practice
- •For speech technology developers: Build speech models that identify and group speakers across many languages without labeled training data.
- •For voice assistant engineers: Improve speaker clustering performance in voice applications supporting multiple languages where labeled data is limited or unavailable.
Authors
Prakriti Subedi, Howard Prioleau, Saurav K Aryal
Abstract
We describe our submission to the Unsupervised Speech in the Wild (UPS) Challenge at Interspeech 2026, a bidirectional Mamba-2 (BiMamba2) encoder trained with masked discrete-unit prediction following the HuBERT-style paradigm. The 47.88M-parameter model is trained on 250 hours of speech across 67 languages from the MLCommons Unsupervised People's Speech dataset, with no labeled data. The objective combines masked k-means pseudo-label prediction with language identification supervision and VICReg regularization. On official evaluation, the system achieves an Adjusted Rand Index of 0.735, exceeding four baselines on speaker clustering. Language identification macro-F1 (0.073) and character error rate (0.870) remain below supervised baselines. We analyze a local-official discrepancy in metric scale and checkpoint ranking, highlighting limitations of in-distribution diagnostics for predicting Dynabench probe outcomes.