Improving acoustic learning from imperfect teachers in low-resource settings
Learning from imperfect teachers for low-resource acoustic generalization
Artificial Intelligence
Summary
Teaching a speech recognition system can be tricky when the teacher model is trained with little or biased data, since it might give confusing guidance. The authors developed a new way to fix the teacher’s predictions by correcting mistakes and focusing on the most useful parts of its advice. This helps the student system learn better and generalize more accurately, even when the teacher’s knowledge is imperfect. Their experiments show this method works well on small and mismatched datasets.
What this means in practice
- •For speech recognition developers: Improve model training when annotated audio data is limited or imbalanced by using corrected knowledge distilled from imperfect teacher models.
- •For voice assistant engineers: Build more robust acoustic models that generalize better across different labeling budgets and data scarcity scenarios.
Authors
Shuanglin Li, Ruxiao Qian, Jian Liu, Haijun Lin, Wenwu Wang, Siyang Song
Abstract
Knowledge distillation (KD) improves low-resource acoustic learning by enriching one-hot supervision with the softened predictive distribution of a fixed teacher network. However, a teacher trained with limited or imbalanced annotations may produce a biased distribution whose components are not uniformly reliable. Although this distribution can still encode useful knowledge, direct full-distribution matching may also transfer teacher-induced biases, thereby distorting the student's decision boundary and degrading its generalization performance. To address this limitation, we propose Boundary-Anchored Mass-Partitioned Distillation (BA-MPD), a logit-based distillation objective composed of Boundary-Anchored Correction (BAC) and Mass-Partitioned Distillation (MPD). BAC addresses missing ground-truth labels in the set of the teacher's top predictions by swapping the true label for the lowest-ranked entry of the set, thus keeping the mass and uncertainty of the set unchanged. MPD then distills this corrected distribution through separate losses that enforce relational consistency within the set, balance the mass between high- and low-confidence groups, and weight lower-confidence dependencies. Ultimately, BAC and MPD together suppress harmful ranking errors and noisy low-confidence details, while retaining all useful teacher information. Experiments on two acoustic benchmarks under multiple label budgets show that BA-MPD consistently improves over supervised-learning baselines and vanilla KD while remaining competitive with strong logit-based KD baselines. Cross-budget results further show that BA-MPD remains effective when the teacher and student models use mismatched label budgets, demonstrating its ability to exploit imperfect teachers across supervision gaps. Implementation available at https://github.com/ShuanglinLi/BA-MPD.