Correlation loss improves merging speech and music encoders
Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders
Sound
Summary
Combining separate models for speech and music into one makes it simpler to handle both kinds of audio. The researchers found a better way to teach these small models so that when combined, they work more smoothly together. They showed that changing the way the model learns — using a correlation-based loss instead of the previous loss — helps the merged model perform better on speech tasks and makes the merging process easier. This improvement works for two different merging techniques, suggesting the key is in how the model represents sounds.
What this means in practice
- •For voice assistant developers: Deploy a single compact encoder to handle both speech and music in voice-enabled devices more efficiently with improved merging.$Commercial implications: Enables producing a cost-effective single model for commercial smart speakers and assistants that process speech and music seamlessly.
- •For audio software engineers: Build unified audio processing tools that handle speech and music with a better-merged encoder, reducing the need for multiple models.
Authors
Fabian Ritter-Gutierrez
Abstract
A single compact encoder for both speech and music removes the need to maintain a separate model per domain. A practical recipe distils a speech teacher and a music teacher into two small students, then merges them into one model. Two merging algorithms exist: interpolating the student weights from a shared initialisation, or permuting their channels into alignment before averaging. Prior work fixed the distillation loss in both, leaving open whether the loss itself affects how well the students merge. We show that it does. Holding architecture and evaluation fixed, we replace the DistilHuBERT loss \Lkd{} with a correlation-based loss \Lcl{}. Under weight interpolation, \Lcl{} students stay closer to their own better endpoint in $17$ of $18$ comparisons and lead on the speech tasks at every interpolated weight. Under activation permutation, they need fewer channels rearranged at three layers, their matched channels correlate more strongly, and they again lead on speech. The two merging algorithms share no mechanism, yet both improve under the same loss substitution. This suggests that the gain lies in the representation \Lcl{} produces rather than in either algorithm.