Cross correlation drives noise robustness in speech model compression
What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction
Sound
Summary
Huge speech AI models are accurate but too big for many devices. Compressing them often loses their ability to work well with noisy sounds. The researchers found that a specific part of the compression method, called cross-correlation, actually helps the smaller model ignore noise better. Another part, called self-correlation, helps the model work well on clear sounds but doesn't improve noise handling. So, tuning these parts separately during compression can make smaller speech models both noise-resistant and effective.
What this means in practice
- •For speech recognition engineers: Create smaller speech recognition models that maintain noise resistance by emphasizing cross-correlation during knowledge distillation.
- •For audio signal processing teams: Design feature extraction pipelines for music and speech that balance noise robustness and clean-signal accuracy through distinct correlation terms.
Authors
Fabian Ritter-Gutierrez, Nancy F. Chen, Eng Siong Chng
Abstract
Self-supervised learning (SSL) speech models are accurate but large. Knowledge distillation compresses them, but the student loses the teacher's noise robustness. Correlation-based distillation addresses this with two terms: cross-correlation aligning student and teacher's representations, and self-correlation decorrelating the student's features. Both the original method and De'HuBERT credited the self-correlation term without isolating it. We show the opposite. On LibriSpeech-100 distillation with held-out CHiME-3 noise at 10\,dB, an unbiased full-dimensional probe, a Pearson-variance decomposition, a same-noise causal control, and a per-dimension analysis identify the cross-correlation diagonal as the mechanism that encourages noise invariance, lowering noise-classification accuracy from 76.98\% to 55.02\%, whereas adding the self-correlation term back leaves it at 55.55\%. The self-correlation term instead reorganises the feature space and improves accuracy on the clean downstream tasks, but removes essentially no noise. Across nine speech and music tasks this yields a concrete design rule for distillation: weight the cross-correlation term for noise robustness, tune the self-correlation term for better downstream performance, and sample teacher and student noise independently.