Prosody-based speech features show no clear benefit beyond fusion method

Does a prosody-trained representation help beyond trainable fusion? A parameter-matched study with frozen HuBERT

Computation and Language

Summary

The study looks at whether adding special speech features related to how people speak, like pitch and energy, helps improve automatic speech recognition beyond just using smarter ways to combine information. The authors compared three setups with and without these extra features, keeping everything else the same. They found that while using the fusion method alone improved accuracy, adding the special speech features made almost no difference. This means the benefit likely comes from how the systems combine data, not from these additional speech features themselves.

What this means in practice

  • For speech technology engineers: Design speech recognition systems by focusing on fusion techniques rather than adding separate prosodic feature modules for accuracy gains.
  • For voice assistant developers: Optimize natural language understanding pipelines by prioritizing effective representation fusion over explicit prosody encoding.

Authors

Ki Woong Moon, Daniel Brenner

Abstract

Explicit prosodic cues may help automatic speech recognition (ASR) of spontaneous speech, but auxiliary representations typically require additional trainable components, making it unclear whether gains come from the auxiliary information or the fusion mechanism. We address this using a frozen HuBERT backbone and a 64-dimensional representation trained to predict log F0, voicing, Delta log F0, log energy, and spectral tilt. We compare a frozen-backbone recognizer (Baseline), trainable fusion with zero auxiliary input (Null), and the same fusion supplied with the learned representation (Learned). Across Buckeye, Switchboard, and AMI IHM, Null reduces WER by 0.71-1.45 points over Baseline, whereas Learned differs from Null by +0.07, -0.09, and +0.00 points, with no significant differences. However, removing or mismatching the representation at inference increases Learned WER. Thus, Learned depends on the representation yet shows no measurable incremental WER benefit over the parameter-matched control.