Speech intensity controlled continuously with learned neural converter

Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training

Machine LearningSound

Summary

Sometimes people want to change how loud a recorded voice sounds without just making it louder or quieter, because simple volume changes don't capture how intensity feels in speech. The authors created a computer method that can change a voice recording to sound like it was spoken louder or softer at any level you want, not just a few preset steps. Their method uses a kind of AI called a GAN with a speech encoder and vocoder to do this without needing matching original and modified recordings. They tested different ways to represent the voice features and found one that works well for matching the desired loudness changes and also slightly changes the pitch, making the output sound more natural.

What this means in practice

  • For voice assistant developers: Generate voice responses at precisely chosen loudness levels to suit different environments or user preferences without needing reference recordings.
  • For audiobook producers: Modify vocal intensity of narrations continuously across recordings to enhance expressiveness without re-recording or complex editing.$Commercial implications: Enables production tools that can custom-adjust narration loudness with natural quality, improving audiobook listening experience.

Authors

Quentin Le Tellier, Albert Rilliard, Olivier Perrotin, Marc Evrard

Abstract

Vocal intensity conversion renders a recorded utterance at a different vocal intensity, rather than simply scaling its amplitude, while preserving its content and speaker identity. Existing systems control intensity through a reference utterance or a few discrete categories. Instead, we control the conversion with any target sound pressure level (SPL) on a continuous dB scale. A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder resynthesizes the speech. We compare three representations, log-mel spectrograms, WavLM, and SpeechTokenizer, and introduce an objective protocol that measures the achieved intensity change against the target one on a calibrated corpus. With WavLM, the converter achieves a measured 83% of the target change in intensity and also shifts the fundamental frequency. A perceptual evaluation confirms that perceived vocal intensity increases with the target level on amplitude-normalized stimuli.
View PDFOpen arXiv