Differentiable vocal tract simulation reconstructs speech anatomy from sound
Reconstructing the Vocal Tract with Differentiable Acoustic Simulation
SoundComputation and Language
Summary
Voices are shaped by the vocal tract, which changes shape to produce different sounds. The authors developed a computer simulation that can mimic how sound travels through the vocal tract and can learn the shape of the vocal tract just by listening to the sounds it makes. They made this simulation faster and more precise by using math tricks and neural networks, allowing it to handle tricky sounds like consonants. This lets them rebuild detailed vocal tract shapes from speech alone, potentially helping with speech research and medical imaging without needing expensive scans.
What this means in practice
- •For speech technology developers: Automatically reconstruct vocal tract shapes from speech audio to improve speech synthesis and analysis without direct imaging data.
- •For medical imaging engineers: Generate moving vocal tract images from speech recordings for non-invasive diagnostics and therapy monitoring without paired MRI scans.
Authors
Eric Ming Chen, Jin Woo Lee, Vincent Sitzmann
Abstract
The vocal tract is the region of the human body responsible for filtering one's voice to create speech. In this paper, we present a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it is readily integrated with other deep learning pipelines to enable novel linguistics and medical imaging applications. (1) We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and (2) we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's moving vocal tract from only their speech without paired data.