Diffusion model improves harmonic sound mixture parameter estimation
Differentiable Digital Signal Processing Mixture Model-Guided Diffusion for Synthesis Parameter Estimation from Harmonic Sound Mixtures
Sound
Summary
When multiple musical sounds play together, it's hard to figure out each sound’s individual pitch, loudness, and tone over time. The authors built a new computer method that uses two techniques together to better guess these details in mixtures of sounds. Their approach uses a special model to make sure the guesses change smoothly over time, avoiding unrealistic jumps. Tests showed this method worked well for woodwind and string instrument sounds playing at the same time.
What this means in practice
- •For audio software developers: Improve tools for separating and editing individual instrument sounds in recordings with more realistic parameter tracking over time.
- •For music technology companies: Create enhanced music synthesis products that estimate instrument tones more smoothly from complex audio mixtures.$Commercial implications: Enables selling advanced instrument sound extraction and synthesis software with better temporal accuracy for music producers.
Authors
Kengo Takemoto, Tomohiko Nakamura, Hiroshi Saruwatari
Abstract
A differentiable digital signal processing (DDSP) autoencoder reconstructs a monophonic harmonic sound through three types of synthesis parameters: fundamental frequency, loudness, and timbre features. To handle mixtures of harmonic sounds within the DDSP approach, we have previously proposed a DDSP mixture model (DDSPMM). It represents a mixture as the sum of source signals synthesized by the decoders of pretrained DDSP autoencoders. Although DDSPMM enables direct estimation of synthesis parameters of each source from mixtures, it does not explicitly model temporal variations in the synthesis parameters and can produce excessive temporal fluctuations. In this paper, we propose a method for estimating synthesis parameters with temporally plausible trajectories by incorporating a denoising diffusion probabilistic model (DDPM) into the DDSPMM-based estimation. The DDPM is trained as a generative model of synthesis parameters. During estimation, the proposed method guides the DDPM reverse diffusion process with the reconstruction error between the observed mixture and the mixture synthesized by DDSPMM from the current estimates. Experiments on woodwind and string instrument ensembles showed that the DDPM-based regularization improves synthesis parameter estimation by imposing temporal plausibility on the estimated trajectories.