Music inpainting improves audio repair with similarity guided diffusion
Music Inpainting With Similarity-Guided Latent Diffusion
SoundMachine Learning
Summary
Fixing damaged or missing parts of music recordings is hard. The authors developed a new method that finds similar parts within a song and uses them to help an AI system generate better replacements. This makes the fixed music sound more natural and blends well with the rest of the track. Their tests show it works better than older methods on different types of music.
What this means in practice
- •For audio engineers: Restore or replace damaged or missing music segments with contextually fitting audio replacements for smoother edits.
- •For music producers: Improve multi-genre music editing workflows by generating coherent audio substitutes guided by existing music similarity.
Authors
Sean Turland, Eloi Moliner, Vesa Välimäki
Abstract
We propose a music inpainting technique combining similarity matching and a diffusion-based generative model to replace large segments of audio recordings with suitable alternatives. The proposed hybrid method, Similarity-Guided Diffusion Posterior Sampling (SimDPS), retrieves candidate replacement segments from the wider signal based on contextual similarity and uses them to guide the diffusion process toward coherent reconstructions. Being model-agnostic, the framework is compatible with any pre-trained diffusion model operating in either the waveform or latent domain. Objective evaluations demonstrate that SimDPS outperforms widely adopted RePaint and DPS methods across acoustic features and neural-embedding distance metrics. Subjective evaluations on multi-genre music datasets confirm that SimDPS enhances perceptual plausibility compared to previous diffusion-based methods and demonstrates robustness to poorly matched similarity candidates. These findings pave the way for improved music restoration and context-aware audio editing.