System clones instrument sounds directly from full music recordings
Point-and-Play: Instrument Timbre Cloning from Music Mixtures
Sound
Summary
It’s hard to copy the sound of a specific instrument when it plays with others in a song. The authors made a system that can pick out the sound of one instrument right from a recording with many instruments playing together. This system then uses that sound to create new notes with the same instrument’s timbre. Their method works better than first separating the instruments and then copying the sound, matching the quality of using a clean solo instrument recording.
What this means in practice
- •For music producers: Generate new musical notes with the exact sound of an instrument extracted from a mixed recording without needing isolated tracks.$Commercial implications: Enables music production tools that clone instrument sounds directly from mixed songs, appealing to producers wanting custom sounds without multi-track access.
- •For audio engineering teams: Improve audio editing workflows by extracting and synthesizing individual instrument sounds from full mixes without separate source separation.
Authors
Baotong Tian, Zhiyao Duan
Abstract
Neural instrument synthesizers can clone the timbre of a solo instrument, letting music producers make music with the sound they desire. However, an instrument of interest is often embedded in a fully produced, multi-instrument recording, beyond the reach of existing methods. Source separation before timbre cloning is an approach, but it may suffer from separation artifacts and is also likely a detour for the task. In this work, we propose Point-and-Play, a system that clones the timbre of a cued source directly from a polyphonic mixture and uses it to synthesize new pitches and velocities. To achieve this, we introduce a Cue-Gated Statistics Pooling (CGSP) module that extracts the timbre code of the cued instrument from adapted CLAP embeddings of the music mixture. This extracted timbre code is then used to condition a separately trained note synthesizer together with pitch and velocity inputs. Experiments show that our proposed system outperforms separate-then-clone baselines in terms of timbre fidelity and synthesis quality, achieving comparable performance with that cloned from a clean solo instrument.