MuseTimbre transfers instrument sound from one recording to another
MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator
Sound
Summary
Changing the sound of one musical instrument performance to sound like another instrument usually needs special systems trained only for that task. The authors created MuseTimbre, which uses a music generator already trained on many sounds and music pieces to do this timbre transfer just by listening to a sample sound. Their system can change the sound of complex music with multiple notes playing at once while keeping the melody intact. They showed that their method matches the new instrument sound better without messing up the music’s notes.
What this means in practice
- •For music producers: Re-voice polyphonic music tracks to sound like different instruments using an audio clip instead of text descriptions for timbre control.$Commercial implications: Enables creating novel instrument sounds on existing recordings for production and sound design workflows.
- •For audio software developers: Integrate timbre transfer features in music generation tools by conditioning pretrained models on audio instrument samples.
Authors
Yuan-Chiao Cheng, Zhiyao Duan
Abstract
Instrument timbre transfer re-voices a performance using the timbre of another instrument. Extracting the target timbre from an audio reference capture more nuances than inferring it from a text prompt. Systems that read timbre from such a clip train a dedicated model for the task, which captures the timbre cleanly but stays a narrow, single-purpose system. More versatile approaches add control to a pretrained music generator, yet a reference clip entangles timbre with genre and melody, so these systems fall back on text to name the timbre. We present MuseTimbre, the first system, to our best knowledge, that transfers timbre from an audio reference to a polyphonic source through conditioning a pretrained music generator. This system employs a multi-pitch estimator to extract pitch information from the source and finetune a CLAP encoder to extract timbre information from the reference audio. Experiments show that across four datasets of real polyphonic recordings, MuseTimbre achieves pitch alignment on par with the baselines while matching the reference timbre far more closely. Results also show that the finetuned CLAP-based timbre extractor is robust to pitch variations, making it useful in timbre similarity measures.