LiveSynth enables real-time neural instrument cloning and synthesis
LiveSynth: A Streaming Neural Synthesizer for Instrument Cloning and Text-to-Instrument
Sound
Summary
Making digital instruments that sound like real ones usually takes time and must be done before you can play. The authors created LiveSynth, a system that can listen to a recording or read a text description and then produce music in real time, as you play notes. It uses advanced AI models to generate the instrument sounds continuously without delay. This means musicians can interact naturally with new digital instruments that sound very much like real ones.
What this means in practice
- •For music software developers: Create virtual instruments that generate real-time sounds matching any reference recording or text description without pre-processing.$Commercial implications: Enables live digital instrument products that adapt instantly to user input and custom sounds for musicians and producers.
- •For live performance technicians: Integrate real-time neural synthesizers with low latency into live shows to provide novel instrument sounds that respond instantly to performers.
Authors
Kyungsu Kim, Yejin Kim, Kyogu Lee
Abstract
Recently, generative neural synthesizers have enabled zero-shot instrument cloning from reference recordings and timbre generation from text prompts. However, existing systems generate complete audio segments offline, precluding their use as playable instruments. We present LiveSynth, a streaming neural synthesizer with flexible timbre control and interactive generation capabilities, including zero-shot instrument cloning and text-to-instrument synthesis. A feedback-free causal Transformer conditioned on MIDI events and a CLAP timbre embedding outputs continuous audio latents at 100 Hz, which a causal variational autoencoder decodes into audio. Frame-wise timbre conditioning enables continuous timbre morphing, while MIDI-conditioning dropout enables autonomous continuation from a performed prefix. On the NSynth and Slakh datasets, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines. On an M3 MacBook Air, LiveSynth required 8.8 ms of computation per 10 ms audio frame (an RTF of 0.88) with 10 ms algorithmic latency, enabling real-time interaction with a performer.