AutoSynth generates editable synthesizer programs from audio and text
AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text
Sound
Summary
Synthesizers can create sounds using sets of instructions, but it is hard to get those instructions directly from sounds or text descriptions. The authors developed AutoSynth, a system that learns to reverse-engineer sounds into exact synthesizer programs or turn text into these programs through audio. This method allows easy editing of musical notes and sound parameters, improving over models that only produce sound waves. The training combines large datasets of sound-program pairs and an optimization step that rewards similarity in meaning, pitch, and sound quality.
What this means in practice
- •For music producers: Create editable synthesizer patches from recordings or text descriptions to simplify sound design workflows.$Commercial implications: Enables software tools that convert audio or text into customizable synthesizer settings for music creation.
- •For audio software developers: Integrate program inference from sound into audio editing tools to provide users with direct control over synthesizer components.
Authors
Tristan Wu, Daniel Chin, Liwei Lin, Junan Zhang, Gus Xia
Abstract
Audio generation models can translate natural-language descriptions into sound, but their outputs are typically waveforms. Their audio quality is constrained by audio compression, and their outputs do not readily support direct edits to notes, timbral parameters, or modulation relationships. We present AutoSynth, which represents MIDI performance events, fixed synthesizer parameters, and variable-length modulation routes as a unified sequence for a synthesizer, and learns their dependencies with an audio-conditioned autoregressive model. A single model supports both tasks. Given reference audio, the model directly predicts a synthesizer program; given text, it uses a pretrained audio generation model and converts the generated audio into a program. Training consists of two stages: supervised learning on large-scale audio-program pairs automatically constructed from a small set of native presets, followed by group-relative policy optimization with a mixed reward combining semantic similarity, pitch-related features, acoustic similarity, and sound usefulness. The pipeline requires neither paired text-target-program annotations nor a differentiable synthesizer. Experiments show that AutoSynth produces complete, editable synthesizer programs and achieves competitive results in both synthesizer inversion and text-driven generation. Audio demos and source code are available at https://auto-synth.github.io/.