Modeling joint audio and parameter distributions improves synthesizer inversion
Off-manifold robustness in synthesizer inversion with joint distribution flow matching
Sound
Summary
Synthesizers turn settings (parameters) into sound, but figuring out which settings created a particular sound can be tricky because many settings can produce similar sounds. The authors propose a new method that learns the connection between sounds and settings by looking at both together, even when some real-world sounds don’t have known settings. This lets their method guess the settings more accurately for real recordings. They also found that adding some noise to sounds during guessing helps improve the results.
What this means in practice
- •For audio software developers: Improve parameter recovery from real-world synthesizer sounds to enable better sound editing and customization tools.$Commercial implications: Enables commercial audio software to offer more accurate automatic parameter extraction for complex synthesizers, enhancing user experience in sound design.
- •For game audio engineers: Enhance audio asset processing by accurately retrieving synthesizer settings from recorded sounds for consistent sound reuse.
Authors
Ben Hayes
Abstract
Recent work on synthesizer inversion shows that generative models outperform deterministic approaches by explicitly modeling the ambiguity in mapping audio to parameters. Training such models, however, requires audio-parameter pairs, which are typically obtained by rendering sampled or preset parameters through the synthesizer itself. This creates a train-test mismatch that can degrade performance on off-manifold real-world recordings, for which ground-truth parameter annotations do not exist. To circumvent this obstacle, we propose to model the joint distribution of audio and parameters with a multi-modal continuous normalizing flow using independent noise schedules for each modality. This formulation allows us to train joint and conditional densities with paired synthesizer data, while unpaired real recordings can train the audio marginal alone, exposing the model to off-manifold signals without requiring parameter labels. Further, because the model learns to map from audio to parameters at all noise levels, we find that partially noising the audio reference at inference improves real-audio reconstruction, consistent with reducing sensitivity to distribution-specific detail while preserving coarse structure. Evaluating on Surge XT and Dexed, we find that modelling the joint distribution substantially improves both inversion of real-world and in-domain audio.