Joint embeddings link audio effect code with its sound output
PEACE: Joint Embeddings of DSP Effects Code and Audio
Sound
Summary
Matching computer code for audio effects with the sounds they produce is hard. The authors created PEACE, a technique that learns to connect audio effect code written in a special language with the sound it creates. PEACE uses two different ways to understand the code and compares them with the sound, helping it recognize how the code changes the audio. This makes it easier to search for audio effects or understand how a particular sound was made by looking at the code behind it.
What this means in practice
- •For audio software developers: Improve search tools to find audio effects by linking sounds directly to their effect code, enhancing effect discovery and design.
- •For music production engineers: Use embeddings to better understand and recreate audio effect chains from sounds without needing direct parameter knowledge.
Authors
David Braun, Adam Finkelstein
Abstract
This paper introduces PEACE, the first joint embedding of audio effect code and output audio. Building on SLAP's multimodal objective, we pair an AFx-Rep audio encoder with two code encoders for Faust, a functional language for audio signal processing. First, we evaluate a fine-tuned T5 transformer over Faust source code. Second, we evaluate a message-passing graph neural network over an intermediate representation of the Faust compiler, capturing both topology and UI parameters. We evaluate on audio-to-code retrieval, where masking UI parameters at inference yields embeddings that encode effect chain topology alone. When parameters are visible, the two code encoders tie on retrieval of mixed-length chains but have tradeoffs on single-effect galleries. With parameters fully masked, PEACE recovers ordered chain topology far above chance without the limitations of supervised methods. PEACE outperforms pretrained models on an out-of-distribution reverb retrieval benchmark and can improve frozen audio-only representations. Its dual understanding of topology and parameters lays the groundwork for music information retrieval systems that search, generate, and condition on DSP code.