Evaluating Loss Functions in Differentiable Out-of-Domain Sound-Matching with Partial Parameter Distance
SoundArtificial Intelligence
Summary
The authors studied how to measure how well a sound synthesizer can copy sounds it wasn’t originally designed to make, which is tricky because usual methods rely on shared settings between synthesizers. They created a new way called Partial Parameter Distance (PPD) that focuses only on important shared settings, allowing them to test different loss functions automatically. By comparing different loss functions across various sound changes, they found that each function works better for specific sound features. Their method matched well with human listening tests, showing it can help understand which techniques work best for different sound tasks.
Authors
Amir Salimi, Daniel Penner, Kalvin Eng, Abram Hindle, Osmar R. Zaïane
Abstract
In out-of-domain (OOD) sound-matching, a synthesizer is optimized to mimic a sound it did not generate. OOD evaluation of loss functions is underexplored in part because the standard "parameter loss" metric requires a shared parameter space between target and imitator, which OOD settings lack. We introduce Partial Parameter Distance (PPD), which applies parameter loss only to the critical parameters that mismatched synthesizers share (e.g., filter cutoffs), enabling automatically evaluated OOD experiments; we verify its results with blinded listening tests. Across seven scenarios involving band-pass filtering, amplitude modulation, and pitch-bending, we evaluate four differentiable loss functions (SIMSE_Spec, L1_Spec, JTFS, DTW_Envelope). Loss-function effectiveness remains tightly coupled to the method of synthesis: SIMSE_Spec excels at filter-cutoff recovery, DTW_Envelope at amplitude-modulation recovery, and JTFS at smooth pitch trajectories. Parameter-based evaluation agrees with listening tests on the top-ranked loss function in five of seven scenarios, demonstrating its utility as a diagnostic tool.