Exploring the Design Space of Representation Learning for Audio Transformations
Sound
Summary
The authors study how to make better audio features that help understand not just sounds but also how sounds are changed by processing. They introduce a new way to learn audio features using three goals that capture different aspects of audio processing. Their method creates two types of features that work well for different tasks: one for finding similar audio effects and another for analyzing the processed sounds. They show that their approach, along with improvements in training, works better than previous methods on tasks like retrieving sounds, analyzing audio characteristics, and changing audio style.
Authors
Sungho Lee, Marco Martínez-Ramírez, Junghyun Koo, Wei-Hsiang Liao, Kyogu Lee, Yuki Mitsufuji
Abstract
Neural audio representation learning has enabled a range of content-oriented applications, but the resulting features remain limited for tasks involving audio processing. Furthermore, it is not obvious what processing-aware representations should capture: the processing itself, abstracted away from source content, or the processed audio that retains it. Existing approaches implicitly commit to one or the other and also differ in their models, data, and evaluation, obscuring which design choices drive their behavior. We address both questions within a unified framework of three objectives: processing consistency, description alignment, and equivariance via forward prediction. We compare all combinations of the objectives under a controlled setup and reveal their relative strengths and interactions. Our framework produces both a transformation embedding and a processed-audio embedding, and we find that the two play complementary roles: distance-based tasks favor the former, while probe-based tasks favor the latter. Combined with improvements in network architecture and training pipeline, our representations outperform prior baselines across retrieval, probe-based evaluation, and style transfer.