Music stem retrieval improved by separate slot embeddings for tracks
Retrieving Individual Stems from Music Mixtures with Slot Embeddings
Sound
Summary
When music producers want to find isolated parts of songs like a guitar or drums, they search large libraries of these parts called stems. The authors propose a new method named Stembed that breaks down a full song mix into several separate pieces called slot embeddings, each representing a possible stem. This method better matches parts of a mixture to individual isolated stems compared to previous approaches that looked at the entire mix as one single piece. It works even without knowing which instrument family to focus on and can predict how many stems there are.
What this means in practice
- •For music producers: Search large stem libraries more accurately for parts matching specific instruments or song sections without needing instrument family labels.$Commercial implications: This enables commercial music software tools to offer more accurate stem-based search features for producers and audio engineers.
- •For podcast editors: Isolate individual sound components in mixed audio recordings to improve editing by retrieving matching isolated audio stems.
Authors
David Braun, Junyi Fan, Pranay Manocha, Donald S. Williamson, Adam Finkelstein
Abstract
Music producers search libraries of isolated instrument recordings, called stems, for sounds resembling parts of an existing song. Neural retrieval systems address this by mapping audio to embeddings and ranking library stems by their similarity to the query. The leading method, Contrastive Instrument Retrieval (CIR), encodes the mixture as a single embedding, but it works best when a user specifies the target's instrument family. We introduce Stembed, which encodes a mixture as several slot embeddings representing candidate stems. During training, we construct mixtures from stems of the same song and match their slot embeddings to those of the isolated stems. The slot embeddings from mixtures inherit the stem identities of their assigned solo embedding, enabling a contrastive loss. On mixtures from held out MoisesDB artists, Stembed outperforms a CIR-style baseline when both search the full stem library. Even when predicting the stem count itself without family labels, Stembed exceeds the baseline's family-filtered R@1. Our website demonstrates how users can select a slot by inspecting the tags of its retrieved stems.