StemFX: Learning Mixing Style Representations via Autoregressive FX Chain Prediction on Source-Separated Stems
Sound
Summary
The authors developed StemFX, a system that learns how audio mixing styles work by predicting the order and types of effects used on separate parts of a song. It uses advanced machine learning techniques, like Transformers and CNNs, to understand and recreate how mixing effects are applied to different audio stems. To train their model, they created a large dataset from over 100,000 songs and used a tool that applies various audio effects. Their approach performs better than earlier methods in tasks like retrieving mixing styles and transferring them between tracks, while also being much faster.
Authors
Yuan-Chiao Cheng, Jui-Te Wu, Brian Chen, Yen-Tung Yeh, Yu-Hua Chen, Yi-Hsuan Yang
Abstract
Audio mixing style encompasses the artistic and technical decisions a mix engineer makes, including level balancing, spatialization, and the choice, ordering, and parameterization of audio effects (FX) on each stem. FX chains are a key determinant of this style, yet existing approaches to modeling them remain limited. Some operate on stereo mixtures without explicit per-stem FX chain modeling, others fix the number or type of effects per track, and many require differentiable effect implementations or scarce multitrack datasets. We present StemFX, a framework that learns mixing style representations by autoregressively predicting variable-length FX chains on source-separated stems. A Transformer decoder predicts tokenized FX chains autoregressively, while a band-split multi-band CNN encoder with FiLM conditioning captures per-stem spectral structure. To enable large-scale paired training, we extract pseudo-stems from about 105K songs via source separation and augment them using MultiAFx, a toolkit unifying 85 audio effects from 7 Python libraries. Evaluated on mixing style retrieval, StemFX outperforms all baseline models across all tested chain lengths. On paired mixing style transfer, StemFX achieves the best spectral fidelity and the highest listener preference, over 4000 times faster than iterative optimization.