Model transforms everyday sounds into music notes with real-time control
Sori: Streamable and Controllable Model for Non-musical Audio to Symbolic Music Transformation
Sound
Summary
Turning everyday sounds into musical notes is hard because such sounds don't have a single correct musical form. The authors introduce a task called NASMT that tries to convert any non-musical audio into symbolic music representations, which can vary widely. They built Sori, a Transformer-based model that learns to bridge the gap between musical and non-musical sounds and lets users control how faithful or musical the output is. Sori runs fast enough for live use on a regular computer, and tests show it works well compared to other models.
What this means in practice
- •For audio software developers: Integrate real-time conversion of arbitrary sounds into music notation with controllable fidelity and musicality.
- •For sound designers: Create musical representations from non-musical sound effects for use in multimedia projects.
Authors
Kyungsu Kim, Yejin Kim, Kyogu Lee
Abstract
We introduce Non-musical Audio to Symbolic Music Transformation (NASMT), a novel task that converts non-musical audio into symbolic music representations. Unlike automatic music transcription, which assumes musical input and a unique correct score, NASMT is inherently one-to-many and requires evaluation along two axes: input correspondence and musicality. We present Sori, an encoder--decoder Transformer that addresses this task through domain adversarial training (DAT), which bridges the gap between musical and non-musical audio in the encoder's feature space, and classifier-free guidance (CFG) with a continuous parameter~$γ$ that controls the trade-off between correspondence and musicality at inference time. Across objective metrics and a listening study, Sori provides effective continuous control via $γ$ and compares favorably with instrument-agnostic transcription models. Sori supports real-time streaming on a consumer CPU. Audio examples are available at https://soridemo0320.github.io/sori-demo/.