Papers for

game sound designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Open stem audio dataset enables advanced music AI training

SPDMX: Large-Scale Open Stem Audio From Symbolic Music

Abstract: Large public datasets have driven recent progress in machine learning, from text and vision to speech and general audio. Music remains an outlier, as copyright blocks redistribution of training audio, while many modern tasks need individual stems rather than mixtures. Existing multitrack resources trade off realism, scale, and licensing, while synthetic alternatives rely on proprietary VSTs or naive open rendering that yields weak stems. We introduce SPDMX, a listening-guided rendering of public-domain PDMX scores into an openly-licensed stem corpus with aligned scores and an open render pipeline: $253{,}982$ songs, $490{,}159$ mono stems, and $5{,}469$ song-hours at $44.1$ kHz under CC BY 4.0, with paired MIDI and identifiers linking to PDMX metadata. $88{,}403$ songs ($34.8\%$; $2{,}734$ song-hours) provide multitrack supervision. To our knowledge, this is the largest openly-licensed synthetic multitrack subset by song-hours, enabling source separation, stem-conditioned synthesis, MIDI-to-audio, transcription, and joint symbolic-audio modeling. We release the corpus, metadata, and open pipeline for reuse on other symbolic collections. Demos and experimental notes are on the project page.

Fri 6 NovSoundArtificial IntelligenceMachine Learning
The gist
Music AI research struggles because it needs isolated instrument tracks (stems), but music recordings are often copyrighted and not shared publicly. The authors created a large public collection of synthetic stems by converting public-domain music scores into high-quality audio, fully open to everyone. This dataset contains hundreds of thousands of songs with their stems and aligned MIDI information, allowing AI systems to learn tasks like separating instruments, generating music from MIDI, and transcribing music. Their work releases the dataset, metadata, and the tools used so others can build on it.
Open → 2611.09140v1

MFlowAudio speeds up text to audio with efficient flow matching

MFlowAudio: Efficient Text-to-Audio Synthesis via Mamba-based Stateful Flow Matching

Abstract: Recent advancements in audio generation have been largely driven by Transformer-based diffusion models. However, these models suffer from quadratic complexity of self-attention, severely bottlenecking the long-form audio synthesis. To overcome this limitation, we propose MFlowAudio, a novel latent audio generation framework that synergizes the continuous-time dynamics of Flow Matching with a custom-designed TCMamba backbone. TCMamba uses an innovative dual-scan mechanism: a TimeMamba module to capture long-range causal dependencies with linear complexity, and a ChannelMamba module to model cross-channel correlations such as harmonic structures. Exploiting this structural foundation, we formulate the Stateful Flow Matching (SFM) paradigm. This framework inherently enables chunk-wise training and streaming generation, maintaining an $\mathcal{O}(1)$ caching complexity without incurring extra computational overhead. To enable fine-grained controllable synthesis, we devise a novel guidance mechanism that neutralizes vector field collisions precipitated by on-the-fly semantic transitions of prompts. Comprehensive empirical evaluations confirm that MFlowAudio yields generative fidelity on par with state-of-the-art baselines while establishing significant computational efficiency, achieving above $14.5$% acceleration in generation and facilitating perceptually seamless streaming synthesis with a constant latency of $1.8$ seconds. Demo: https://huggingface.co/spaces/mflowaudio/MFlowAudio

Fri 30 OctSound
The gist
Generating long pieces of audio from text is usually slow because the common methods have to look at everything all at once. The authors introduce MFlowAudio, which uses a new way of processing audio that breaks down the task into smaller parts working together efficiently. This method reduces the computing power needed and allows audio to be generated steadily while listening, with delays kept very short. Their tests show this system is as good in sound quality as current leading methods but faster and more practical for long audio.
Open → 2610.41852v1

Soundwich generates video with separate editable audio tracks

Soundwich: Video Generation with Layered and Controllable Audio

Abstract: Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.

Wed 30 SeptComputer Vision and Pattern RecognitionMultimediaSound
The gist
Creating videos with matching sounds is common, but usually the sounds come as one mixed track, making it hard to edit parts like speech or music separately. The authors made Soundwich, a tool that splits sounds into separate tracks for speech, music, effects, and more, all perfectly timed with the video. It keeps these sound tracks connected so they make sense together, yet lets you change, mute, or move any part on its own. This approach helps people work with video and sound like in real studios, letting creative editing while keeping everything synced.
Open → 2610.00691v1

EmoRES improves emotional speech control in text to speech systems

EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

Abstract: Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. We therefore study vector steering, a training-free approach that modifies the internal representations of a frozen model. CoCoEmo, a conventional vector steering method for emotion TTS, treats each emotion vector as an indivisible direction controlled by a single global strength, limiting adherence to the requested emotion. In this work, we first discover that an emotion vector can be decomposed into a shared component that moves speech away from neutral expression and a residual component that directs generation toward the requested emotion. Building on this finding, we propose Emotion Residual-Enhanced Steering for TTS (EmoRES), a novel method that controls the two components without retraining the backbone. On IEMOCAP, EmoRES outperforms CoCoEmo across all four objective emotion metrics on the IndexTTS-2 and CosyVoice2 backbones. Rank correlation improves by 26.13 and 12.97 percentage points, corresponding to relative gains of 118.8% and 33.1%, while emotion hit rate improves by 12.95 and 6.92 points, corresponding to relative gains of 20.1% and 9.8%. Human evaluation further shows a relative improvement up to 35.0% in the rate at which listeners correctly identified the dominant requested emotion and up to a 17.3% improvement in fidelity, while listeners prefer EmoRES for naturalness in up to 63.8% of pairwise comparisons. Component ablations further demonstrate that effective control benefits from preserving the shared component while strengthening the residual of the emotion steering vectors.

Tue 29 SeptSoundComputation and Language
The gist
Sometimes, computer systems that turn written text into spoken words struggle to express emotions correctly. The authors found that an emotion signal inside these systems can be broken into two parts: one that just changes speech away from neutrality, and another that steers it toward a specific emotion. They created a method called EmoRES that adjusts these parts separately without needing extra training. This approach helped machines produce speech that better matches emotions and sounds more natural, according to tests and listener feedback.
Open → 2609.38157v1

Text to audio generation improved by joint learning of sounds and models

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation

Abstract: Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. However, reconstruction-oriented representations may be suboptimal for generation, motivating joint representation and generative learning. To this end, we introduce \textbf{Unite-Audio}, to our knowledge, is the \textbf{first} to jointly learn continuous audio representations and latent flow matching for TTA. By coupling reconstruction with self-supervised generative prediction, Unite-Audio allows the generative objective to directly shape the latent space rather than treating it as a fixed intermediate representation. We further employ Flow-GRPO post-training to improve text-conditioned generation. Experiments show competitive TTA performance with a compact latent flow model, while ablation studies confirm the benefit of jointly learning the audio representation and generative model. Audio samples are available at https://runwushi.github.io/Unite-Audio.

Wed 23 SeptSound
The gist
Creating audio from text usually involves first turning sounds into a code and then generating new sounds from that code. The authors found that making the code and the sound generation model learn together leads to better results. Their new method, called Unite-Audio, improves how natural the generated audio sounds based on text descriptions. They also added a way to fine-tune the model after training to produce even better audio.
Open → 2609.28206v1