Papers for

game audio developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Relational synthesis creates new consistent sounds from paired audio clips

Relational Synthesis: Structure-Mediated Concatenative Synthesis for Foley and Retrieval-Augmented Audio Generation

Abstract: We ask: given a retrieved source audio $S$ and a separate reference audio $R$, can we synthesize novel audio $Y$ out of this pair $(S,R)$ such that $Y$ remains acoustically consistent with $S$, while not persistently copying segments of $S$ or $R$? The first clause is a well-known goal in Foley audio production, and the second is a well-known issue in neural RAG when $S$ and $R$ are naively injected into neural generators. We show that both clauses can be addressed simultaneously using a method we coin relational synthesis, a variation of concatenative synthesis where target cost is replaced by a relational Gromov-like structural cost. Rather than imitating the content of $R$, relational synthesis exploits it from the "other side of the hill": it transfers the temporal structure and directed amplitude motion of $R$ to reorganize and concatenate the grains of $S$ in a novel manner that protects $S$'s acoustic information. Our experiments show that relational synthesis integrates naturally with neural RAG and produces Foley audio that performs well on metrics measuring temporal agreement, acoustic fidelity, and leakage persistence, while maintaining distribution-level quality and text alignment.

Mon 5 OctSoundArtificial IntelligenceMachine Learning
The gist
Making new sound effects that match an original audio clip without copying parts directly is a tough problem for sound designers. The authors propose relational synthesis, which rearranges tiny pieces of one audio clip using the timing and loudness patterns of another clip, creating a fresh sound that stays true to the original source. Their method avoids copying large segments and blends well with AI audio generation, making it useful for realistic Foley sound production and retrieval-augmented generation tasks.
Open → 2610.05768v1

PLACE improves binaural audio generation from text video and audio prompts

PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation

Abstract: We present PLACE, a method that extends the pretrained any-to-audio model AudioX for binaural generation from arbitrary combinations of text, video, and optional audio prompts. PLACE augments video conditioning with Perception Encoder Core features, aligns text and video representations to derive spatial cues, and applies a conditioning-dependent low-rank transformation to the generated latent. The adapter is supervised via decoded-audio interaural level and time difference objectives. Trained on MRSAudio, PLACE improves most metrics over ViSAGe on FAIR-Play and achieves an improved SpatialCLAP score over SpatialSonic on the BEWO-1M Single Static test split. Listener evaluations favor PLACE for video-to-audio and out-of-distribution text-to-audio generation, demonstrating flexible multimodal control and improved spatial consistency.

Wed 30 SeptSoundMultimedia
The gist
Creating realistic 3D sound from text or video is tricky because it needs to simulate how we hear sounds around us. The authors introduced PLACE, a method that extends a previous model to better handle sound direction and space cues using text, video, and optional audio inputs together. They trained their model to pick up on subtle differences in sound timing and volume that our ears use to locate where sounds come from. Tests show PLACE produces clearer and more accurate spatial sounds than previous methods, and listeners prefer its results in many cases.
Open → 2610.00630v1

Model edits multi-instrument music mixes from MIDI control

Synthesis and editing of multi-instrument audio mixtures using scalar-quantised latents with MIDI Span conditioning

Abstract: Music creation often involves iterative refinement, changing selected musical details while retaining the rest. To support such refinement, we introduce SpanSynth-Edit, a flow-matching model for MIDI-guided synthesis and editing of multi-instrument audio mixtures using low-frame-rate scalar-quantised latents. MIDI Span encodes instrument-labelled note lifecycles as unordered event sets with continuous-valued attributes and pools each set into one conditioning vector per audio-latent frame. The model uses contextual audio for instrument-specific timbre guidance and supports editing by resynthesising the target region from revised MIDI. Experiments on single- and multi-instrument benchmarks show competitive performance and demonstrate within-frame onset control. We also discuss limitations of transcription-based note-adherence evaluation.

Tue 22 SeptSoundMachine Learning
The gist
Music often needs small changes without redoing everything. The authors created a model called SpanSynth-Edit that helps change parts of a multi-instrument music recording guided by MIDI notes. It breaks music into simple audio chunks linked to musical notes, letting it resynthesize only the parts you want to edit. Tests show it works well on music with one or many instruments and can control when notes start within short time frames. The authors also point out challenges in judging how well these edits match the original notes based on current methods.
Open → 2609.25546v1

Procedural audio pre training improves with source scaling and mask tuning

Rethinking Procedural Audio Pre-training: Source Scaling and Objective Adaptation

Abstract: Procedural audio has emerged as a viable source for transferable audio representation learning, but its design principles remain unclear.We revisit two questions: how a procedural source should be scaled, and whether training choices developed on natural audio should transfer unchanged to procedural data.Using a controlled source, we separate scale into formula-class coverage C and within-class rendering diversity I.Experiments with FDSL and AudioMAE show that these two forms of scale provide different benefits and depend on the learning formulation and downstream task. A matched AudioMAE study further shows that procedural audio favors low mask ratios (10%--25%), whereas AudioSet-28K favors 50%--75%. Shared-codebook analysis reveals lower patch diversity and stronger temporal predictability in procedural audio. These results motivate source-aware procedural pre-training, where source scaling and learning configuration are considered jointly.Code is available at https://github.com/Cross-Innovation-Lab/Formula-Bank.

Mon 14 SeptSoundArtificial Intelligence
The gist
Procedural audio, which is artificially generated sound, can help teach machines to understand audio better. The authors studied how to best create and use these sounds to teach machines by looking at two things: how varied the sound formulas are and how much variety there is in the sounds they produce. They found that different ways of making these sounds help in different learning tasks, and that settings used for regular audio don't always work well with procedural audio. Their work suggests adjusting training methods depending on the kind of audio source being used.
Open → 2609.15067v1

Neural networks synthesize controllable musical distortion tones in real time

Learned Continuous Synthesis of Quadratic Difference Tone Spectra

Abstract: Quadratic difference tones (QDTs) are a species of auditory distortion product in which a "phantom" pure tone, absent from the acoustic signal, is clearly audible to listeners. Exploiting this phenomenon, one can synthesize harmonically rich tones for musical purposes, a technique called Quadratic Difference Tone Spectrum (QDTS) synthesis. Previous works have introduced numerical methods to synthesize QDTS based on the distortion function, which links a target QDTS and an overtone-structured carrier signal. While accurate, these methods were stochastic and discontinuous, making them difficult to control for musical purposes and effectively limiting them to stationary signals. This paper proposes a neural network-based approach that learns an approximate inverse of the distortion mapping in an autoencoder-like configuration, producing a continuous approximation that addresses prior limitations. Experimental results show that, although slightly less numerically precise, the method is sufficient for perceptual and musical applications. We also implement a real-time version in Max and evaluate its performance. Various sound examples demonstrate its expressive and musical potential. The source code, audio examples, tutorials, and software accompanying this work are available at https://cordutie.github.io/projects/qdts.html

Wed 9 SeptSound
The gist
Sometimes, people hear extra tones called quadratic difference tones (QDTs) that aren't actually played but are created by the ear. These tones can make music sound richer and more interesting. The authors created a neural network that learns how to produce these tones more smoothly and continuously, overcoming problems in older methods that were harder to control and only worked on steady sounds. While this new method is slightly less precise in calculations, it sounds good enough for making music and runs in real time. They also made easy-to-use software so musicians can try it out.
Open → 2609.10913v1