Spatial audio generation and editing with improved 3D sound control
Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion
SoundArtificial Intelligence
Summary
Creating and editing spatial audio, which places sounds in 3D space, is tricky because it requires coordinating what sounds happen, where, and when. The authors offer a new system called AMBIT that uses detailed executable plans to specify the sound sources and their movements. Their approach also uses a special machine learning model to keep the complex audio details and spatial relationships between sound channels accurate. Finally, they improve the system by training it to better predict and correct its outputs during generation. This leads to clearer, more precise 3D sounds and better editing of existing spatial audio.
What this means in practice
- •For audio software developers: Implementing advanced spatial audio editing features that tightly coordinate source position, activity, and acoustic detail in 3D sound environments.
- •For game audio engineers: Generating realistic 3D audio scenes with precise control over sound sources and trajectories to enhance immersive gaming experiences.
Authors
Kam Man Wu, Yu Zhang, Pengjun Fang, Mingfei Chen, Weijia Chen, Zeyue Tian, Qifeng Chen
Abstract
Spatial audio authoring coordinates source decisions with soundfield synthesis. For editing, plans specify what to change and retain, while reference audio supplies further acoustic detail. We present AMBIT, a unified framework for native First-Order Ambisonics (FOA) generation and editing with three connected contributions. First, a shared Transfusion backbone couples autoregressive planning with rectified-flow rendering through executable ScenePlans that specify source content, activity, and 3D trajectories. For editing, the plan describes the full target scene, including retained sources; contrastive features ground the requested change, while full reference latents supply residual acoustic detail for preservation. Second, an FOA-aware continuous VAE combines shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to preserve content and inter-channel structure. Third, joint Transfusion on-policy self-distillation (OPSD) refines planning on student-generated prefixes and rendering at sampling states reached under student plans, with target-conditioned teachers supervising both modes of the shared student. Evaluations across speech, music, and sound show improved generation content and spatial quality, acoustic fidelity during instruction-guided editing, and unedited-source preservation over the compared baselines. Project page: https://wjm-george.github.io/AMBIT/