Spatial transfusion unifies spatial audio understanding generation and editing
Spatial Transfusion: From Structured Spatial Scene Understanding to Generation and Editing
SoundArtificial Intelligence
Summary
Sounds in the environment come from different places and interact with objects around them, but current technology treats understanding, creating, and changing these spatial sounds separately. The authors developed Spatial Transfusion, a system that builds on existing models to handle all three tasks together for 3D audio scenes. It keeps the original understanding skills intact while adding abilities to generate and edit sounds at precise locations using both structured commands and natural language. This allows for more flexible and controllable spatial audio experiences.
What this means in practice
- •For spatial audio engineers: Generate and edit 3D audio scenes controllably using both structured commands and natural language interfaces.
- •For video game developers: Integrate spatial audio generation and interactive editing to create more immersive sound environments in games.$Commercial implications: Enables advanced spatial audio features in games, enhancing player experience, and creating marketable immersive soundscapes.
Authors
Jaeyeon Kim, Sebastian Braun, Hannes Gamper
Abstract
Spatial audio conveys not only what sounds are present, but where they originate and how they interact with the environment. Recent audio-language models increasingly understand and reason about such scenes, yet spatial audio understanding, generation, and editing remain largely separate. We present Spatial Transfusion, a unified first-order ambisonics (FOA) framework that extends a pretrained spatial audio understanding model to generation and editing through flow matching over VAE latents. We formulate editing as a structured scene transformation over a taxonomy of atomic edits. Keeping the pretrained backbone frozen, we introduce modality-based LoRA routing and a curriculum that progressively adds generation, editing, and natural-language interaction. Experiments show that Spatial Transfusion largely preserves spatial understanding while outperforming full fine-tuning in generation and editing. The resulting model supports controllable FOA generation and editing through both structured and natural-language interfaces. Audio samples are available at https://microsoft.github.io/spatial-transfusion/