SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation
2026-08-10 • Sound
Sound
AI summaryⓘ
The authors present SonicWeave, a new model that can create audio scenes containing speech, music, sound effects, and singing all together from text descriptions. Their method uses a special routing system that decides how to process chunks of sound based on both overall information from the text and specific details from the audio as it is being generated. This approach helps the model handle different audio types and overlaps more smoothly than previous methods. Tests show SonicWeave performs better on various audio generation tasks and creates more coherent mixed audio scenes.
text-to-audio generationmixture-of-experts (MoE)diffusion modelsaudio scene synthesisrouting mechanismconditional computationflow matchingacoustic signal processingcompositional audiospeech and music synthesis
Authors
Yunrui Cai, Xu Li, Yucheng Zhou, Jinchao Li, Dingdong Wang, Dongchao Yang, Xixin Wu, Chen Zhang, Zhiyong Wu, Pengfei Wan, Helen Meng
Abstract
Text-conditioned general audio generation is moving beyond isolated speech, music, and sound-effect synthesis toward a single model that can compose them into controllable, coherent audio scenes. This unified setting is particularly challenging: heterogeneous components impose conflicting structural requirements on a shared backbone, while a complex mixed scene may contain locally distinct or overlapping content that demands fine-grained adaptation within the same clip. Existing audio mixture-of-experts (MoEs) mainly route at the domain level, while token-wise routing overlooks the local continuity inherent to acoustic signals. We propose SonicWeave, a flow-matching model for unified audio scene generation. At its core is a chunk-routed MoE with a conflict-gated prior-evidence routing mechanism (CPE-MoE). CPE-MoE routes contiguous acoustic chunks by combining a global prior that encodes the structured text condition and diffusion phase with local evidence from the evolving acoustic state. A learned conflict gate favors the prior when local states are unreliable, while allowing local evidence to influence routing when a region departs from the global scene context. SonicWeave supports speech, music, sound effects, singing, and their fine-grained mixtures with a single set of weights. Across TTS, TTA, and TTM benchmarks, SonicWeave consistently improves over controlled Dense and Base-MoE baselines. Complex-scene evaluation further demonstrates improved compositional quality, while routing analyses reveal content-dependent expert specialization across diffusion phases. These results suggest that temporally coherent, prior-evidence routing is an effective conditional-computation strategy for unified audio generation. Project page: https://caiyunrui.github.io/SonicWeave.