MFlowAudio speeds up text to audio with efficient flow matching

MFlowAudio: Efficient Text-to-Audio Synthesis via Mamba-based Stateful Flow Matching

Sound

Summary

Generating long pieces of audio from text is usually slow because the common methods have to look at everything all at once. The authors introduce MFlowAudio, which uses a new way of processing audio that breaks down the task into smaller parts working together efficiently. This method reduces the computing power needed and allows audio to be generated steadily while listening, with delays kept very short. Their tests show this system is as good in sound quality as current leading methods but faster and more practical for long audio.

What this means in practice

  • For voice assistant developers: Create streaming audio responses from text prompts with low latency enabling smoother conversational AI voice interactions.
  • For game sound designers: Produce long-form background audio or dynamic soundscapes efficiently during gameplay without heavy processing overhead.

Authors

Hao Dai, Panyu Chen, Jagmohan Chauhan

Abstract

Recent advancements in audio generation have been largely driven by Transformer-based diffusion models. However, these models suffer from quadratic complexity of self-attention, severely bottlenecking the long-form audio synthesis. To overcome this limitation, we propose MFlowAudio, a novel latent audio generation framework that synergizes the continuous-time dynamics of Flow Matching with a custom-designed TCMamba backbone. TCMamba uses an innovative dual-scan mechanism: a TimeMamba module to capture long-range causal dependencies with linear complexity, and a ChannelMamba module to model cross-channel correlations such as harmonic structures. Exploiting this structural foundation, we formulate the Stateful Flow Matching (SFM) paradigm. This framework inherently enables chunk-wise training and streaming generation, maintaining an $\mathcal{O}(1)$ caching complexity without incurring extra computational overhead. To enable fine-grained controllable synthesis, we devise a novel guidance mechanism that neutralizes vector field collisions precipitated by on-the-fly semantic transitions of prompts. Comprehensive empirical evaluations confirm that MFlowAudio yields generative fidelity on par with state-of-the-art baselines while establishing significant computational efficiency, achieving above $14.5$% acceleration in generation and facilitating perceptually seamless streaming synthesis with a constant latency of $1.8$ seconds. Demo: https://huggingface.co/spaces/mflowaudio/MFlowAudio