Dynamic sparse attention speeds joint video audio generation training
Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training
Computer Vision and Pattern Recognition
Summary
Training video and audio generation models at very high resolution is slow because standard attention methods look at every single piece of data, which gets expensive as data grows. The authors propose Prism, a new way to focus attention only on important parts by dividing the video and audio into zones that pay more attention to places with more change or stronger sound connections. This approach makes training faster and helps the model learn better by concentrating on what's meaningful. Prism improves speed by over twice while producing better results than traditional methods.
What this means in practice
- •For video generation engineers: Train high-resolution joint video-audio generation models more efficiently by focusing computational resources on content-rich regions to accelerate training without losing quality.
- •For audio-visual effects developers: Improve model responsiveness to sound-related visual details in media creation tools by dynamically adjusting attention zones based on audio-visual coupling strength.
Authors
Shuyuan Tu, Qi Tian, Yinming Huang, Yue Wu, Xintong Han, Kaihang Pan, Weijie Kong, Jiangfeng Xiong, Jian-Wei Zhang, Zuxuan Wu, Yu-Gang Jiang
Abstract
Natively training joint video-audio generation models at higher resolutions empowers them to learn richer visual details and sharper motion dynamics. However, full attention incurs quadratic cost and, as resolution increases, spreads attention over increasingly redundant tokens, diluting learning signals for informative content and disrupting pretrained priors. Existing sparse attention methods either target training-free acceleration or overlook the unique structure of joint video-audio data, where cross-modal interactions are inherently concentrated around sound-producing regions. To address this, we propose Prism, a dynamic sparse attention framework for natively training joint video-audio generation models at 2K. In particular, Prism organizes the token sequence into spatiotemporal macro-zones, enabling the attention structure to adapt to local content. For each zone, it estimates local information structure via video feature variance along the channel and feature norms from the audio-to-video cross-attention, jointly capturing how visual content varies directionally and how strongly audio influences each visual region. Based on these signals, Prism dynamically assigns a tailored block shape to each zone, applying finer partitioning along axes of rapid visual content variation and strong audio-visual coupling. This encourages tokens within each block to remain semantically coherent, allowing block-level features to capture both visual content and joint video-audio interaction patterns. Prism further adopts a hybrid block selection strategy to dynamically determine per-query sparsity. Experiments show that Prism achieves 2.5$\times$ training speedup compared to full attention, while surpassing it in generation quality.