DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionSound
AI summaryⓘ
The authors developed DreamX-Creator 1.0, a system that generates videos and matching audio together, rather than separately. It starts with a single video frame and a text description, then creates audio and video streams that interact through special attention mechanisms. Their training uses a multi-stage process including reinforcement learning that helps the audio and video improve together. They also designed a method to make high-resolution outputs efficiently with fewer computation steps. Overall, their approach produces synchronized audio-video content with quality competitive to existing open-source models and is made accessible for future research.
audio-video generationcross-modal attentiondenoisingreinforcement learningautoregressive refinementmultimodal feedbacktext-to-video synthesishigh-resolution generation
Authors
Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
Abstract
Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.