OmniOPD improves joint audio and video generation stability
OmniOPD: Stable On-Policy Distillation for Joint Audio-Video Generation
Computer Vision and Pattern RecognitionArtificial IntelligenceSound
Summary
Producing videos with matching sound is tricky because the computer must learn from both visuals and audio, which can interfere with each other. The authors found that teaching the computer using separate specialized models for audio and video helps, but audio signals often get weaker during this combined learning. To fix this, they developed OmniOPD, which uses cleaner video information and smarter sampling techniques so the audio model learns better alongside video. This results in more stable and balanced creations of videos with good audio and visuals together.
What this means in practice
- •For multimedia software developers: Enhance multimedia generation tools to produce better-synchronized and higher-quality audio-video content by integrating stable joint audio-video training methods.
- •For interactive media creators: Create engaging audio-video experiences with improved synchronization and generation quality leveraging stable on-policy distillation techniques.
Authors
Zihao Fan, Xin Lu, Jie Huang, Hexin Zhang, Xueyang Fu, Zheng-Jun Zha
Abstract
Reinforcement learning (RL) elevates joint audio-video generation, but optimizing heterogeneous, modality-specific audio and video rewards often introduces cross-modal interference. On-policy distillation (OPD) alleviates this issue by transferring the capabilities of specialized audio and video RL teachers into a single student model. However, a significant imbalance in cross-modal token counts leaves audio with sparse supervision, causing unstable distillation. We observe that naively increasing the audio loss weight fails to solve this problem since audio learning relies heavily on visual context, indicating richer visual cues are crucial to offset sparse audio signals. To address this, we propose OmniOPD, a modality-wise on-policy distillation framework designed for balanced and stable audio-video generation. First, we construct privileged visual information using asymmetric noise scheduling, allowing the audio teacher to generate more informative targets from cleaner video context. Second, we introduce grouped multi-noise sampling to expand audio supervision coverage and stabilize training. Third, we introduce timestep-wise importance weighting based on teacher-student transition divergence, preserving reliable supervision while down-weighting unreliable learning signals, to stabilize fluctuations in audio learning. Evaluated on JavisBench and VBench demonstrate OmniOPD preserves audio quality while significantly enhancing video generation, achieving effective joint distillation.