Structured scripts improve multi-shot audio visual video generation
Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generation
Computer Vision and Pattern Recognition
Summary
Making videos that show multiple scenes with matching sound and consistent characters is hard because descriptions usually focus on each scene without explaining how characters or sounds carry over. The authors propose a special way to write detailed scripts that keep track of who is who, how they change, and what makes the sounds across scenes. They also created a big dataset of videos with these detailed scripts to help machines learn and keep video and sound consistent across different shots. Their tests show better results in keeping characters the same, showing changes clearly, and syncing sounds with actions.
What this means in practice
- •For video content creators: Create multi-scene videos with consistent characters and coherent sound across shots using structured script guidance.
- •For game developers: Generate character-driven scenes with coordinated audio and visual elements that maintain identity and event continuity through multiple shots.
Authors
Chen Zhao, Lehan Shen, Qilong Huang, Hongyu Li, Zhuoliang Kang, Xiaoming Wei, Jian Yang, Ying Tai
Abstract
Coherent multi-shot audio-visual video generation requires maintaining consistent identities while depicting evolving states and coordinating sounds across shot boundaries. Although dense captions describe individual shots in detail, they rarely specify which entities persist, how their states change, or which entities and actions produce sounds. To address this limitation, we propose Structured Stateful Relational Scripts (SSRS), a textual supervision representation that separates persistent entity and voice identities from shot-specific visual and auditory states. SSRS explicitly encodes sound-source bindings and temporal relations, while a global audio-visual event sequence connects local actions and sounds, including events that continue across shot boundaries. We further introduce MSAV-Script-100K, a large-scale dataset of curated multi-shot audio-visual videos paired with detailed SSRS annotations, enabling models to learn identity persistence, state transitions, and event continuity from structured supervision. Experiments demonstrate improvements in cross-shot consistency, adherence to specified state changes, and audio-visual alignment, highlighting the effectiveness of our SSRS and MSAV-Script-100K for coherent multi-shot audio-visual generation. Our proposed dataset is available at \href{https://huggingface.co/datasets/shenlehan/MSAV-Script-100K}{MSAV-Script-100K}.