Multi shot videos controlled precisely in 3D space for storytelling
Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating stories with videos often needs multiple video clips that look consistent and fit well together. The authors found that current methods can keep video clips consistent but can’t control details well. They made a new system called Mise-en-Video that lets creators arrange scenes in 3D and then generate videos matching those scenes with good control over each shot. They also made a big dataset to train and test the system. Tests show their method makes better, more consistent, and more controllable multi-shot videos than other approaches.
What this means in practice
- •For video production teams: Generate multi-scene videos with precise 3D scene layouts for consistent storytelling across shots.
- •For virtual event organizers: Create staged multi-shot promotional videos with fine-grained spatial control to showcase products or venues.
Authors
Yingshuang Zou, Xiaoyu Shi, Yawen Luo, Ziang Li, Qinghe Wang, Mengfei Li, Chengfeng Zhao, Xintao Wang, Pengfei Wan, Kun Gai, Yuan Liu
Abstract
Recent advances in text-to-video generation have enabled high-fidelity single-shot synthesis, yet practical storytelling creation requires multi-shot videos with both strong inter-shot consistency and fine-grained controllability. Existing multi-shot video generation approaches achieve strong consistency but still fall short in controllability. In this paper, we propose Mise-en-Video (MeV), a highly controllable multi-shot video generation framework that allows video creators to conduct staging the multi-shot in the 3D space and then generate the corresponding multi-shot video. To achieve this, our framework combines a hierarchical attention mechanism for multi-level narrative consistency with 3D-aware layout guidance for precise, per-shot spatial control. To support training and evaluation, we develop standardized annotation pipelines and construct a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes. Extensive experiments demonstrate that MeV consistently outperforms strong baselines in video quality, inter-shot consistency, and subject-level controllability.