4Director improves control of video scenes using 3D object geometry
4Director: Controlling Video World Models with Rigid 3D Geometry
Computer Vision and Pattern Recognition
Summary
Controlling how cameras and objects move in videos is tricky because existing methods either only work roughly or lose important 3D details. The authors created 4Director, which builds a detailed 3D model of each object from an input image and moves it precisely using fixed 3D transformations. This way, the video remains consistent from different viewpoints and the shapes don’t change unexpectedly. They also made a special system to turn this 3D info into high-quality videos and a new dataset to train and test their approach. Their experiments show their method controls objects and cameras better than earlier techniques.
What this means in practice
- •For video production teams: Control camera and object movements precisely in video editing using a 3D mesh-based interface that maintains scene consistency.$Commercial implications: Enables advanced video production tools with precise 3D manipulation and consistent scene rendering for filmmakers and editors.
- •For game developers: Create realistic cutscenes by applying rigid 3D transformations to object meshes, ensuring stable appearance across camera angles.
Authors
Wei Cao, Hao Zhang, Vikram Voleti, Yuqun Wu, Mallikarjun B R, Shimon Vainer, Mark Boss, Yaoyao Liu
Abstract
Precise control over camera and object motion is essential for professional video production. Existing methods control objects only coarsely, through image-plane cues that are ambiguous in depth and rotation or through 3D tracks and blobs that lack complete geometry and lose consistency across viewpoint changes. We introduce 4Director, a video world model conditioned on an explicit 4D scene representation: each object is reconstructed once from the input image as a canonical mesh and moved by one prescribed rigid transformation per frame. This representation provides an intuitive 3D control interface and prevents unobserved geometry from being regenerated independently in every frame. We render the controlled scene as a depth video and introduce a Motion Adapter that transforms this geometric scaffold into video while synthesizing view-consistent appearance, illumination, and non-rigid dynamics. For training, we construct RealCOD-Rigid, a new dataset of 20,774 clips annotated with rigid 3D scenes by our automatic pipeline. We further introduce Identity-Gated IoU (IG-IoU), which jointly evaluates adherence to prescribed object motion and preservation of object identity. Experiments demonstrate that 4Director consistently outperforms prior methods in visual quality and in camera and object control.