Photorealistic video creation controlled by simple 3D shapes on grid
CoaG: Cylinders on a Grid: Coarse 3D Layout Control for Video Generation
Computer Vision and Pattern Recognition
Summary
Creating videos where you control where people stand and where the camera moves usually requires complicated design. This work shows you only need to draw a simple grid on the ground and put one cylinder shape for each person. By moving these cylinders and the camera along a timeline, the system generates a realistic video where people follow the cylinder positions and camera view. The people’s appearance comes from text descriptions and background images, while the system learns from automatically created example pairs without any manual labeling.
What this means in practice
- •For video game developers: Generate controlled in-game character and camera movement videos simply by placing cylinder-shaped markers on a grid to prototype scenes rapidly.
- •For advertising teams: Easily create storyboard-like videos with controlled people placement and camera angles using minimal geometry and text prompts for product promotion scenes.$Commercial implications: Enables agencies to produce customizable marketing videos with precise spatial and camera control, streamlining creative workflows and reducing production costs.
Authors
Zhangsihao Yang, Mengyi Shan
Abstract
We ask how little geometry a person has to draw to control both where people stand and where the camera moves in a generated video. Our answer is a ground plane and one cylinder per person. A user draws a grid on the ground, places one cylinder where each person should stand, moves the cylinders and the camera over 81 frames, and the model renders a photoreal video in which the people occupy the cylinders' positions, move as the cylinders move, and are seen from the drawn camera. Appearance comes from a text prompt and a background reference image; layout and motion come from the geometry. Because no dataset pairs such a signal with video, we build the pairs ourselves: an automatic engine writes 2000 captions from a combinatorial seed, generates a clip for each with a text-to-video model, and lifts every clip back to its geometry with person tracking, background inpainting, an agentic ground-mask loop, feed-forward multi-view reconstruction and a plane fit, with no real footage and no manual labels. A LoRA on Wan2.2-Fun-Control trained on 1935 such tuples follows drawn layouts and camera paths on hold-out clips: the generated people match the cylinders' count, order, position and height, the text changes who they are, the reference image changes where they are, and dolly-in, orbit, pan and crane paths are followed, dolly-out only weakly.