FilmFlow creates detailed camera moves from words or videos in 3D scenes
FilmFlow: Agentic System for 3D Virtual Film Cinematography
Computer Vision and Pattern Recognition
Summary
Filmmakers often need complex camera moves to tell a story visually, but making those moves in 3D scenes can be hard. The authors developed FilmFlow, a system that turns simple instructions or example videos into exact camera movements inside 3D virtual spaces. It uses separate smart parts to understand directions, pick starting views, make camera plans, and check the results. They also made a big set of 3D scenes and tasks to test how well it works. Their system performed better and was preferred by users compared to other methods.
What this means in practice
- •For virtual reality developers: Generate realistic and complex camera movements in virtual 3D environments based on simple user instructions or example footage.
- •For game designers: Create dynamic in-game cinematic camera shots that follow precise storytelling directions without manual programming for each scene.
Authors
Yiming Zhang, Jiarui Guo, Haojia Wei, Yifei Liu, Yifan Yang, Hongjie Zhang, Xue Yang, Zhihang Zhong
Abstract
Complex camera motion is central to visual storytelling, yet translating cinematographic intent into controllable, executable camera trajectories remains challenging. We introduce FilmFlow, an agentic framework that converts natural-language instructions or reference videos into explicit camera trajectories for dynamic 3D environments. To generate these trajectories, a Director agent constructs executable camera plans by composing high-level motions from concurrent atomic operations that jointly control translation, rotation, and framing. The Director is supported by three specialized agents: an Interpreter that extracts scene-grounded cinematographic intent, a Scout that selects initial viewpoints, and a Critic that evaluates rendered results and provides visual feedback for iterative plan refinement. We also introduce CinemaBench, comprising 150 Blender scenes from self-created and licensed sources, with five camera instructions per scene for 750 text-directed tasks and 60 reference videos associated with a subset of scenes. CinemaBench supports evaluation of both text-directed camera generation and video reference-directed camera generation. Across a diverse set of evaluated baselines, FilmFlow achieves the strongest benchmark performance and the highest user preference, demonstrating effective spatial reasoning, temporal coordination, and visual composition for executable 3D cinematography.