Video generation controlled precisely by all camera settings
Gen Through the Lens: Camera-Controllable Video Generation via Cross-Coordinate Conditioning Fusion
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphicsMachine Learning
Summary
Making videos from a single image usually means guessing how the scene moves or changes. This paper shows a way to create videos that let you control the camera movement, focus, and framing more realistically and accurately. The authors combine three different ways to describe camera and object positions so the video looks natural and consistent over time. Their approach works better than older methods and lets users change camera angles and focus with more ease.
What this means in practice
- •For video production teams: Create realistic videos from images with precise camera movements and framing controls for film and animation projects.$Commercial implications: Enables software products that generate controllable, photorealistic video content for filmmakers and content creators.
- •For virtual reality developers: Generate consistent video scenes that adapt to user-controlled camera changes in VR environments using real-world geometry.
Authors
Shuai Li, Jinxu Xiang, Xinyuan Li, Hanlei Guo, Xiyang Tan, Alan Zhao, Xiaotong Zhao, Ran Zhang
Abstract
We present Gen Through the Lens, a camera-controllable image-to-video generation framework that synthesizes videos from a given reference image while enabling realistic and physically consistent camera operations. Unlike previous models that treat camera motion as abstract conditioning or rely on a single coordinate system, our approach explicitly decomposes and fuses control signals across three coordinate spaces: the camera coordinate (modeled by intrinsic parameters), the world coordinate (represented by spatial geometry hints), and the object coordinate (described by 3D bounding boxes of target subjects). Through cross-coordinate conditioning fusion, these heterogeneous cues are unified to guide the generative process, enabling precise control of camera motion, focus, and framing while preserving temporal and spatial coherence. Extensive experiments demonstrate that our method produces controllable and visually consistent results across diverse camera operations, outperforming existing approaches in both realism and editability.