OmniCam generates 3D camera paths guided by scene and text
OmniCam: Omni-Camera Trajectory Generation via Geometry-Grounded Pose Token Learning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating camera movements that follow a specific path described by text is difficult because it requires understanding both the scene’s 3D shape and the target viewpoint. The authors present OmniCam, a model that takes a single panorama image and a text description to generate a smooth camera path in 3D space. They train and test their model on a large new dataset of camera trajectories and show it reduces errors and collisions compared to past methods. This advance can help with video creation and enabling robots to navigate and observe scenes better.
What this means in practice
- •For video production teams: Generate smooth and accurate camera motion paths from text and panoramic images to automate virtual camera control in video creation.
- •For robotics developers: Enable robots to plan collision-free camera trajectories that align with task-specific viewpoints based on scene geometry and textual goals.
Authors
Zhenyang Liu, Chenjie Cao, Yisu Zhang, Xuhui Zuo, Xiangyang Xue, Yanwei Fu, Tengfei Wang, Chunchao Guo
Abstract
Camera trajectories control viewpoint changes in video generation, scene reconstruction, and robotic perception. Generating them from language requires both scene geometry and target-aware framing. We introduce OmniCam, an autoregressive model that generates camera pose sequences from a single panorama and textual trajectory descriptions. Its geometry-grounded pose token learning combines three components: a panoramic point-cloud encoder for omnidirectional geometric context; hybrid absolute-rotation and relative-translation tokenization with temporally consistent quaternion signs; and separate geometric and semantic conditioning streams with an explicit 3D target anchor. We also construct OmniCaT, containing 267,700 trajectories across four camera behaviors. On the reported OmniCaT evaluation, OmniCam reduces trajectory errors by 28--47% and collision rate by 65.8% relative to GenDoP retrained on OmniCaT. Against the best baseline for each metric, the ATE and collision reductions are 43.0% and 62.3%, respectively. Component ablations support the use of geometric and target-aware conditioning, while downstream experiments examine camera-controlled video generation and robotic active perception.