Text and keyframe inputs create smooth camera motion for video scenes
TKCAM: Text and Keyframe to Camera Trajectory Generation
Computer Vision and Pattern Recognition
Summary
Creating smooth and realistic camera movements is important for making videos and understanding 3D scenes. The authors developed TKCAM, a system that takes written descriptions and pictures at specific times to generate natural camera paths between those points. Their method breaks down camera movement into parts like position and rotation, then uses a two-step transformer model to learn and improve these movements. They also created a new dataset and benchmark to test their system and showed it works better than previous methods.
What this means in practice
- •For video production teams: Generate smooth, controllable camera paths using text and key visual frames for enhanced AI-assisted cinematography.
- •For augmented reality developers: Create realistic camera motions within 3D scenes by integrating text descriptions and visual keyframes to guide virtual camera trajectories.
Authors
Haozhe Yang, Zhiyang Dou, Zekai Gu, Cheng Lin, Wenping Wang, Yuan Liu, Taku Komura
Abstract
Generating high-quality and controllable camera motion is essential for AI-assisted cinematography, video synthesis, and 3D scene understanding. We introduce TKCAM, a Text- and Keyframe-conditioned CAMera-motion synthesis framework based on generative masked modeling. We represent camera dynamics using a 12-dimensional kinematic feature comprising position, velocity, and a continuous rotation representation and discretize them into hierarchical motion tokens via a Residual Vector Quantizer (RVQ). A two-stage masked transformer architecture then learns to reconstruct and refine these tokens, utilizing explicit self- and cross-attention modules for multimodal conditioning. A central feature of our framework is sparse visual keyframe conditioning: users can provide free-form text prompts together with RGB observations at selected timestamps, which provide temporally localized visual guidance for generating coherent in-between trajectories. Furthermore, to advance evaluation standards, we curate RealEstate10K-Cap, a large-scale text-camera dataset, and establish a cross-domain benchmark with a Universal CLaTr Evaluator. Extensive experiments demonstrate that TKCAM surpasses recent state-of-the-art baselines on Fréchet distance (FID), text-motion matching scores, and retrieval metrics (R@K), while additional analyses evaluate temporal smoothness and cross-domain generalization. Code is available at https://github.com/linearalgebrayhz/TKCAM.