Semantic camera control improves subject tracking in existing videos
SemCam: Semantic Camera Motion Control for Video Generation
Computer Vision and Pattern Recognition
Summary
It can be hard to keep a camera focused on a moving person in a video, especially when you want the camera to keep looking at them from specific angles. The authors created a new way to control the camera based on the subject’s movement and desired viewing style, without needing exact paths for the camera to follow. Their method, called SemCam, can change how the camera moves relative to the person while keeping the original video’s appearance. They tested their system on hundreds of videos and found it better at keeping the camera focused correctly compared to earlier methods.
What this means in practice
- •For video production teams: Automatically adjust camera angles relative to moving subjects in existing footage to improve framing and storytelling in post-production.
- •For virtual reality developers: Generate viewpoint changes following user or character movements without predefining explicit camera trajectories to enhance immersive experiences.
Authors
Janna Bruner, Omer Talmi, Ianir Ideses, Lior Fritz, Lior Wolf, Sagie Benaim
Abstract
Controlling the camera relative to a moving subject in an existing video is challenging: behaviors such as maintaining a frontal view require the camera to adapt to the subject's changing position and orientation, making the desired trajectory difficult to specify in advance. Existing camera-controlled video-to-video methods typically rely on explicit trajectories or reference motions, which do not directly express these dynamic camera--subject relationships. We introduce semantic camera motion control, a novel video-to-video task in which a reference video and a target motion label specify the desired subject-relative camera behavior without an explicit target trajectory. Our method, SemCam, learns to realize this behavior while preserving source content. It combines shared-basis low-rank adaptation with motion-conditioned modulation, while a background-consistency loss encourages fidelity in regions visible in both reference and target videos. We construct 661 paired videos covering eight semantic camera behaviors and evaluate on a separate 109-scene benchmark using subject-relative motion metrics, appearance measures, and a user study. SemCam achieves a semantic-motion success rate of 68.6%, compared with 45.3% for Vista4D, the strongest evaluated baseline, while maintaining comparable subject identity preservation.