PE-Field 4D: Video Generation Models as Canvas
Computer Vision and Pattern Recognition
Summary
The authors studied how video-generating AI models called diffusion transformers handle changes in camera angle and movement. They found that encoding positions based on where parts of the video should appear from a new viewpoint helps the model better match and control the video content. They created a new method that combines this positional encoding with a way to relate different video frames using depth and 2D projections, making it easier to edit videos with changing viewpoints while keeping the video's natural flow. Their approach improves control over video editing tasks like changing camera paths or creating new views.
Authors
Yunpeng Bai, Haoxiang Li, Qixing Huang
Abstract
Diffusion Transformers have recently achieved strong performance in video generation, yet controlling scene geometry under viewpoint changes and camera motion remains challenging. In this work, we revisit the role of positional encoding in video diffusion transformers and show that it provides a useful spatial bias for geometry-aware control. Specifically, if reference tokens are encoded according to their projected locations in the target view, the denoising model is encouraged to retrieve content from position aligned regions of the input video. Building on this observation, we introduce a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens derived from reference images or frames. To establish correspondence between the reference content and the target camera trajectory, we equip the context tokens with a projected positional encoding scheme that combines target-view 2D reprojection with depth-aware disambiguation. At the same time, we preserve the original spatiotemporal positional encoding of the generated video latent, allowing geometric guidance to be injected while maintaining consistency with the video model's native latent structure. The resulting framework provides a simple and effective approach for controllable video generation. It improves spatial controllability in viewpoint-dependent editing tasks, including camera re-trajectory, novel-view video synthesis, and geometry-aware video editing, while preserving the generative prior of the underlying video diffusion model. The code is available at: https://github.com/MTLab/PE-Field.