Affine alignment improves gaussian video representation quality for camera motion
Affine-Aligned Atlas for Canonical Gaussian Construction in Video Representation
Computer Vision and Pattern Recognition
Summary
Videos with lots of camera movement are hard to represent efficiently because frames get misaligned, making the computer work harder to match them. The authors propose shifting video frames first using a type of transformation called affine alignment, so the main video model can represent frames more accurately without extra complexity. This new approach fits easily into existing models and helps reconstruct videos better when there’s a lot of motion. Overall, it makes video representation clearer and faster for tricky videos.
What this means in practice
- •For video compression engineers: Improve video compression quality for videos with significant camera motion by integrating affine-aligned Gaussian modeling.
- •For augmented reality developers: Enhance real-time video rendering in AR devices by reducing misalignment effects caused by camera movement.
Authors
Masaya Takabe, Hiroshi Watanabe, Sujun Hong, Tomohiro Ikai
Abstract
Gaussian splatting has recently emerged as an efficient representation for images and videos due to its explicit structure and fast rendering capability. Existing Gaussian-based video representations often decompose a video into canonical Gaussians and temporal deformation. However, when a video contains large global motion such as camera movement, the canonical representation may become misaligned with individual frames, increasing the burden on the temporal deformation model. In this paper, we propose an affine-atlas canonical Gaussian representation, which constructs canonical Gaussians in a larger affine-aligned atlas space. Frame-wise affine transforms absorb global motion before canonical Gaussian construction, reducing the gap between the canonical representation and target frames. Since the proposed method only modifies the canonical construction stage, it can be integrated into existing canonical-Gaussian-based methods with negligible additional parameter cost. Experiments show that our method improves reconstruction quality especially for sequences with large camera motion.