Sparse 4d autoencoder compresses dynamic scenes for novel view synthesis
A Compact Explicit 4D Representation for Dynamic Scenes
Computer Vision and Pattern Recognition
Summary
Capturing what changes in a moving scene and how it looks from new angles is challenging and can require lots of data. The authors present Sparc4D, a method that compresses video of dynamic scenes into a compact, sparse 4D representation. This approach separates static parts seen throughout the video from moving features that change over time, reducing storage while keeping image quality high. Their method also stores some texture pixels to preserve details and works well on videos with multiple camera views without extra training.
What this means in practice
- •For 3d graphics developers: Create efficient 3D models of moving scenes from monocular videos that support realistic viewpoint changes with reduced storage needs.
- •For augmented reality teams: Integrate time-varying scene data compressed into sparse slots for dynamic object rendering in AR experiences without expensive recalibration or retraining.
Authors
Di Yang, Zhihao Li, Yanhai Xiong, Yufei Wang
Abstract
A compact dynamic-scene representation must retain both the surfaces seen over time and the appearance needed to render them from new viewpoints. We present Sparc4D, a feed-forward autoencoder that encodes a monocular video with known cameras into a sparse 4D scene state. Static features are shared across the clip, while spatially anchored temporal slots compress time-varying features. A sparse decoder produces 2D Gaussian surfels, while stored source pixels preserve fine texture through geometric re-projection. The state includes one full source frame and dynamic-region pixels sampled every fourth frame, alongside learned features and sparse occupancy. For a 32-frame MultiCamVideo clip, it averages 0.95M 32-bit-equivalent values on random windows and 0.92M on the first-32 protocol. On first-32, Sparc4D reaches 21.70\,dB, compared with 20.40\,dB for MoVieS. On randomly placed windows, their PSNR scores are comparable. With stored texture disabled, temporal slots compress the time-varying feature state by a median $4.0\times$ and reduce the mean state from 1.04M to 0.42M values, with essentially unchanged target-view reconstruction quality. Without fine-tuning on real data, Sparc4D transfers to DyCheck and Neu3D, where stored texture improves LPIPS while slightly reducing PSNR.