SplatFlux enables fast 4D scene creation and video-based generation
SplatFlux: Feed-Forward 4D Gaussian Reconstruction and Generation from Video Latents
Computer Vision and Pattern Recognition
Summary
Creating 4D pictures that capture changes over time, like a moving scene, is usually slow and complicated. The authors show a new way called SplatFlux that quickly makes these 4D scenes from videos or other video-like data. It separates still parts of the scene from moving parts to build the full picture more efficiently. They also developed a learning method to teach the system about motion without needing expensive labeled data. Their tests show this method works well for both reconstructing and creating 4D content.
What this means in practice
- •For augmented reality developers: Generate detailed 4D models of real-world scenes from video input to improve AR experiences with dynamic environments.
- •For visual effects studios: Create complex dynamic 4D scenes quickly from existing video footage for enhanced visual effects production workflows.
Authors
Runmin Zhang, Binglin Ren, Zhu Yu, Fudong Wang, Jiangke Lin, Yichun Shentu, Si-Yuan Cao, Yi Yuan, Hui-Liang Shen
Abstract
This work presents SplatFlux, a feed-forward latent-to-4DGS framework for both video-based 4D reconstruction and conditional 4D generation. Our key observation is that observed videos and conditionally generated content can be represented in the same pretrained video latent space, allowing both tasks to be formulated as latent-to-4DGS decoding. To construct compact and temporally coherent 4D scenes, SplatFlux introduces a static-dynamic decoupled 4D Gaussian representation that explicitly predicts dynamic probabilities for Gaussian primitives. This representation separates static scene structures from moving regions, enabling complete 4D scene construction in a feed-forward manner through static aggregation and dynamic motion interpolation. To learn static-dynamic decomposition and dynamic Gaussian motion without costly annotations, we further develop an auxiliary distillation strategy that provides pseudo dynamic masks and optical-flow cues. Extensive experiments across 3D and 4D reconstruction and generation demonstrate that SplatFlux achieves state-of-the-art performance.