Feed-forward latent model improves dynamic scene rendering across views and time
FL4N: Feed-forward Latent Model for 4D Novel View Synthesis
Computer Vision and Pattern Recognition
Summary
Creating videos of moving scenes from new angles and time points is hard because you need to understand 3D shapes and how they change over time. The authors developed a method called FL4N that uses a single hidden representation to capture the whole 3D scene and its changes, instead of relying on explicit 3D shapes. This method learns how things move as small changes on top of a stable background, which helps it work well even with little training data. It works fast enough to render scenes in real time and can handle real-world videos while separating viewpoint changes from object motion.
What this means in practice
- •For video game developers: Render dynamic game environments in real time from new viewpoints and moments smoothly using learned latent 4D scene representations.$Commercial implications: Enables selling real-time dynamic scene rendering technology that improves game graphics and performance.
- •For film visual effects teams: Generate dynamic scene views from monocular footage efficiently for special effects and scene reconstruction workflows.
Authors
Eunbeen Hong, Hyeonseo Yu, Honggyu An, Jooho Son, Hyunah Ko, Soowon Son, Chaehyun Kim, Jahyeok Koo, Jisoo Kim, Junhwa Hur, Seungryong Kim
Abstract
4D novel view synthesis (NVS) aims to render dynamic scenes across viewpoints and timestamps, which requires jointly inferring 3D structure and temporal dynamics. Existing monocular feed-forward methods often rely on explicit 4D Gaussian representations, which are prone to temporal inconsistency and severe rendering artifacts. We present FL4N, the first feed-forward latent 4D NVS model. Instead of relying on explicit 4D primitives, FL4N encodes a 4D scene prior into a unified latent representation across space and time. Our core insight is that learning temporal dynamics as residuals on top of a pretrained static latent prior enables effective 4D modeling with limited dynamic training data. To this end, we introduce a lightweight dynamic encoder that leverages static scene features to disentangle object motion from camera-induced parallax. Trained solely with an image synthesis objective, FL4N achieves state-of-the-art 4D NVS quality while rendering in real time. Its learned latent representation generalizes to real-world videos and disentangles viewpoint from time.