Online Neural Space Time Memory for Dynamic Novel View Synthesis
2026-07-16 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionGraphicsMachine Learning
AI summaryⓘ
The authors address a challenge in creating new video views in real time from multiple camera streams, especially when parts of the scene are temporarily hidden. Normally, methods update their memory of the scene every frame, which is slow and unstable for real-time use. Instead, the authors suggest updating memory less often but still using it every frame to keep up with changes. They introduce special techniques to help the system remember scenes consistently over time and avoid losing important information. Their method works well for videos with moving people and can remember scenes for several minutes while running in real time.
novel view synthesismulti-view streamingtest-time trainingreal-time processingmemory updatescross-view attentiondynamic scenesmemory cachingcatastrophic driftvideo occlusion
Authors
Baback Elmieh, Lynn Tsai, Zeman Li, Srinivas Kaza, Tiancheng Sun, Gabor Csapo, Ali Behrouz, Yuan Deng, Stephen Lombardi, Steven M. Seitz, Xuan Luo
Abstract
Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints. While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts. Given that memory updates are more demanding than memory application and video content is largely redundant, we propose to decouple the frequencies of these two processes. Our approach performs periodic memory updates while applying the memory on a per-frame basis, using cross-view attention to manage deformations between the prior memory state and the current frame. To lock in the historical context, we introduce two critical mechanisms: an auxiliary Memory Loss that forces persistent internalization of the scene, and a Memory Caching strategy that regularizes active weights against catastrophic drift. Our method demonstrates real-time, state-of-the-art performance on scenes with dynamic human motion as well as minute-scale online memorization.