Streaming 3D models improve memory by writing keyframes smartly
SceneBound: How Much Should a Streaming 3D Reconstruction Model Remember?
Computer Vision and Pattern RecognitionRobotics
Summary
Streaming 3D reconstruction models remember past video frames to build a 3D scene as you move around. But saving too many frames uses too much memory and can create errors. The authors propose SceneBound, a simple way that remembers a new frame only when the camera moves far enough from previously saved viewpoints. This keeps the number of saved frames just enough to cover the visited places well, improving accuracy and saving memory without needing additional training.
What this means in practice
- •For augmented reality developers: Enable AR apps to maintain accurate 3D maps by selectively storing only necessary keyframes as users move in a scene.
- •For robotic navigation teams: Improve robot localization and mapping by reducing memory needs and error drift during real-time 3D environment reconstruction.
Authors
Ziming Wang, Chuyue Gong, Chutian Wang, Jingmin Zhu, Hongliang Cao, Yuheng He, Xiuyu Li
Abstract
Streaming 3D reconstruction models keep past video frames in a recurrent state, fast weights from test-time training, or a key-value (KV) cache. Each written frame shifts the coordinate frame of later cameras or occupies GPU memory. Training-free rules decide what to write from a signal of the current frame. How much they write accounts for their mean accuracy: spending the update budget of a gate at uniformly spaced frames, with no signal, matches nine published gates, and the budget bounds the drift of the state wherever the writes fall. We propose SceneBound, a training-free rule that writes a frame only when its predicted camera pose lies at least a fixed radius from every stored keyframe. Its keyframe count stays within a constant factor of the covering number of the visited poses, the fewest views that keep a keyframe near every camera, so the count grows with the places visited and stays flat when the camera returns. In large scenes, SceneBound closes a submap once decoding its first frame again reveals drift. SceneBound uses one radius across four released models and three memory types. In the recurrent state of CUT3R and the KV cache of StreamVGGT, it achieves the lowest mean trajectory error among the published training-free rules we evaluate, in one pass and over nine passes of a scene, where its keyframes grow by 3% to 6%.