Slot3R improves memory for 3D scene reconstruction in real time
Slot3R: Set-Associative Spatial Memory for Streaming 3D Reconstruction
Computer Vision and Pattern Recognition
Summary
Building a 3D model of a scene from video frames is hard because each new image must be combined correctly with what has been seen before. The authors show that treating nearby spots in space as one memory location causes problems when multiple different surfaces or views appear close together. They introduce Slot3R, a method that allows several separate memory slots at the same place so details aren't mixed up too early. This leads to more accurate 3D models and better tracking, while handling long video sequences efficiently without running out of memory.
What this means in practice
- •For augmented reality developers: Maintain detailed and accurate 3D maps of environments during live camera scanning without losing distinct scene details.
- •For robotics navigation teams: Improve real-time spatial memory to track a robot’s trajectory and environment more reliably for online decision making.
Authors
Xiyuan Zhang, Yanming Yang, Kaiyuan Xu, Ruibo Li, Chi Zhang
Abstract
Streaming 3D reconstruction must preserve evidence from each frame while processing an expanding scene online. Spatial memory is a natural fit because it organizes history by reconstructed 3D location. Yet Point3R uses spatial proximity both to associate a new observation with an existing memory entry and to decide whether to fuse it, conflating co-location with state identity. Because pointers summarize image patches, nearby pointers may encode distinct surfaces, viewpoints, or visibility conditions; averaging them can destroy complementary evidence before later frames disambiguate it. We argue that location should determine address, not whether observations must merge. Slot3R realizes this principle as a training-free, set-associative retrofit that lets multiple states coexist at a shared address while keeping the pretrained Point3R backbone frozen. A bounded sparse readout further decouples persistent storage from per-frame decoder access. At 300-500 sampled frames, Slot3R reduces Point3R's point-cloud accuracy error (Acc) by 57.1%-63.1% on 7Scenes and 64.0%-72.0% on NeuralRGBD, lowers Sim(3)-aligned absolute trajectory error (ATE) on all three pose benchmarks, and remains competitive on video-depth estimation. It completes all evaluated settings from 600 to 1000 sampled frames at about 19 FPS under the same protocol, whereas Point3R and InfiniteVGGT run out of memory at 800 frames and beyond.