Spatiotemporal memory system for changing 3D scenes improves reasoning

Gaussian Episodic Memory for Long-Term Spatiotemporal Reasoning

Computer Vision and Pattern RecognitionRobotics

Summary

Robots and other machines need to remember both how places look and how they change over time to understand their surroundings well. The authors developed GEMS, a system that keeps a detailed 3D memory of scenes and updates only when parts change, saving storage and detail. It also uses this memory to answer questions about objects and events from different viewpoints. They tested GEMS in real and simulated places, showing it performs better than existing methods at recognizing objects and answering questions about scene changes.

What this means in practice

  • For robotics developers: Provide robots with memory that efficiently tracks spatial changes over time, enabling them to better understand and reason about dynamic environments.
  • For augmented reality creators: Enhance AR experiences by allowing systems to render updated 3D views of changing real-world scenes and answer spatial questions about them.

Authors

Ling-Chia Chen, Abdelrhman Werby, Kaan Aytekin, Kai O. Arras

Abstract

Real-world environments undergo continual semi-static changes, requiring a memory representation that captures both the spatial and temporal evolution of a scene. Existing spatio-temporal memories for robots typically rely either on structured representations, which are limited to the entities and attributes captured during construction, or on image-based memories, which are constrained to previously observed viewpoints. We introduce GEMS, a framework that combines a persistent spatio-temporal 3D Gaussian Splatting (3DGS) map with an agentic reasoning pipeline. The 3DGS map is reconstructed once and subsequently updated only in regions where the scene changes, storing only these modifications while preserving visual detail across time. The agentic pipeline leverages this map to render novel views on demand, iteratively accumulating evidence to answer spatio-temporal questions and ground objects from natural-language queries. To advance future research, we introduce GEMS-Bench, a spatiotemporal question-answering benchmark focused on semi-static environments. It comprises 3 real-world and 12 synthetic environments, spanning 473 question-answer pairs. We evaluate GEMS on novel-view synthesis quality across scene updates, 3D object grounding, spatiotemporal question answering, and storage efficiency. GEMS outperforms recent baselines in 3D object grounding and spatiotemporal question answering, while achieving rendering quality on par with dedicated scene-update methods. Code and data: https://abwerby.github.io/GEMS/