Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning
2026-05-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and Language
AI summaryⓘ
The authors study how vision-language models struggle with precise spatial tasks like understanding viewpoints and distances from multiple images or videos. They propose a system called Reasmory that builds a clear 3D memory of the scene and uses a simple programming language to help the model ask questions and transform views in a controlled way. This method helps avoid mistakes common in less structured approaches and improves performance on spatial reasoning tasks by 6-18%. The results suggest that having an explicit 3D memory is most helpful when the model accesses it through carefully checked steps.
Vision-Language ModelsSpatial Reasoning3D ReconstructionPoint CloudsDomain-Specific LanguageProgram ExecutionViewpoint TransformationSpatial MemoryMulti-view Imaging
Authors
Jixuan He, Xueting Li, Chieh Hubert Lin, Ming-Hsuan Yang
Abstract
Vision-Language Models (VLMs) exhibit emerging spatial reasoning capabilities, yet they remain unreliable on tasks requiring precise spatial understanding, such as viewpoint reasoning, directional comparison, and distance estimation. In multi-view images and monocular videos, relevant spatial cues are often sparse and distributed across redundant observations, making them difficult to organize and exploit. Reconstruction-based Vision Foundation Models (VFMs) offer a natural way to aggregate such observations into explicit spatial memory, such as point clouds. However, simply exposing reconstruction models as free-form tools is brittle, VLMs may invoke tools incorrectly, skip required spatial transformations, or misuse intermediate results. We propose \textbf{Reasmory}, a framework that formulates spatial reasoning as structured program execution over reconstructed spatial memory. Reasmory constructs explicit 3D memory, augments it with semantically grounded 3D object instances, and introduces a lightweight Domain-Specific Language (DSL) that constrains how VLMs query objects and cameras, transform viewpoints, and render observations during reasoning. Generated programs are parsed and validated before execution, enabling more reliable interaction with spatial memory than unconstrained tool use. Experiments on multi-view image and video spatial reasoning benchmarks show consistent gains of 6--18\% over strong baselines, including GPT-5-mini and Gemini-3-flash, indicating that explicit 3D memory is most useful when accessed through constrained, validated operations rather than free-form tool calls.