Vision-Language-Motion Maps: An Open-Vocabulary, Uncertainty-Aware, Queryable Motion Attribute for 3D Scene Maps
2026-07-17 • Robotics
Robotics
AI summaryⓘ
The authors present Vision-Language-Motion Maps (VLMM), a new type of 3D map that lets robots answer questions about both what objects are in a scene and how those objects might move. VLMM combines knowledge about which things can move with actual observed motion between frames and keeps track of uncertainty for each object. They tested their approach in a simulator and real-world videos, showing that using both semantic motion priors and observed motion, along with uncertainty, helps robots better understand moving versus still objects. The authors show that previous methods lack one or more of these key features, making VLMM a unique map representation.
3D mapsopen-vocabularyvision-language modelsmotion detectionsemantic priorsRGB-D datauncertainty estimationrobot perceptionlanguage queryingAI2-THOR simulator
Authors
Dibyendu Ghosh, Ayushi Shakya
Abstract
Open-vocabulary 3D maps let robots answer language queries about what and where, but they assume a static world and cannot answer queries about how scene elements behave. We introduce Vision-Language-Motion Maps (VLMM), an open-vocabulary, natural-language-queryable 3D map in which each element carries a fused motion attribute: a VLM/LLM semantic movability prior combined with geometrically observed cross-frame motion, together with a per-element uncertainty. Queries reduce to attribute filters that distinguish what has been seen to move, what could move but has not, and what stays still. On a controlled simulator benchmark with exact ground truth (AI2-THOR, three scene types) we show through ablation that the schema fields are non-substitutable: a semantic-only baseline fails motion queries even with strong features, and neither motion field substitutes for the other (the prior cannot answer "what is moving," observed motion cannot answer "what could move"). On real dynamic RGB-D (TUM and Bonn, six sequences) we show the uncertainty channel-our key difference from prior fused-motion work-consistently improves moving-vs-static average precision and reduces false motion flags, and is robust to estimated (noisy) poses. The raw confidence is not calibrated, but post-hoc isotonic calibration reaches an expected calibration error of 0.10. VLMM is a representation contribution: the closest prior maps each lack at least one of the four properties-open-vocabulary, language-queryable, fused prior-and-observed motion, and per-element uncertainty-that our combination provides.