LiDAR and large language models improve spatial object grounding
Retrieve-to-Localize: Bridging Large Language Models and LiDAR Geometry for Spatial Grounding
Computer Vision and Pattern RecognitionRobotics
Summary
LiDAR sensors capture the exact shapes and positions of objects in the environment, but they alone don’t understand complex questions about where things are. The authors connect advanced language models, which understand language well, to LiDAR data to better find and identify objects based on detailed spatial questions. They created a new dataset and a method that uses language to guide the search in LiDAR data, pinpointing exact object locations without relying on text descriptions alone. Their system performs better than previous models at precisely locating objects based on language instructions.
What this means in practice
- •For autonomous vehicle engineers: Precisely locate and identify objects in 3D LiDAR data based on complex language queries for better decision-making in autonomous driving.
- •For outdoor robotics developers: Enable robots to understand spatial instructions and accurately find objects using LiDAR sensing combined with language understanding.
Authors
Byounggun Park, Giyong Moon, Jusung Kim, Soonmin Hwang
Abstract
LiDAR provides precise geometric information for spatial perception tasks such as object detection in autonomous driving and outdoor robotics. However, recognizing and localizing individual objects is not sufficient to answer questions that require composing spatial relations and grounding the intended target. Motivated by recent advances in large language models (LLMs) for autonomous driving, we leverage their language priors to interpret complex spatial questions and ground the referred target in LiDAR geometry. To support this spatial grounding capability, we introduce SpatialLiDAR-QA, which combines single- and multi-step relational grounding with complementary spatial understanding tasks. We further propose SpatialLiDAR-LM, which aligns LiDAR point features with an LLM and grounds target coordinates through language-conditioned, position-aware proposal retrieval and local point refinement. This design derives target coordinates directly from local LiDAR geometry rather than through textual language decoding. Experiments demonstrate substantial improvements over representative LiDAR--language models and multi-camera VLMs on precise coordinate prediction tasks. Our dataset and model training code will be publicly released.