LiDAR language models struggle to grasp space and time relationships
Do LiDAR Language Models Really Understand Spatio-temporal Relationships?
Computer Vision and Pattern RecognitionArtificial IntelligenceRobotics
Summary
LiDAR language models try to understand how objects move and relate in space and time, but the authors found these models often guess without really understanding. They created a new test with thousands of questions that check if the models truly know object positions, movements, and timing. The results showed the models often fail to tell when objects move sideways or change over time. This means that just looking at overall accuracy misses important mistakes in understanding spatial and temporal details.
What this means in practice
- •For autonomous vehicle engineers: Improve testing protocols to better detect failures in LiDAR-based scene understanding systems for self-driving cars.
- •For robotics developers: Use a reliable diagnostic benchmark to evaluate robot perception models on spatial and motion understanding from LiDAR data.
Authors
Runyi Yang, Murat Akkoyun, Di Wen, Ruiping Liu, Yufan Chen, Junwei Zheng, Xiaoye Wang, Kailun Yang, Danda Pani Paudel, Luc Van Gool, Kunyu Peng
Abstract
Recent 4D LiDAR language models aim to reason about objects and their evolving spatial relationships. Yet, in our evaluation, always selecting the same option nearly matches the multiple-choice accuracy of two B4DL-derived configurations. We introduce LiDAR-Hallu, a geometry-referenced benchmark and diagnostic protocol with 10,000 questions across 150 nuScenes scenes. It covers object existence, ego-relative position, distance ordering, relative motion, and temporal localization, with explicit rules for selecting objects, comparing times, and determining reference answers. Our protocol combines fixed-answer and candidate-content controls, cross-scene pairs with identical prompts but opposite reference answers, and relation-specific recall. Analysis of 100,000 recorded responses reveals failures hidden by aggregate accuracy. Candidate duration alone makes temporal answers predictable without observing LiDAR. On paired questions, the models frequently give the same answer to scenes requiring opposite answers. Relation-specific analysis further shows that both configurations miss every positive lateral-motion case across all tested conditions. Temporal-shuffle contrastive decoding provides little net improvement, as repairs are largely offset by new errors and the main failures persist. These results show that evaluating spatio-temporal reasoning requires testing whether models distinguish the queried physical relationships, rather than relying on individual-answer accuracy alone. The source code, checkpoints, and data are released at https://github.com/Awesome4D/4DMLLM_Hallucination_Bench.