3-D road hazard detection improved by linking LiDAR and camera data
ReinSeg: Retrieval-Guided Bounded Multimodal Refinement for 3-D Road Anomaly Segmentation
Computer Vision and Pattern Recognition
Summary
Detecting road hazards in 3-D is hard because many normal road features can look similar to dangers. The authors created ReinSeg, a method that uses LiDAR data combined with images from cameras around the vehicle to better spot unusual road obstacles. Their method searches for matching scenes in stored images and adjusts the LiDAR readings, helping the system tell dangers apart from normal road parts more accurately. Tests show this approach significantly reduces false alarms while keeping fast processing speeds.
What this means in practice
- •For autonomous vehicle engineers: Improve detection of road hazards by integrating LiDAR with surrounding camera images for clearer autonomous driving perception.
- •For smart city infrastructure teams: Enhance road condition monitoring systems by using combined LiDAR and image data to detect unexpected anomalies in 3-D environments.
Authors
Haifa Zhang, Yijing Wang, Zhiqiang Zuo
Abstract
3-D road anomaly segmentation is essential for risk-aware perception in autonomous driving. Its central challenge is to rank sparsely observed hazards ahead of abundant normal road structures while preserving point-wise metric localization. LiDAR provides accurate 3-D point support, whereas surround-view imagery offers complementary semantic context for separating unexpected hazards from challenging normal structures. We propose ReinSeg, a retrieval-guided bounded multimodal refinement framework for point-wise road anomaly segmentation. Occupied LiDAR bird's-eye-view cells retrieve scene-specific context from a shared memory of eight surround-view images without supplying calibration variables to the retrieval module. A relative-norm-bounded residual returns this context to the LiDAR point features, so the frozen LIDO anomaly-scoring interface is preserved, and the refinement is optimized with in-distribution semantic labels only. On the Spotting the Unexpected (STU) benchmark, ReinSeg improves validation average precision from 27.54\% to 35.67\% and reduces the false-positive rate at 95\% anomaly recall from 34.85\% to 10.61\%, shrinking the high-recall screening set from 67.54 to 20.55 million points. Relative to REL, AP is 24.99 percentage points higher and FPR95 is 0.99 points lower. Matched interfaces and component isolation show that retrieved scene imagery and the relative-norm bound are both essential, and ReinSeg stays above LiDAR-only performance under missing, corrupted, or misaligned camera inputs while running at 21.8 frames per second. These results establish bounded retrieval of scene-specific visual evidence as an effective route to reliable 3-D road-hazard perception.