Multimodal mapping improves construction robot navigation safety
Robust 2D Traversability Mapping for Construction AMRs via Failure-Mode-Aware Fusion of LiDAR Geometry and Monocular Semantics
RoboticsComputer Vision and Pattern Recognition
Summary
Mobile robots working on construction sites have trouble navigating because some dangerous areas look flat but are unsafe, like wet mud, while some safe features like speed bumps look like obstacles to them. The authors created a system that uses laser sensors to map shapes and a camera to recognize materials, combining both to better tell safe from unsafe ground. This helps robots avoid hazards they couldn't detect before. They also provide a new dataset to help improve navigation in construction sites.
What this means in practice
- •For construction robotics teams: Improve navigation safety by integrating laser and camera data to detect hazards like wet surfaces that pure geometry misses.
- •For industrial robot developers: Use the released multimodal dataset and code to train and test advanced robot navigation systems in complex construction environments.
Authors
Manoj Karnekar, Om Mandhane, Gautham Ramkumar
Abstract
Autonomous Mobile Robots (AMRs) on active construction sites face severe navigational challenges: geometry-based traversability mapping (e.g., LiDAR) misses visually hazardous but geometrically flat surfaces like wet mud and ponding concrete, while abrupt geometry on drivable speed-breakers and inclines produces phantom obstacles. We propose a real-time, failure-mode-aware multimodal traversability pipeline on an NVIDIA Jetson AGX Orin, where LiDAR is the primary geometric safety estimate and monocular semantics act as a selective, class- and confidence-gated corrective signal. The representation retains distinct traversable classes, namely flat road, terrain, and rocky terrain, while flagging construction hazards. We also release a multimodal construction-site dataset from a custom AMR: four closed-loop ROS 2 sequences from two active sites (RGB, depth, LiDAR, IMU, GPS-RTK, odometry) plus 506 annotated frames across 28 semantic classes. By projecting LiDAR onto dense semantic masks, resolving sparsity via morphological in-painting, and applying failure-mode-aware fusion with Patchwork++, the system corrects complementary geometric failure modes for a local AMR costmap.