Trimodal system improves 3D person detection for rescue robots
TriSAR-3D and PaintFusion: A Novel Trimodal LiDAR-RGB-Thermal Dataset and Method for 3D Person Detection in Search and Rescue
Computer Vision and Pattern RecognitionRobotics
Summary
Finding people quickly and accurately during rescue missions is tough because they can be in many different poses and environments. The authors created a new dataset that uses 3D laser scanning, regular color images, and thermal images together to capture people in varied indoor and outdoor scenes. They also developed a new method that combines these three types of data to better detect people in 3D. This method is more accurate and faster than others tested on the dataset, helping rescue robots see people in challenging situations more reliably.
What this means in practice
- •For rescue robotics teams: Equip robots with a system combining LiDAR, RGB, and thermal data to better detect and locate people in varied poses and environments during search and rescue operations.
- •For security monitoring teams: Use fused 3D, RGB, and thermal data to improve detection of people in complex, occluded, or low-visibility settings at facilities and public areas.
Authors
Markus Kramer, Jan Wachenfeld, Oskar von Stryk
Abstract
Reliable 3D person detection is important for rescue robotics to give first responders situational awareness. Yet most multimodal 3D detection research targets automotive scenarios in which pedestrians mostly appear in a narrow range of upright, near-road poses. It is therefore unclear how much of the reported performance transfers to rescue settings, as a person may be lying, partially occluded, or in unstructured indoor and outdoor environments. To investigate this gap, we introduce TriSAR-3D, a multimodal person detection dataset recorded with a high-resolution 128-channel LiDAR, an RGB camera, and a thermal camera. It captures varied poses in indoor as well as outdoor locations at different times of day to study their influence on 3D object detector performance. Furthermore, we propose a novel trimodal 3D detection pipeline, PaintFusion, that fuses PointPainting-based evidence using a simple MLP. We show that the derived method reaches 65.4 AP@50 and thereby outperforms strong LiDAR-only (CenterPoint, 53.4), RGB+LiDAR (BEVFusion, 60.5), and adapted trimodal (Tri-BEVFusion, 60.7) baselines on the dataset. At the same time, it is substantially faster than other fusion-based methods such as BEVFusion, with a per-frame latency of 51.6 ms compared to 101 ms. Additionally, we test how the performance of different 3D object detectors generalizes to unseen pose categories. Removing lying instances from training data collapses the lying-pose AP@50 to near zero across all evaluated detectors. The results indicate that dataset diversity exerts a large effect on 3D object detection performance. The dataset and code are available at https://github.com/MarkusKramer1/3D-Trimodal-People-Detection.