Multimodal drone dataset enables better aerial perception across sensors
MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency
RoboticsComputer Vision and Pattern Recognition
Summary
Understanding scenes from drones is tricky because different cameras and sensors see the world differently. The authors created MultiFly, a large dataset with images and measurements from regular cameras, heat sensors, laser scanners, and radar on drones flying low over towns. They cleverly labeled a small number of images by hand and then used geometry to copy those labels to many other images and sensors, saving a lot of manual work. This dataset helps computers learn to recognize objects consistently in all these different types of data from drones.
What this means in practice
- •For drone navigation teams: Develop and test algorithms for recognizing objects and terrain using combined camera, thermal, LiDAR, and radar data from low-flying drones.
- •For emergency response planners: Create reliable real-time scene understanding tools from multimodal drone sensors to assist in disaster or search-and-rescue missions.
Authors
Markus Gross, Andreas Greiner, Taehyoung Kim, Sivasubiramaniam Subbiah, Tomaž Cotič, Sai Bharadwaj Matha, Conrad Christoph, Oussema Dhaouadi, Simon Zieher, Surya Vijaya Kumar, Gordon Elger, Henri Meeß, Olaf Wysocki, Paul Spannaus, Daniel Cremers
Abstract
We introduce MultiFly, a real-world, low-altitude UAV dataset for semantic perception across RGB, thermal, LiDAR, and radar modalities. MultiFly provides 17,272 synchronized samples from four suburban scenes with frame-wise annotations for 15 semantic classes, together with calibration and GNSS-RTK/IMU measurements. To avoid costly and inconsistent modality-specific annotation, we propagate labels from only 115 manually annotated RGB images through shared geometric representations to all four modalities. This approach generates semantic labels for 17,157 additional RGB images, 17,272 thermal images, 840M LiDAR points, and 3.4M radar points. Transferred annotations achieve 89.93% average agreement with held-out manual annotations, and 90.94% average semantic consistency across all six modality pairs. We further establish semantic segmentation benchmarks for all four modalities, revealing distinct architectural behavior for dense LiDAR and sparse radar data. Taken together, MultiFly provides a scalable foundation for multimodal aerial perception and, to the best of our knowledge, the first public real-world low-altitude aerial benchmark that combines consistent frame-wise semantic annotations for RGB, thermal, LiDAR, and radar. Data at https://github.com/markus-42/multifly.