DroneGround improves drone payload recognition with synthetic data and language models
DroneGround: Open-Vocabulary Drone Payload Characterization Using Synthetic Data and Grounded Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Identifying objects attached to drones from far away is hard because there aren’t many real pictures to learn from, and new types of objects keep showing up. The authors created a detailed set of fake drone images using a simulation engine and combined it with smart AI that understands both images and language. Their system, DroneGround, can spot drones, describe what’s attached to them in everyday words, and find parts of the image that show those objects, even if it hasn’t seen those exact objects before. This makes it better at recognizing all kinds of drone payloads, helping improve surveillance and safety.
What this means in practice
- •For security monitoring teams: Identify and describe unknown objects attached to drones in restricted areas using open-vocabulary AI models.
- •For wildlife researchers: Analyze drone payloads remotely to verify sensors or equipment types during environmental studies without prior payload categories.
Authors
Ami Pandat, Rajasekhar Punna, Gopika Vinod, Rohit Shukla
Abstract
Automated drone surveillance has become increasingly important for public safety, critical infrastructure protection,and restricted airspace monitoring. While existing vision-based systems achieve strong performance for drone detection and tracking, reliable payload characterization remains highly challenging under long-range imaging conditions due to limited availability of annotated real-world datasets, and substantial distribution shifts encountered during deployment. Existing approaches formulate payload characterization as a closed-set object detection problem, limiting their ability to recognize previously unseen payloads and generalize beyond the training distribution. To address these challenges, we generate a photorealistic synthetic drone-payload dataset using Unreal Engine 5 and Cosys-AirSim and propose DroneGround: Grounded Vision-Language Payload Characterization, a two-stage framework for robust open-vocabulary payload analysis. DroneGround first employs a YOLO26s detector to localize drones and extract drone-centric image crops, followed by a LoRA-fine-tuned PaliGemma vision-language model that generates seman- tic descriptions of the detected drones and their attached payloads, enabling open-vocabulary payload characterization beyond predefined categories. An occlusion-based grounding module further provides interpretable payload localization by identifying image regions responsible for the generated descriptions. Extensive experiments on both synthetic and real-world drone imagery demonstrate that DroneGround substantially improves robustness under synthetic-to-real distribution shifts, outperforming a conventional closed-set payload detector by improving the F1-score from 82.5% to 96.3%, while achieving significantly better generalization to previously unseen payload categories (80.4%versus 42.7% F1). Dataset and code will be released upon acceptance of the paper.