Summary
Identifying what a drone is carrying from far away is hard because real training data is limited and objects can look different in new pictures. The researchers created a computer-generated dataset of drones with payloads and built a two-step system called DroneGround. First, it finds the drone in an image, then it uses a smart language-based AI to describe the drone and its payload, even if it hasn’t seen that object before. This approach works better than traditional methods, especially when recognizing new things, and helps explain which part of the image led to the description. Their tests show this method can handle different types of images more accurately.
drone payload characterizationsynthetic datasetvision-language modelYOLO object detectionopen-vocabulary recognitiondistribution shiftsemantic descriptionocclusion groundingdrone surveillancepayload localization
Authors
Ami Pandat, Rajasekhar Punna, Gopika Vinod, Rohit Shukla
Abstract
Automated drone surveillance has become increasingly important for public safety, critical infrastructure protection,and restricted airspace monitoring. While existing vision-based systems achieve strong performance for drone detection and tracking, reliable payload characterization remains highly challenging under long-range imaging conditions due to limited availability of annotated real-world datasets, and substantial distribution shifts encountered during deployment. Existing approaches formulate payload characterization as a closed-set object detection problem, limiting their ability to recognize previously unseen payloads and generalize beyond the training distribution. To address these challenges, we generate a photorealistic synthetic drone-payload dataset using Unreal Engine 5 and Cosys-AirSim and propose DroneGround: Grounded Vision-Language Payload Characterization, a two-stage framework for robust open-vocabulary payload analysis. DroneGround first employs a YOLO26s detector to localize drones and extract drone-centric image crops, followed by a LoRA-fine-tuned PaliGemma vision-language model that generates seman- tic descriptions of the detected drones and their attached payloads, enabling open-vocabulary payload characterization beyond predefined categories. An occlusion-based grounding module further provides interpretable payload localization by identifying image regions responsible for the generated descriptions. Extensive experiments on both synthetic and real-world drone imagery demonstrate that DroneGround substantially improves robustness under synthetic-to-real distribution shifts, outperforming a conventional closed-set payload detector by improving the F1-score from 82.5% to 96.3%, while achieving significantly better generalization to previously unseen payload categories (80.4%versus 42.7% F1). Dataset and code will be released upon acceptance of the paper.