Vision transformer estimates drone pose using synthetic images
Synthetic-to-Real ViT-Based Pose Estimation of a Noncooperative UAV
Computer Vision and Pattern Recognition
Summary
Estimating the direction and position of drones that cannot be controlled or fitted with special equipment is important but hard to do with real images alone. The authors used images generated by computers to train a vision transformer, a type of AI model, to recognize drone poses from one camera view. Their model learns from synthetic images but performs well on real photos, helped by special techniques to handle confusing views and drone symmetry. They also improved accuracy by using a filtering method during prediction.
What this means in practice
- •For drone security teams: Identify and track noncooperative drones remotely using a single camera without needing prior drone instrumentation.
- •For robotics developers: Incorporate synthetic-image-trained vision transformer models for real-time pose estimation in drone navigation and control systems.
Authors
Krishnanujam Srinivas, Hanish Acharla, Brij Agrawal, Leonardo Herrera
Abstract
Remote pose estimation of noncooperative Unmanned Aerial Vehicles (UAVs) from imagery is critical, as they cannot be influenced or instrumented in advance. Deep-learning-based approaches offer a promising solution; however, their development is constrained by the cost and difficulty of acquiring large-scale real-world datasets with accurate pose labels. Synthetic imagery provides an alternative, but models trained on synthetic data must overcome the synthetic-to-real domain gap to generalize to real-world imagery. This work investigates the inherent synthetic-to-real generalization capability of a Vision Transformer (ViT)-based model for monocular UAV pose estimation. The proposed approach employs a self-supervised DINOv2 backbone and is trained exclusively on labeled synthetic imagery while being evaluated on labeled real-world imagery. Pose ambiguity-aware strategies are incorporated during training and inference to address ambiguities arising from the projection of a three-dimensional target onto a two-dimensional image plane and from target symmetries. An $α$-$β$ filter is further integrated during inference to improve pose estimations. To assess the model under operational requirements, it is evaluated in terms of Mean Angular Error (MAE) and inference time, both before and after filtering, using a real-world dataset containing 77,077 labeled UAV images. Before filtering, the model achieves an MAE of $19.18^{\circ}$ and an inference time of $13.25$ ms, whereas after filtering, these values are $8.74^{\circ}$ and $13.42$ ms, respectively.