Multi-view model improves locating objects from drone and street images
Multi-View Mixture-of-Experts with Vision-Language Reranking for Cross-View Object Geo-Localization
Computer Vision and Pattern Recognition
Summary
Finding the exact location of an object on a map using images from drones or street views can be tricky because different perspectives show different details. The authors developed MVLGeo, a single model that works with multiple viewpoints and shares knowledge across them, reducing the need for separate detectors. They also add text describing the scene to help tell apart objects that look very similar. Their approach improves accuracy and works well even if image quality drops or views change.
What this means in practice
- •For drone mapping teams: Use a unified model to more accurately locate objects on satellite maps based on drone images from various angles and conditions.
- •For autonomous vehicle developers: Improve vehicle geo-localization by integrating cross-view data from street and aerial cameras with context-aware reranking.
Authors
Xuyu Fan, Qi Ming, Zhu Han, Liuqian Wang, Siyuan Cao, Xiaohan Zhang, Xudong Zhao, Mingjing Zhao, Yuhan Zhang
Abstract
Cross-view object geo-localization (CVOGL) locates a target in satellite imagery using drone or street-view queries. Existing methods train separate detectors for each viewpoint, leading to parameter redundancy and impeding cross-view knowledge sharing. Moreover, top-ranked satellite candidates are often visually similar, so visual appearance and categorical labels alone are insufficient to resolve such ambiguity. To address these, we propose MVLGeo, an efficient framework designed to unify multiple viewpoints and reduce model redundancy. First, we introduce environmental contextual text from the query view as cues to distinguish visually similar candidates via Vision-Language Reranking (VL-Rerank). Second, we design a multi-view Mixture-of-Experts architecture (MV-MoE) with a shared encoder and view-specific experts to reduce redundancy and promote knowledge sharing, while cross-view contrastive learning aligns their representations for consistency. Third, we introduce an adaptive elliptical prior (ESAM-Prior) as auxiliary positional encoding for anisotropic geometric perception. Extensive experiments on the CVOGL benchmarks confirm that MVLGeo, as a unified model for multiple query viewpoints, achieves state-of-the-art performance, demonstrating robustness to input degradation and generalization across viewpoints. Code and models will be available on GitHub to facilitate future work.