Geolocalization accuracy drops when inserted people mismatch regions
GeoBiaset: A Counterfactual Benchmark for Demographic Bias in World-Level Geolocalization
Computer Vision and Pattern Recognition
Summary
Geolocalization systems try to guess where a photo was taken by analyzing its content. This paper shows that these systems can be biased by the race of people shown in the image, even when the background stays the same. The authors made a special set of photos where the same place is shown with different inserted faces of various races. They found that models often get worse at guessing location if the face doesn’t match the typical demographics of the region. This suggests current models pay attention to people’s facial features when landmarks are missing, which can lead to bias.
What this means in practice
- •For geospatial software developers: Improve geolocation software accuracy by testing model robustness to demographic variations in people inserted into images.
- •For machine learning fairness teams: Use the GeoBiaset benchmark and POPI metrics to identify and mitigate racial bias in visual geolocation models.
Authors
Niccolò Niccoli, Silvia Dani, Marco Mistretta, Andrew D. Bagdanov, Marco Bertini, Lorenzo Seidenari
Abstract
World-level image geolocalization is an increasingly important task, yet the biases of current models remain poorly understood. Prior work has identified coarse dataset imbalances noting, for instance, that some cities are absent from training sets and that models tend to perform better in wealthier regions. However, no one has yet examined which visual cues drive these disparities. To do so, we introduce GeoBiaset, the first benchmark designed to measure how geolocalization models are influenced by the apparent race of a person inserted into the image foreground. The dataset consists of 9,541 images, pairing existing geotagged background images with counterfactual edits spanning seven races, holding the ground-truth coordinates fixed. Controlled masking experiments show that occlusion and background drift do not fully explain geolocalization degradation. Alongside the benchmark, we introduce the Person-Origin Perturbation Indicators (POPI), novel metrics that quantify model resilience to these demographic perturbations by measuring whether predictions are pulled toward the region associated with the inserted group. Our evaluation of state-of-the-art geolocalization and multimodal large language models reveals a 16-47% drop in continent-level performance when a subject from a demographically mismatched group is inserted. A landmark versus no-landmark analysis and gradient-weighted heatmaps show that models preferentially attend to geographic landmarks when present, but shift attention to facial features in their absence.