ShadowTransfer benchmark reveals limits of geographic shadow detection
ShadowTransfer: A Geographic Transfer Benchmark for Overhead Shadow Detection
Computer Vision and Pattern Recognition
Summary
Detecting shadows in aerial images is important but models trained in one city often fail in others. The authors created ShadowTransfer, a dataset with hand-verified shadows across three U.S. cities to test this problem. They found that models' performance drops notably when applied to new cities, and many usual methods to adapt them actually worsen results. To understand why, they diagnosed specific failures mostly in the model’s features and confidence. Using these insights, they designed a new module, Spectral Information Bottleneck, which improves transfer performance closer to the best within-city results.
What this means in practice
- •For geospatial software developers: Improve shadow detection models to maintain accuracy when deploying aerial image analysis across different cities.
- •For urban planning teams: Use enhanced transfer shadow detection models to support consistent mapping and analysis from aerial images in multiple urban areas.
Authors
Shagun Mittal, Raymond A. Yeh, Satish Ukkusuri
Abstract
Overhead aerial imagery feeds dense-prediction pipelines that are routinely trained in one geographic region and deployed in another, yet no benchmark measures how shadow detection models transfer across regions. Existing overhead shadow datasets either lack a geographic transfer protocol or rely on automatically generated labels. We introduce ShadowTransfer, a benchmark of 4,500 human-verified shadow masks across three U.S. cities at two native resolutions, paired with a leave-one-city-out evaluation protocol. Under vanilla training, holding out the test city costs 3.5 to 7.8 mIoU across four architectures, the gap remains significant when the training set grows from 225 to 900 images, and six of eight adaptation methods degrade more than vanilla training on average. We propose a three-probe framework that diagnoses where and how the transfer breaks: the failure operates in feature space, beyond the reach of pixel-histogram matching, shows up as a confidence collapse on shadow pixels while background confidence stays intact, and is recoverable by retraining different architectural components depending on the model: encoder for some, decoder for others. These findings replicate on INRIA building footprints, suggesting they describe geographic transfer beyond shadows. Guided by the diagnosis, we build Spectral Information Bottleneck (SIB), a plug-in module with one component per finding. Across the 9 (architecture, held-out city) cells, SIB has the smallest mean gap to the within-city upper bound of the ten methods we evaluate (6.42 mIoU), level with FDA. The benchmark, LOCO protocol, code, and trained models are released to support reproducible evaluation of geographic transfer in overhead dense prediction.