Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels
2026-07-12 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on improving image segmentation when the labels (like maps) don't line up perfectly with the images, which is common in satellite or aerial data. They created a method called Align and Segment (AnS) that learns how to fix the misalignment while teaching the segmentation model at the same time. Their approach uses a technique to adjust the labels automatically and avoids common learning mistakes by adding a special rule during training. They tested this on both fake and real city data and showed it works well without needing perfectly matched labels beforehand.
image segmentationlabel alignmentspatial transformeraffine transformationsemantic segmentationremote sensingself-supervised learningdata augmentationco-registrationOpenStreetMap
Authors
Venkanna Babu Guthula, Oswin Krause, Dimitri Gominski, Hui Zhang, Johan Mottelson, Ankit Kariryaa, Nico Lang, Christian Igel
Abstract
Supervised learning for image segmentation typically requires spatially aligned image and label sets. When images and labels originate from different sources, the pairing may be misaligned, which can significantly deteriorate the performance of the learned models. This is especially common in remote sensing, when aerial or satellite images are co-registered with labels from another source (e.g., OpenStreetMap). In this work, we propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment. Our align and segment (AnS) approach builds on the spatial transformer module to transform the misaligned labels using an affine transformation to provide a better learning target for a canonical semantic segmentation network. We prevent shortcut learning of misaligned labels in these semantic segmentation networks through a self-supervised regularization loss and show that it is complementary to data augmentation, especially for systematically misaligned training data. A decisive characteristic of our AnS approach is that it learns without requiring any golden labels. We experimentally show on both synthetic and real-world data from different cities that our approach enables high-quality building segmentation and precise label-image alignment at the same time. Code and derived datasets are available at https://github.com/venkanna37/align-and-segment