Benchmark reveals challenges in describing long term image changes
Spot-the-shift: Evaluating Grounded Image Difference Captioning of Long-term Changes
Computer Vision and Pattern Recognition
Summary
Understanding changes in images taken from the same place over time is tricky but important for things like keeping maps updated. The authors created SPOT-THE-SHIFT, a test dataset with image pairs, descriptions in everyday language, and marked areas that show changes. They also made a way to check how well computer models can notice and explain these changes. Testing current models showed they find it hard to describe detailed changes over multiple images. The authors improved a model by generating synthetic training data that helps it learn better without losing other skills.
What this means in practice
- •For urban infrastructure teams: Automatically detect and describe long-term structural changes in city images to support maintenance and monitoring efforts.
- •For mapping service engineers: Develop tools that highlight and narrate differences in revisited geographic images to keep digital maps more accurate over time.
Authors
Benedetta Liberatori, Nermin Samet, Paolo Rota, Matthieu Cord, Elisa Ricci, Andrei Bursuc, Monika Wysoczańska
Abstract
Long-term change understanding from images of the same place revisited over time is a challenging task with applications in map maintenance and urban infrastructure monitoring. Prior work addresses it either through pixel-level prediction or difference captioning, neither of which is sufficient to reliably measure how well models detect and describe such changes. We introduce SPOT-THE-SHIFT, a human-verified benchmark for grounded image difference captioning of long-term changes in real-world driving scenes. Our benchmark provides natural language captions and spatial masks for structural changes across each image pair. We further propose an evaluation protocol that reliably assesses models' captioning ability, validated through human studies. Benchmarking state-of-the-art MLLMs, we find that models struggle with the fine-grained multi-image spatial capability required for this task. Finally, we develop a synthetic data generation pipeline that improves an off-the-shelf MLLM without sacrificing general capabilities.