Animal identification accuracy tested by measuring use of real features
CounterFactual Retrieval for Evidence-Aware Individual Animal Identification
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Identifying individual animals automatically is getting better, but current tests don’t show whether a model really looks at the animal itself or just background clues. The authors created a new way to test this by showing the model different parts of an image: the whole picture, just the animal, or just the background. This helps reveal if a model truly recognizes the animal or is relying on other signals. They also provided lots of image masks and tools so others can use this testing approach.
What this means in practice
- •For wildlife monitoring teams: Evaluate and improve animal ID models by testing if they truly use animal features or rely on background cues.
- •For security camera integrators: Confirm that individual animal recognition systems deployed for perimeter monitoring identify animals based on actual features.
Authors
Ankit K. Upadhyay, Aleksandr Algasov, Ekaterina Nepovinnykh, Fedor Zolotarev, Tuomas Eerola, Heikki Kälviäinen, Charles V. Stewart, Lukas Picek
Abstract
Automated identification of individual animals is moving from small curated studies toward shared benchmarks and reusable foundation models. This progress is useful, but current evaluation leaves the evidence behind a match unresolved, i.e., high retrieval accuracy does not show whether a model used animal features or identity-correlated shortcuts. We address this with Counterfactual Foreground Retrieval (CFR), an evidence-aware evaluation protocol. CFR evaluates the same query-gallery task under different views: full image, foreground-only, and shortcut-only. The full-image measures standard retrieval performance; the foreground-only measures foreground-supported retrieval; and shortcut-only view measure how much identity signal is recoverable from background, or silhouette. This turns a retrieval score into an evidence profile: how well the model performs on the original images, how much of that performance remains when only the animal is visible, and how much identity signal remains recoverable when the animal is removed. We release around 0.5M masks for 34 datasets and code for mask generation and CFR evaluation.