Object detector mismatch helps catch unknown objects fast
SAM3-O2D2: Zero-Shot Object Out-of-Distribution Detection by Object Class Prompting of the SAM3-Image Model
Computer Vision and Pattern Recognition
Summary
Object detectors can wrongly identify items they've never seen before, which can cause problems in real-world uses like driving or security. The paper shows a new method to spot these unknown objects without any prior examples. The method uses a model called SAM3, comparing its guesses with those from the main detector to see if they agree. If they don't match, it suggests the object is unfamiliar. This approach outperforms earlier methods while being more efficient.
What this means in practice
- •For autonomous vehicle engineers: Improve the detection of unexpected road objects by quickly identifying unknown items that standard detectors misclassify.
- •For security surveillance teams: Enhance alerts for unusual or unknown objects in camera feeds by validating detector predictions against SAM3-based checks.
Authors
Lucas Görnhardt, Timo Bartels, Tim Fingscheidt
Abstract
Object detectors have shown remarkable performance in various fields, among these medical imaging, surveillance, and autonomous driving. However, they are prone to overconfidence when encountering unseen objects in real-world deployments, causing potential safety issues. To address this, detecting out-of-distribution (OOD) objects is essential for reliable object detection. Modern approaches leverage the broad semantic knowledge of foundation models such as CLIP for post-hoc few- and zero-shot OOD detection. However, these methods typically perform OOD assessment in feature space, which can be sensitive to object detector localization errors and variations in object appearance. Moreover, the current state-of-the-art (SOTA) zero-shot method performs computationally costly diffusion in inference. In this work, for our proposed zero-shot object OOD detection method SAM3-O2D2, we employ the SAM3-image foundation model in an efficient manner. Specifically, we prompt SAM3 only with the object detector's predicted classes and compare the predictions of the object detector and SAM3. An object is in-distribution (ID), if SAM3 also detects an object at the corresponding location. If SAM3 does not detect the prompted object, this indicates a mismatch between the detector's prediction and the image content, suggesting that the object is OOD. Experimental results show that our method significantly surpasses the so-far zero-shot SOTA method. Specifically, we achieve new SOTA AuROC and FPR95 metrics over both ID datasets Pascal-VOC and BDD100K and both OOD datasets MS-COCO and OpenImages.