Zero shot object detection improves identifying unknown items safely
SAM3-O2D2: Zero-Shot Object Out-of-Distribution Detection by Object Class Prompting of the SAM3-Image Model
Computer Vision and Pattern Recognition
Summary
Recognizing objects that a system has never seen before is important for safety in places like self-driving cars or security cameras. This paper presents a new method that uses two different AI models together to check if a detected object matches what is actually in the image. If the second model does not confirm the object, it means the object might be unknown or out-of-distribution. The proposed method works faster and better than previous approaches without needing extra training. The authors tested it on several datasets and found it to be more accurate at spotting unfamiliar objects.
object detectionout-of-distribution detectionzero-shot learningfoundation modelsSAM3-image modelCLIPAuROCFPR95Pascal-VOCBDD100K
Authors
Lucas Görnhardt, Timo Bartels, Tim Fingscheidt
Abstract
Object detectors have shown remarkable performance in various fields, among these medical imaging, surveillance, and autonomous driving. However, they are prone to overconfidence when encountering unseen objects in real-world deployments, causing potential safety issues. To address this, detecting out-of-distribution (OOD) objects is essential for reliable object detection. Modern approaches leverage the broad semantic knowledge of foundation models such as CLIP for post-hoc few- and zero-shot OOD detection. However, these methods typically perform OOD assessment in feature space, which can be sensitive to object detector localization errors and variations in object appearance. Moreover, the current state-of-the-art (SOTA) zero-shot method performs computationally costly diffusion in inference. In this work, for our proposed zero-shot object OOD detection method SAM3-O2D2, we employ the SAM3-image foundation model in an efficient manner. Specifically, we prompt SAM3 only with the object detector's predicted classes and compare the predictions of the object detector and SAM3. An object is in-distribution (ID), if SAM3 also detects an object at the corresponding location. If SAM3 does not detect the prompted object, this indicates a mismatch between the detector's prediction and the image content, suggesting that the object is OOD. Experimental results show that our method significantly surpasses the so-far zero-shot SOTA method. Specifically, we achieve new SOTA AuROC and FPR95 metrics over both ID datasets Pascal-VOC and BDD100K and both OOD datasets MS-COCO and OpenImages.