Localization accuracy drops when naming falls outside detector categories
Attribution Gaps in Zero-Training LLM+OVOD Pipelines: A Fine-Grained Analysis of the CAAP--SNAP Discrepancy
Computer Vision and Pattern RecognitionComputation and Language
Summary
The paper looks at a mismatch in how well AI systems find objects and how well they name them without extra training. It finds that when the AI uses words outside the detector’s known categories, it struggles to link the word to the right object. But this often isn’t because the AI is wrong—it’s because the labeled data only covers a fixed set of object types. The authors tested this on a large image set and with different AI models, showing the problem is common and tied to limits in the labels, not AI errors.
What this means in practice
- •For computer vision engineers: Improve evaluation methods by accounting for annotation limits, reducing false flags of AI object grounding errors in detection pipelines.
- •For multimodal ai developers: Design better AI systems by recognizing that naming errors often reflect label set constraints rather than real failures, improving model robustness.
Authors
Yu-Feng Yen
Abstract
LAOD and similar zero-training LLM+open-vocabulary-detector (OVOD) pipelines score two things separately: class-agnostic localization accuracy (CAAP) and semantic naming accuracy (SNAP). The two consistently diverge, and nobody has asked why. This paper asks why, on the full 5,000-image COCO-Val split (27,273 detections) rather than the small subset the original work evaluated on. Object visual complexity turns out not to be the driver -- small and occluded objects are, if anything, localized better than large ones. Vocabulary novelty is: once the LLM's wording falls outside the detector's native category set, localization accuracy falls from 80.9% to 31.6%. That drop is not spread evenly across unfamiliar phrasing, though. Almost all of it comes from cases where the novel wording actually names a different object than the one COCO annotated (true synonyms still score 89.3%; semantically unrelated "noise" labels score 12.0%). A closer look at a further failure subset tells a similar story: 78-88% of what looks like complete localization failure is really the model correctly finding a real object that COCO's non-exhaustive 80-category scheme simply never labeled, not hallucination. Swap the detector backbone (YOLO-World for Grounding DINO) or the LLM (Gemma-3 for Qwen2.5-VL) and both the effect and its rough size hold up, so this looks like a general property of the pipeline family rather than a quirk of one model pairing. The upshot is that a large share of the apparent CAAP--SNAP gap traces back to closed-category annotation limits rather than a real grounding failure, which matters for how we detect hallucination, analyze failure modes, and design evaluation for grounded multimodal systems meant to work in the open world.