AI summaryⓘ
The authors address the problem of large vision-language models sometimes mentioning objects that aren't actually shown in the image, called object hallucination. They note existing methods try to detect these errors using signals from inside the model, but these methods can't always tell if the object mention is truly supported by the image or just by the text context. To fix this, the authors propose VisER, which looks at each object mention from two angles: whether the image really shows the object (Visual Evidence) and whether the image supports the object more than the text prefix does (Visual Reliance). Their method improves detection accuracy without needing extra training or object verification steps.
Object HallucinationVision-Language ModelsToken LikelihoodAttention MechanismVisual EvidenceVisual RelianceImage-Text SimilarityAUROCAUPRTraining-Free Detection
Authors
Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani
Abstract
Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding. Recent training-free detectors use internal signals such as token likelihood, attention, visual confidence, or image-text similarity to identify hallucinated objects. These signals are useful, but they are often source-confounded. They measure how strongly an object is supported inside the model without distinguishing whether that support comes from object-specific visual evidence or the generated text prefix. In difficult cases, a hallucinated object can still receive high internal support because it fits the scene, is associated with nearby visual cues, or follows naturally from the generated text prefix. We propose VisER, a training-free two-sided metric for object-level hallucination detection. VisER evaluates each generated object mention from two complementary views. Visual Evidence measures whether object-context compatibility is backed by object-specific evidence from image tokens. Visual Reliance measures whether the object is supported more by the image than by the generated prefix. Combining these views gives a more source-aware grounding score, while avoiding additional object-level verification generations. Across multiple LVLMs and benchmarks, VisER improves AUROC and AUPR over a range of baselines.