HalDetect detects visual hallucination in images with QLoRA fine tuning
HALDETECT at ImageEval 2026 Shared Tasks: Answer-First Contrastive Grounding with QLoRA
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Some large AI models can mistakenly add details that aren't really in an image, which can cause problems when interpreting pictures. The authors created HALDETECT to pick out the one true statement about an image from three options that all seem believable but only one is visually accurate. They improve how the model learns by focusing on contrast between choices, giving the answer first, and fine-tuning a language-vision model with a technique called QLoRA. Their system performed well in a competition, ranking third among eight teams. They also found that how the answer is ordered can affect results more than making the model bigger.
What this means in practice
- •For multimodal ai developers: Improve visual hallucination detection in AI systems that interpret images and text together by using contrastive answer-first fine-tuning techniques.
- •For content moderation teams: Use systems like HALDETECT to identify when AI-generated image descriptions contain false visual details, helping ensure accuracy in multimedia content.
Authors
Syed Mohaiminul Hoque, Md Sakhawat Hossain
Abstract
Large multimodal models tend to hallucinate visual detail fluently, which limits their deployment for fine-grained interpretation. We present HALDETECT, our system for the English hallucination-detection track (Task 1b) of ImageEval 2026, in which a system must identify, from an image and three culturally plausible statements, the single visually grounded one. We frame the item as one contrastive decision, emit the answer before its explanation, and structure reasoning around colour/texture, shape/form, and context. Our best submitted adapter fine-tunes Qwen2.5-VL-7B-Instruct with 4-bit QLoRA while freezing the vision encoder and reaches Contrastive Instability (CI) 0.035 on the 1,000-item test set; we placed third of eight teams. Development experiments show that answer order can matter more than model scale and that adaptation beats prompting alone. Retrospective paired analysis of the released gold labels confirms the QLoRA gain over the best prompt but not the small gap between the devtest-selected and best-test adapters, and reseeding all four training sizes shows that the apparent data-scaling curve does not survive a seed change. The 35 residual errors are culturally plausible function, material, and recognition distinctions; naive adapter voting does not help.