SKstars combines zero-shot and adapted models to spot image hallucinations
SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Models
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Sometimes, AI models that describe images make mistakes called hallucinations, where they say things that aren't actually in the picture. The SKstars team built a system that uses two different AI image-text models working together to catch these mistakes more accurately. They combined a very large model that works without specific training and a smaller model fine-tuned for the task, and then carefully merged their outputs. Their system ranked in the middle among many teams, showing this method works but also that it's hard to improve much on hidden test data.
What this means in practice
- •For ai product developers: Integrate combined zero-shot and adapted vision-language models to better detect and label hallucinated content in automated image descriptions.$Commercial implications: This approach enables commercial products that automatically verify image caption accuracy to reduce user confusion caused by hallucinated details.
- •For software quality teams: Use the ensemble method to improve monitoring and quality checks of vision-language models used in AI-assisted content generation pipelines.
Authors
Ali Athar, Imran Ahsan, Joon-Yong Jung
Abstract
This paper describes the SKstars submission to SHROOM-Visions 2026, a shared task on fine-grained hallucination detection in large vision-language model outputs. The task requires systems to identify hallucinated character spans, assign hallucination categories, and provide confidence estimates for their predictions. Our approach combines zero-shot predictions from Qwen2.5-VL-72B-Instruct with those of a LoRA-adapted Qwen2.5-VL-7B-Instruct model. The outputs of the two models are integrated through a lightweight ensemble procedure, followed by span refinement and confidence adjustment. We evaluate the main system components on a small internal development subset and report the performance of the submitted system on the official English test set. SKstars achieved a Cor+Lbl score of 0.2902, ranking 15th among 29 teams, and obtained Cor and IoU scores of 0.3642 and 0.3151, respectively, ranking 18th on both metrics. The results show that combining a large zero-shot model with a smaller adapted model provides a practical framework for multilingual and fine-grained hallucination localization, while also highlighting the difficulty of transferring development-set improvements to hidden test data. Code and predictions: https://github.com/aliathar1401/SK-Stars-shroom-visions-2026