Small and large AI models combined to detect visual text errors
Two-Token Features and Small-Large Ensembles for VLM Hallucination Detection
Computation and Language
Summary
Sometimes AI that looks at pictures and reads text can make up information that isn’t there, called hallucination. The authors made a system that uses a small AI spot-checking one piece of text at a time, plus a big AI that judges the whole thing without retraining. They combined these two to better detect when the AI is making up text in images. Their method worked well in a competition testing multiple languages.
What this means in practice
- •For software engineers: Improve AI systems that read text in images by reducing false text output using combined small and large model checks.
- •For content moderation teams: Detect and flag false or fabricated text in images automatically to uphold content accuracy on social platforms.
Authors
Eli Schwartz
Abstract
We present our system for the SHROOM-Visions 2026 shared task on character-level VLM hallucination detection. A small ($4$B-parameter) VLM is fine-tuned as a per-token classifier reading a two-token feature from its own hidden states, and is ensembled with a $\sim$400B zero-shot VLM judge at prediction time. Both components see off-the-shelf OCR of any visible in-image text. We use synthetic hallucination data generated by the large model as a source of ensemble diversity, and use validation to select feature layer, training data and OCR grounding. Our official entry reaches mean Cor $0.487$ / Cor-lbl $0.387$ on the hidden test set, placing $6$th/$28$ (EN), $6$th/$21$ (FR), $8$th/$21$ (IT) and $7$th/$22$ (ZH) on the task's primary Cor-lbl metric.