Vision language models check image detail changes in super-resolution
VHA: VQA-based Hallucination Assessment for Generative Image Super-Resolution
Computer Vision and Pattern Recognition
Summary
Generative image super-resolution can make blurry images clearer but sometimes adds details that aren't really there, which are called hallucinations. The authors created a new way called VHA that uses models which understand both images and language to ask and answer questions about two images, checking if the details in a clearer image match the original. They also made a faster version, VHA+, that learns to predict these checks without slow computations. They showed their method matches what people see when judging whether details were wrongly added or changed.
What this means in practice
- •For image processing teams: Evaluate if enhanced images maintain truthful content without adding false details after super-resolution.
- •For quality assurance engineers: Automate detection of unwanted alterations in images produced by generative enhancement pipelines.
Authors
Donghun Ryou, Inju Ha, Jaeyoo Park, Bohyung Han
Abstract
Generative image super-resolution produces visually realistic high-resolution images, but may also introduce hallucinations by adding unsupported content or altering existing details. Because existing super-resolution metrics fail to measure whether generated details remain grounded in the reference image, we present a hallucination assessment technique based on bidirectional visual question answering (VQA), which we term VQA-based Hallucination Assessment (VHA). The proposed approach uses a vision-language model (VLM) to determine whether the semantic content and low-level visual evidence in a super-resolved image are supported by its ground-truth counterpart. To this end, it constructs question--answer pairs from each image and evaluates them on both images. Discrepancies between the resulting scores in both directions reveal missing or altered ground-truth information and unsupported additions. However, obtaining these scores requires computationally expensive VLM inference. We therefore develop VHA+, a learned, lightweight model that directly predicts VHA scores from ground-truth and super-resolved image pairs. We construct a hallucination benchmark, VHA-bench, through a human study and show that both VHA and VHA+ correlate highly with human judgments of hallucination. The code, benchmark, and model weights are available at https://github.com/dhryougit/Hallucination-Assessment