HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and Language
AI summaryⓘ
The authors found that vision-language models (VLMs) struggle because their vision components stick to fixed ways of processing image details across different tasks. They introduced a method called HAFI-VLM that adjusts how image information is extracted by focusing on different frequency details (low to high) depending on the task, guided by the text input. This approach helps the models better understand images in context and reduces errors without needing more detailed images. Their experiments showed that this task-aware frequency adjustment improves visual question answering and reduces hallucinations effectively.
vision-language modelsspectral responsefrequency analysiscross-attentionvisual question answeringfine-tuninghierarchical adaptive frequency injectionlanguage model adaptershallucination robustnessimage encoding
Authors
Jin Cui, Chuanchang Su, Jiayi Lu, Xinyue Long, Boran Zhao, Pengju Ren
Abstract
Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence. We identify a previously overlooked cause: spectral response rigidity. Despite substantial frequency variation across images and tasks, pretrained vision encoders exhibit persistent, encoder-specific layerwise spectral profiles that change only marginally under downstream fine-tuning. Since pretrained vision encoders only receive images, they cannot adapt spectral extraction to the evidence required by the current query. We therefore propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation. Hierarchical Adaptive Frequency Injection (HAFI) retrieves complementary low-, mid-, and high-frequency evidence at multiple encoder depths using text-modulated, spatially aligned cross-attention. A Visual Enrichment Layer Adapter further recalibrates shallow LLM attention to effectively utilize the enriched visual tokens. Experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate consistent improvements in general VQA, text-rich understanding, and hallucination robustness, outperforming representation-level enhancement methods and most resolution- or cropping-based approaches without additional high-resolution encoding. Mechanistic analyses show that HAFI restores task-dependent spectral allocation while retaining semantic attention, establishing frequency enrichment as a distinct and effective route for improving VLM perception.