Vision language models improve understanding of negation questions
Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Vision-language models are good at answering many visual questions but have trouble with questions that involve negation, like 'What is not in the picture?'. The authors introduce a new method called Skeleton-and-Strategy Prompting (SSP) that helps these models understand negation without any extra training. SSP works by breaking down the question into a basic structure, finding similar questions, and then teaching the model a simple way to answer negation questions. Tests show this method improves the model’s accuracy on negation tasks efficiently.
What this means in practice
- •For ai application developers: Improve visual question answering systems by enhancing their handling of negation without retraining models.
- •For conversational ai teams: Develop smarter chatbots that correctly answer negation-based visual inquiries using a prompt-based method.
Authors
Yuliang Cai, Mohammad Rostami, Jesse Thomason
Abstract
Despite the strong performance of Vision-Language Models (VLMs) on a wide range of visual question answering (VQA) tasks, these models consistently struggle to understand negation and produce incorrect answers when questions involve negated clauses. To address this limitation, we propose Skeleton-and-Strategy Prompting (\textbf{SSP}), a training-free, in-context learning method that improves VLM negation understanding capabilities without any parameter updates. Given a negation question, our method first abstracts the underlying question structure into a skeleton, retrieves a small set of same-skeleton questions from a lightweight question pool, then prompts the VLM to analyze their shared negation pattern and synthesize a single-sentence answering strategy. The skeleton and strategy are prepended to the test sample to guide the model correctly tackle the negation problems. Experiments on multiple negation VQA benchmarks show that SSP achieves state-of-the-art performance on negation-focused VQA tasks while remaining computationally efficient.