Vision language models improve understanding of negation questions

Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models

Computer Vision and Pattern Recognition

Summary

Vision-language models are good at answering many visual questions but have trouble with questions that involve negation, like 'What is not in the picture?'. The authors introduce a new method called Skeleton-and-Strategy Prompting (SSP) that helps these models understand negation without any extra training. SSP works by breaking down the question into a basic structure, finding similar questions, and then teaching the model a simple way to answer negation questions. Tests show this method improves the model’s accuracy on negation tasks efficiently.

What this means in practice

Authors

Yuliang Cai, Mohammad Rostami, Jesse Thomason

Abstract

Despite the strong performance of Vision-Language Models (VLMs) on a wide range of visual question answering (VQA) tasks, these models consistently struggle to understand negation and produce incorrect answers when questions involve negated clauses. To address this limitation, we propose Skeleton-and-Strategy Prompting (\textbf{SSP}), a training-free, in-context learning method that improves VLM negation understanding capabilities without any parameter updates. Given a negation question, our method first abstracts the underlying question structure into a skeleton, retrieves a small set of same-skeleton questions from a lightweight question pool, then prompts the VLM to analyze their shared negation pattern and synthesize a single-sentence answering strategy. The skeleton and strategy are prepended to the test sample to guide the model correctly tackle the negation problems. Experiments on multiple negation VQA benchmarks show that SSP achieves state-of-the-art performance on negation-focused VQA tasks while remaining computationally efficient.