Vision language models improve verification of left right claims using symmetry
Symmetry-Aware Likelihood-Orbit Aggregation for Selective Left-Right Claim Verification
Computer Vision and Pattern Recognition
Summary
Left-right claims in images (like "the cup is on the left of the plate") are hard for existing vision-language models to verify accurately. The authors developed a method called Relation-Orbit that combines likelihood measurements from eight symmetry-related views without needing extra training. This method uses reflection and entity swapping concepts to better decide when to accept such claims. It was tested on several datasets and models, showing improved selective verification performance and better error detection compared to alternatives.
What this means in practice
- •For computer vision engineers: Improve accuracy and confidence calibration when verifying spatial relationships in image understanding tasks using frozen vision-language models.
- •For natural language processing developers: Enhance multimodal model outputs by robustly integrating symmetry-aware likelihood contrasts to verify directional or relational claims in visual question answering.
Authors
Zhouzhi Xiong, Chuxi Zhang, Weizhen He, Yi Chen, Qi Li, Donglian Qi
Abstract
Frozen vision-language models (VLMs) remain unreliable on fine-grained left-right claims, and raw claim likelihoods need not reliably rank verification errors. After a horizontal-reflection intervention is fixed, how should its induced likelihood measurements be combined into a selective verification signal? We introduce Relation-Orbit, a closed-form contrast with no learned fusion parameters that assigns eight normalized likelihoods to query-supporting and counterfactual roles determined by reflection, inverse relation, and entity exchange. A claim is asserted only when the signed contrast exceeds a threshold selected on held-out data using pointwise Clopper-Pearson upper confidence bounds. On VSR and GQA across four frozen VLMs, Relation-Orbit yields higher mean test coverage at a 10% selective-risk calibration target than an all-eight Orbit-Max baseline in all eight dataset-backbone settings; gains over a nearly abstain-all one-sided intervention score are reported separately. A separate LLaVA-1.5/COCO evaluation, reduced-orbit controls, and a two-sided partition diagnostic further characterize the structural advantage.