Conditional Visual Evidence Utility: State-Dependent Rank Reversals in Frozen Vision-Language Encoders
Computer Vision and Pattern Recognition
Summary
The authors studied how the importance of different visual clues like color, shape, and texture can change depending on what clues have already been seen during a visual search task. They found that ranking clues by static importance doesn’t always reflect their value as more clues are revealed. Their experiments showed that updating the clue order after seeing initial evidence can improve decision-making under various testing conditions. This suggests that importance should be assessed dynamically rather than fixed, helping future designs for adaptive visual search methods.
Authors
Yunxuan Fang, Xinhe Wang
Abstract
Static importance scores compress visual evidence into a single ranking, but the value of remaining evidence can change after one cue has been observed. We study this possibility in controlled compositional visual search, where color, shape, and texture evidence can be independently exposed and their conditional marginal utility measured across acquisition states. In a held-out confirmation on 800 scenes, frozen OpenCLIP and SigLIP exhibit robust state-dependent rank reversals that concentrate in candidate-overlap regimes designed to induce ordering changes. The structure persists across two evidence-accumulation constructions and ten equivalent query wordings, but disappears under query-scene derangement. We also ask whether these reversals matter for decisions. In a post-confirmation exploratory matched-first-action analysis, reranking only after the first acquisition yields positive step-2 utility when decisions are selected under one evidence mode, wording, or backbone and evaluated under another. Together, these results show that evidence importance is state-dependent in this controlled setup and that updating an evidence ordering can retain decision-relevant value across evaluator changes. They motivate evaluating vision-language evidence use conditionally rather than through a single static ranking, while providing a measurable target for future adaptive evidence-selection methods.