Vision language models improve answer selection with visual quality scores
VisConf: Quality-Aware Consensus for Best-of-N in Vision-Language Model Reasoning
Computer Vision and Pattern Recognition
Summary
Sometimes, AI systems that understand both pictures and text try multiple possible answers before picking the best one. But deciding which answer is right can be tricky because some answers don't pay enough attention to the picture or aren't confident enough. The authors found that looking at both how much an answer focuses on the image and how confident the model is together gives a better sense of quality. They created a new way to score answers based on these two signals and showed it picks better answers in several tests.
What this means in practice
- •For multimodal ai developers: Improve answer choice in vision-language systems by weighting candidate responses using combined visual attention and confidence signals.
- •For automated math problem solvers: Enhance problem-solving accuracy by better identifying correct reasoning paths involving visual inputs in mathematical datasets.
Authors
Muhammad Usman, Muhammad Qasim Ayub, Usman Ahad, Muhammad Tahir
Abstract
Best-of-N methods improve text-only LLM reasoning by sampling multiple trajectories before selecting an answer. For Vision-Language Models (VLMs), answer frequency and model confidence alone can be insufficient, as sampled generations also vary in their engagement with the visual input. We find that distributional confidence and visual attention are complementary indicators of rollout quality. Each is limited on its own, while trajectories that show strong signals in both are more likely to be correct. Motivated by this, we introduce VisConf, a rollout-quality score that multiplicatively combines visual engagement with candidate-calibrated Self-Certainty, penalizing rollouts that are weak in either signal. We incorporate Rank-Weighted Consensus (RWC), a tuning-free aggregation rule that converts within-question VisConf ranks into answer-level vote weights. Across three VLMs on MathVista, MMMU-Pro, and MMStar, VisConf with RWC consistently outperforms strong BoN baselines. Our analysis shows particularly large gains when correct trajectories are present but underrepresented in the candidate pool.