Large vision language models improve spatial reasoning with confidence aware grounding
SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding where things are in pictures is hard for many AI models, especially when they need to explain their answers clearly. The authors designed a system called SpatialCORE that helps AI models be more confident about identifying exact locations of objects before answering questions. This method makes the model learn to focus on accurately finding relevant objects, improving how well it understands space in images. SpatialCORE works well even when tested on new, different images the model hasn’t seen before.
What this means in practice
- •For image captioning developers: Enhance accuracy in describing spatial relationships in images by improving model confidence in object localization during reasoning.
- •For augmented reality software engineers: Improve spatial understanding of AR systems for more reliable object identification and interaction in real-world scenes.
Authors
Rafi Ibn Sultan, Xiangyu Zhou, Md. Sajid Alam Chowdhury, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu
Abstract
Large Vision-Language Models (LVLMs) have made remarkable progress across visual perception tasks, yet spatial reasoning remains a persistent weakness, especially for questions that require reasoning over visual space. Recent spatial-reasoning methods incorporate generated grounding, where models predict bounding boxes, masks, or other localization outputs for task-relevant objects as part of their reasoning trace. However, these approaches typically optimize final-answer correctness alone, allowing correct answers to be rewarded even when the model does not reason from confidently localized task-relevant objects. We introduce SpatialCORE (Spatially COnfident REasoning), a post-training framework that turns the model's own confidence in generated grounding into a learning signal for spatial reasoning. Its central idea is to reinforce grounding that is both accurate and confident, encouraging the model to reason from confidently localized task-relevant objects. SpatialCORE realizes this through a self-regulating spatial reward that weights each predicted bounding box's matching quality by its coordinate-token confidence. An answer gate further ties grounding optimization to final-answer correctness. SpatialCORE achieves state-of-the-art results among open-source and specialized spatial reasoning models across diverse benchmarks, and transfers effectively in zero-shot settings to unseen data distributions. The source code is available at https://github.com/rafiibnsultan/SpatialCORE.