Grounded reasoning works best for visual tasks needing indescribable references
Where and Why is Grounded Reasoning Useful?
Computer Vision and Pattern Recognition
Summary
Some computer models solve visual puzzles by pointing to important spots in images as they think through problems. The authors studied when this pointing, called grounded reasoning, really helps. They found it mainly improves counting objects but doesn’t offer much help in other common spatial tasks. Grounded reasoning shines when the task depends on visual details that are hard to put into words. To test this, the authors created a new challenge called BrickVision that needs these tricky visual clues to solve.
What this means in practice
- •For computer vision engineers: Design visual reasoning systems that use grounded reasoning specifically for tasks needing hard-to-describe visual clues.
- •For data scientists: Benchmark visual AI models on the BrickVision dataset to test ability on problems requiring undescribable visual references.
Authors
David Yan, Le Xue, Forrest Huang, Qiuyi Zhang, Dustin Tran, Andrew M. Dai, Yinfei Yang
Abstract
Grounded reasoning, where a model outputs text coordinates of relevant locations in chain-of-thought reasoning, is a common strategy for solving complex visual reasoning problems. However, the benefits and limitations of grounded reasoning are understudied. We study where and why grounded reasoning is most useful. To do so, we introduce a chain-of-thought analysis method to measure how much a model uses its outputted grounding in its thinking trace and perform a detailed ablation study of a grounded reasoning training recipe. We find that grounded reasoning improves counting but shows no significant gain over baselines without coordinate grounding across a wide range of spatial reasoning and perception benchmarks. Using several synthetic tasks, we show that grounded reasoning is most useful when a task requires undescribable references, which are visual concepts that are difficult to describe in language. However, we observe that many existing visual reasoning benchmarks are solvable with only describable text references. To close this gap, we introduce BrickVision, a new benchmark that requires undescribable visual references to solve.