Benchmark tests spatial reasoning using reference frames and images
Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Spatial questions depend on where you look and what visual clues you have. The authors created Frame Evidence, a set of questions with three different reference points and matched images to test how well AI understands these spatial clues. They found that even powerful language models like GPT struggle to get all the answers right when both the viewpoint and images change. This shows that just getting one answer right doesn’t mean the AI fully understands the spatial context.
What this means in practice
- •For ai developers: Test and improve models’ understanding of spatial relations and reference frames using paired images and questions.
- •For robotic navigation teams: Evaluate robots’ ability to interpret spatial commands consistently from different viewpoints and visual contexts.
Authors
Chunxi Luo, Qishuai Ding, Xingchen Peng, Hengwei Ye, Ke Zhang, Wenxin Lian, Tianyi Lu, Chunlong Yu, Zhenya Lin
Abstract
A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctness persists when either changes. We introduce Frame Evidence, a benchmark of 320 matched triplets and 960 Chinese multiple-choice questions from captured and rendered scenes. One family compares three grounded reference frames over fixed images. The other pairs image deletion annotated as insufficient with answer-preserving addition. Joint metrics require correct answers across all three frames and paired evidence variants. Across 20 configurations on 288 held-out questions, GPT three-shot reaches 50.00\% item accuracy, but the best three-frame group accuracy is 5/48 (10.42\%), reached by GPT two-shot. Few-shot prompting and task-specific adaptation improve item accuracy without consistently improving joint correctness. Input controls show where images support matched success, while repeated executions reveal answer instability beneath similar scores.