Vision language models struggle to decide when to gather more physical data
New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and LanguageMachine Learning
Summary
Sometimes, machines look at a picture from an experiment to answer a question about how something moves or reacts. The authors tested if these machines can decide when they have enough information or when they need to ask for more experiments to find out. They found that current vision language models usually don't change their choice even when the right answer depends on new data. This shows that these models have trouble figuring out when to gather new evidence, not just giving the right answer.
What this means in practice
- •For robotics engineers: Improve robot systems that decide when to perform new physical measurements before acting in uncertain environments.
- •For automated testing teams: Design better AI tools that know when to gather additional test data to increase confidence in physical system predictions.
Authors
Sourajit Saha, Shubhashis Roy Dipta, Nobin Sarwar, Shaswati Saha, Yuxuan Jiang, Siyuan Li, Qiheng Wang
Abstract
A model first sees an image from one physical measurement experiment, such as how far a block coasted, and must answer a question about a new trial, such as whether the block will pass a target after a fixed push. The initial experiment may provide enough information to answer, or the model may need another measurement, such as the object's mass, friction, restitution, or spring stiffness. We study whether vision language models can decide when to answer immediately and, when more evidence is needed, which experiment to perform. Current physical reasoning benchmarks usually evaluate only the final answer, so they do not directly measure this decision-making ability. We introduce a controlled evaluation where each problem provides one measurement image and four possible physical worlds created by combining two possible masses and two possible values of another relevant property. The model must either stop and answer or select the cheapest additional experiment that can resolve the question. We construct matched problem pairs where changing either the observed measurement or the question changes the optimal action. Since all possible worlds and experiment costs are known, we can explicitly determine the optimal choice. Across six open models and 144 physical parameter sets, direct responses repeat the same action for 95.1% to 100% of image pairs even when the correct action changes. Brief reasoning improves action switching, but the best model makes both decisions correctly for only 5.9% of image pairs. Additional analysis reveals failures in measurement interpretation, physical reasoning, and response formatting. By evaluating evidence selection separately from final answers, our benchmark reveals limitations in physical reasoning that conventional answer accuracy can overlook.