Camera moves learned to answer questions by predicting useful new views
Rendering-Free Lookahead for Question-Guided Active Vision
Computer Vision and Pattern RecognitionRobotics
Summary
Sometimes a robot needs to move its camera to see important parts of a scene, like looking into a box to find out what’s inside. The authors worked on teaching a robot how to pick the best camera moves to answer questions about what it sees. They trained a model by letting it practice with 3D scenes where it could look ahead and see future views, then taught a student model to guess which moves would help answer questions without actually rendering those views. This approach helped the robot pick better camera moves and gave much better answers when tested in new environments.
What this means in practice
- •For robotics engineers: Automate camera control policies so robots efficiently gather visual information to answer questions without costly rendering during operation.
- •For surveillance system developers: Improve camera movement strategies to quickly focus on relevant areas when monitoring based on natural language queries or alerts.
Authors
Koya Sakamoto, Daichi Azuma, Shuhei Kurita, Naoya Chiba, Yusuke Iwasawa, Yutaka Matsuo, Taiki Miyanishi
Abstract
Active robot vision requires controlling the camera to reveal task-relevant information that is hidden from the current viewpoint. For example, determining what is inside a box may require raising the camera and looking down into it. For viewpoint-dependent question answering, the challenge is to select camera motions that expose the visual evidence needed to answer the question. Although vision-language models (VLMs) can interpret observed images, selecting such motions requires anticipating the usefulness of unseen views. We quantify this usefulness as answerability, a VLM's estimate that a view suffices to answer the question, and present Rendering-Free Lookahead (RFL), a viewpoint-selection policy that ranks candidate camera motions by predicted future answerability. RFL transfers visual lookahead from deployment to offline training. At training, a privileged teacher renders candidate future views in 3D Gaussian Splatting (3DGS) scenes and uses a frozen VLM to compute one- and two-step answerability targets. Through two-stage distillation, a student learns to predict these action values from the question, recent visual observations, and a candidate camera motion. At deployment, RFL uses these predicted values to select camera motions without rendering future views. On 377 E3VS-Bench test episodes in unseen environments, RFL improves the mean judge score by 43\% over a direct-action baseline using the same VLM. These results support learning camera-control policies from privileged visual lookahead for viewpoint-dependent question answering.