Learning reference frames helps models see from different viewpoints
LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning
Computer Vision and Pattern Recognition
Summary
Sometimes computers find it hard to understand what things look like from another person’s point of view. The authors came up with a way to teach computer vision and language models to imagine and mark a viewpoint based on a specific object in a picture. This helps the model think from that object's perspective instead of just the camera's view. They tested this approach on several tasks and found it helped models do better at understanding spatial relationships from different angles.
What this means in practice
- •For augmented reality developers: Align virtual objects accurately in a scene by reasoning from a user-specified viewpoint using learned reference frames.
- •For robotics programmers: Improve robot navigation and manipulation by enabling systems to interpret spatial relations relative to objects rather than fixed camera views.
Authors
Bang Xiao, Wenqi Jia, Ozgur Kara, Tiancheng Shen, Yibo Yang, Bolin Lai, Junho Kim, James Matthew Rehg
Abstract
Perspective taking is a fundamental component of spatial intelligence, requiring models interpret spatial relations from a specified viewpoint, such as that of another entity or an imagined observer. Despite the increasing spatial reasoning capabilities of Vision-Language Models (VLMs), they still struggle with perspective taking, often defaulting to the camera viewpoint when a query requires reasoning from a different perspective. We introduce Learning Reference Coordinate Frames for Perspective Taking (LeRF), a framework that trains VLMs to construct and use explicit reference frames for viewpoint-dependent reasoning. Given an image and a query, LeRF decides whether a coordinate frame is necessary. If so, it grounds the reference entity and predicts the frame's origin and entity-centered reference frame. A lightweight renderer overlays the frame onto the image, enabling subsequent reasoning over these visual cues without external perception models or explicit 3D reconstruction. To learn this process, we first perform supervised fine-tuning to teach selective tool invocation and reference coordinate frame prediction, followed by reinforcement learning on spatial VQA pairs to improve frame-guided reasoning. Across diverse perspective-taking benchmarks, LeRF consistently improves over its backbone and achieves strong performance against existing open-source methods. Further evaluations also show improved reference-frame grounding and orientation estimation, supporting the effectiveness of learned reference frames for viewpoint-dependent reasoning.