EgoAlign improves 3D scene understanding without text descriptions
EgoAlign: Situated 3D Understanding without Situation Descriptions
Computer Vision and Pattern Recognition
Summary
Understanding a 3D scene from a first-person point of view often needs descriptions of where the person is looking or standing. The authors found that models relying on text descriptions aren't always practical and that simply using numbers for the person's position hasn't worked well. They created EgoAlign, a new method that teaches machines to understand 3D scenes based on the person's location and direction alone. This method does better at answering questions about 3D scenes without needing any text describing the situation.
What this means in practice
- •For robot developers: Enable robots to understand their surroundings and answer questions without requiring verbal situation descriptions by using numerical position data with EgoAlign.
- •For augmented reality engineers: Improve spatial understanding in AR devices to interact with users based solely on the device's position and orientation without relying on textual scene explanations.
Authors
Yiyang Li, Tianyi Ma, Zehong Wang, Yanfang Ye
Abstract
Situated 3D Question Answering requires models to interpret a 3D scene relative to an agent's pose. Existing methods commonly convey that pose through a natural-language situation description, which may be unavailable or costly to obtain in real-world deployments. While some methods additionally incorporate numerical pose, we find that randomly shuffling the agent pose causes little performance degradation, indicating limited functional dependence on the numerical pose. To address this gap, we introduce EgoAlign, a simple 3D language model that represents a scene in an agent-centric coordinate frame, together with a dedicated egocentric alignment stage comprising four complementary tasks: object category recognition, egocentric direction prediction, metric distance estimation, and spatial relation prediction. This stage teaches the model to consistently interpret scene geometry relative to the agent's pose before task-specific fine-tuning. EgoAlign achieves the strongest performance on SQA3D, Real-3DQA, and MSQA among compared methods while remaining highly efficient. More importantly, removing textual situation descriptions yields comparable or better downstream performance, whereas shuffling EgoAlign's numerical pose causes substantial degradation. These results show that EgoAlign establishes agent-centric spatial context directly from numerical pose without requiring textual situation descriptions.