FloorSAV enhances spatial reasoning by adding 2D floormaps to audio visual AI
FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs
Computer Vision and Pattern RecognitionMachine Learning
Summary
Understanding where things are and how they move in space is really hard for AI that looks and listens to the world around it. The authors created FloorSAV, which draws a simple 2D map from 3D data and sound information, giving the AI a clear view of its surroundings. This helps the AI answer questions about where objects are and how to get around in real places. They tested this new method on special tasks and found it improves the AI's ability to understand space better than before.
What this means in practice
- •For robotic system developers: Provide robots with improved situational awareness and navigation skills in real-world spaces by integrating synchronized 2D floormaps with audio and video inputs.
- •For augmented reality developers: Enhance AR devices’ understanding of spatial layouts by combining visual, audio, and geometric context for more accurate interactive experiences.
Authors
Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh
Abstract
While 3D spatial reasoning in dynamic egocentric environments is crucial for embodied intelligence, audio-visual large language models (AV-LLMs) lack explicit mechanisms to process and internalize global geometry directly from raw sensory streams. Existing approaches either require costly fine-tuning or underutilize the model's cross-modal reasoning capacities. In this paper, we propose FloorSAV, a novel framework that explicitly grounds spatial audio-visual context by rendering a dynamic 2D floormap. By integrating 3D point clouds, camera trajectories, spatial audio cues, and semantically grounded object landmarks, we inject this floormap into the AV-LLM as a synchronized stream with an egocentric video. AV-LLMs utilize their multi-modal capabilities to jointly reason over visual, auditory, and geometric cues in a single inference with floormap interpretation guidance. We further introduce SAVED-Bench (Spatial Audio-Visual Egocentric Benchmark with Dynamic Agents), constructing essential tasks of spatial capability in real-world scenarios: dynamic relativity, regional, and path reasoning QAs. FloorSAV improves AV-LLMs' spatial reasoning on various tasks from both SAVED-Bench and SAVVY-Bench. Studies with ground-truth floormaps demonstrate the substantial potential of FloorSAV with accurate spatial information.