SceneBind: Binding What and Where Across Vision, Audio and Language
2026-07-16 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceMultimediaSound
AI summaryⓘ
The authors introduce SceneBind, a system that understands realistic scenes by combining what objects are present and where they are using vision, audio, and language together. Unlike previous methods that mainly focus on identifying objects, SceneBind also captures spatial details about each object and the uncertainty in these details. They created a new dataset with real-world audio and visual information, including detailed annotations, to train and test their approach. SceneBind enables better cross-modal retrieval and object localization without extra training for new tasks.
omni-modal representationsemantic embedding3D spatial understandingcross-modal retrievalobject groundingbinaural audio-visual datasetsemantic-spatial slotszero-shot transferlarge-scale pretrained encoders
Authors
Mingfei Chen, Zijun Cui, Ruoke Zhang, Hyeonggon Ryu, Eli Shlizerman
Abstract
We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, we curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for aligning semantic and spatial signals across modalities. SceneBind is compatible with large-scale pretrained semantic encoders, adds lightweight spatial modeling with only a few additional tokens. It achieves state-of-the-art scene and spatial retrieval while enabling strong zero-shot transfer to downstream tasks such as audio-visual localization.