Audio visual descriptions improve understanding of urban scenes
AVSD-Scenes: A Dataset for Audio-Visual Description of Urban Scenes
Artificial IntelligenceSound
Summary
Understanding a busy city scene can be easier if you use both sounds and pictures together. This paper presents a new collection of city scene descriptions that combine what you see and hear. The authors created these descriptions using specialized AI models for sound and images, then merged them to capture more details. Their tests show that using both sounds and visuals helps computers recognize and describe city scenes better than using either alone.
What this means in practice
- •For autonomous vehicle developers: Improve urban environment perception by integrating combined audio and visual scene descriptions for better scene recognition.
- •For smart city infrastructure teams: Enhance monitoring systems by using enriched audio-visual descriptions to detect and classify urban scenes more accurately.
Authors
Dhanunjaya Varma Devalraju, Arshdeep Singh, Mark D. Plumbley
Abstract
Natural language descriptions can provide rich semantic representations of audio-visual urban scenes, yet datasets that jointly describe both auditory and visual information remain limited. In this paper, we introduce AVSD-Scenes, a paired audio-visual scene description dataset for urban environments. The dataset contains 12,291 audio-visual scene descriptions generated from the TAU Urban Audio-Visual Scenes dataset. To construct the dataset, we first generate audio- and visual-based descriptions using Qwen2-Audio-7B and Qwen2.5-VL-7B, respectively. These modality-specific descriptions are then combined using large language models, namely Qwen3-14B, Mistral-Small-3.2-24B-Instruct-2506, and Gemma-3-27B-it, to produce multimodal descriptions that capture complementary information from both modalities. We benchmark AVSD-Scenes using semantic alignment, cross-modal retrieval, scene classification, LLM-as-a-judge evaluation, and human subjective assessment. Results show that multimodal descriptions improve semantic alignment and cross-modal retrieval performance compared with modality-specific descriptions while preserving strong scene-discriminative information. The generated descriptions achieve up to 94.5% accuracy in urban scene classification, while combining audio, visual, and description embeddings further improves accuracy to 95.4%. Furthermore, the descriptions remain highly scene-discriminative even when scene labels are removed from the prompting instructions, indicating that they capture semantic information derived from the audio-visual content rather than merely reflecting label information.