Benchmark reveals challenges in reasoning for audio visual video search
RVG-Bench: A Benchmark for Reasoning-Driven Audio-Visual Temporal Grounding
Computer Vision and Pattern Recognition
Summary
Many current video search tools find parts of videos by matching simple descriptions. But real questions often require deeper thinking, like understanding why something happened or recalling details from earlier. The authors created RVG-Bench, a new set of videos and questions designed to test if computers can handle these tricky reasoning tasks using both sound and pictures. They tested many current models and found they still struggle, showing there's room for improvement.
What this means in practice
- •For multimedia search developers: Develop video search tools that can handle complex queries requiring reasoning about audio and visual context to precisely locate events.
- •For media content analysis teams: Use RVG-Bench to evaluate and improve event detection models that integrate sound and vision for richer video understanding.
Authors
Yue Feng, Boshen Xu, Zihao Yue, Xiaokun Liu, Miao Yang, Xin Tao, Pengfei Wan, Qin Jin
Abstract
Existing temporal video grounding (TVG) benchmarks primarily focus on localizing target video segments according to descriptive text queries of the visual content. However, real-world queries are more diverse and often require the model to first understand the user's intent before localization, such as finding visual evidence, identifying the cause of an event, or recalling a remembered moment, yet such queries remain largely overlooked. To mitigate this gap, we introduce RVG-Bench, a benchmark for reasoning-driven audio-visual temporal grounding, where every query is constructed such that the target segment can only be identified through reasoning about the query's underlying intent. We carefully construct the benchmark and rigorously verify its data quality to ensure models truly comprehend the audio-visual content. RVG-Bench comprises 1,255 videos and 1,890 query-interval pairs across eight reasoning types. We evaluate 22 models and find that the strongest, Qwen3.8-Omni-Flash, achieves only 49.6 mIoU. Through a series of controlled analyses, including oracle-grounding probes and test-time scaling experiments, we systematically examine where and why current models struggle, providing insights for future model development.