Benchmark tests ai models for real time video and audio interaction
DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams
Artificial Intelligence
Summary
Many AI models struggle with keeping up natural conversations using video and sound because existing tests only check if they can answer questions at specific times. The authors created DIVE, a new test that watches how AI handles ongoing interactions with video and sound, like noticing visual clues or knowing when to talk or listen. They tested current AI and found while some respond quickly, they often miss chances to engage or adapt when new information appears. This benchmark helps AI builders understand and improve real-time interactive systems.
What this means in practice
- •For ai developers: Evaluate and improve AI systems for continuous video and audio interaction using detailed real-time benchmarks.
- •For user experience designers: Assess AI responsiveness and timing to enhance natural multimodal interactions in apps and devices.
Authors
Yuhang Chen, Guangyue Liu, Shilong Liu
Abstract
Recent advances in multimodal large language models (MLLMs) have substantially improved spoken interaction and video understanding. Existing online and streaming benchmarks primarily assess causal multimodal perception through timestamped question answering, whereas real-world interaction requires models to continuously identify task-relevant multimodal cues and decide whether, when, and how to respond. This leaves open how well current models can sustain real-time interaction throughout an evolving multimodal stream. To address this gap, we introduce DIVE, a benchmark for evaluating full-duplex interaction with visual engagement under continuous and realistic multimodal streams. It organizes six tasks into three complementary interaction scenarios. (1) Proactive Intervention evaluates whether models can identify emerging visual or semantic cues and intervene at the appropriate moment. (2) Continuous Engagement evaluates sustained participation through continuous visual tracking and visually grounded query interaction. (3) Interaction Coordination evaluates how models coordinate listening and speaking under overlapping and multi-party interactions. DIVE contains 525 audio/video samples with fine-grained human annotations and precise temporal metadata. Its evaluation protocol separates response quality, participation decisions, and response timing. We evaluate representative real-time models using response quality, window response rates, and reaction latency. Experiments show that leading real-time models achieve sub-second average response latency, yet their average window response rates remain at or below 53.77%. Response quality varies substantially across models and tasks, with adapting ongoing responses to newly arriving information remaining particularly challenging. We hope DIVE facilitates progress toward more natural and capable real-time multimodal interactive systems.