Papers for

media indexing teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

OmniCompass benchmark measures video AI understanding with audio and vision

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

Abstract: With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly important. Existing video understanding benchmarks cover only part of the evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding. We introduce OmniCompass, a holistic benchmark for omni-modal video grounding and understanding. OmniCompass defines two evaluation tracks: OmniCompass-VL assesses vision-language models on tasks requiring visual evidence, while OmniCompass-AV assesses omni-modal models on tasks requiring joint audio-visual evidence. Each track includes three complementary tasks: temporal grounding (TG), spatial grounding (SG), and question answering (QA), assessing when a referenced target appears, where it is, and what happens. QA covers five perception and three reasoning categories in OmniCompass-VL, and three perception and two reasoning categories in OmniCompass-AV. OmniCompass-VL contains 820 videos and 2,460 instances across domains and three duration types: short, medium, and long. A narrative- and audio-rich subset forms OmniCompass-AV with 240 videos and 720 instances. Each instance is evaluated under two paired settings: text-only and text+image, specifying the same target through text and visual-language guidance. All instances are manually annotated with multi-round quality verification. Evaluations of 50 MLLMs show that open-source models still lag behind proprietary counterparts, particularly in audio-visual understanding. Precise temporal and spatial grounding remains challenging even for leading models, and their predictions still exhibit inconsistencies across the paired target specifications.

Tue 13 OctComputer Vision and Pattern RecognitionComputation and LanguageMultimedia
The gist
Understanding videos with both images and sounds is hard for AI, and existing tests only check parts of this skill. The authors created OmniCompass, a big test that looks at how well AI can find moments, spots, and answer questions about videos using text, images, and sounds. They tested many AI models and found even the best ones still struggle with precise timing, location, and combining audio and visual information. This benchmark helps show where AI video understanding needs to improve.
Open → 2610.18829v1

Active speaker detection improves with joint audio and face modeling

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

Abstract: Active speaker detection (ASD) requires reliable association between visible faces and acoustic speech, yet existing systems often degrade under challenging domains or incomplete observations. We introduce ROAM-ASD, a robust audiovisual framework that jointly models audio, full-face, and fine-grained mouth representations. A unified joint self-attention mechanism processes all input streams together with modality-agnostic query tokens, enabling direct interaction among available modality inputs. Modality dropout further improves robustness when input streams are unavailable. ROAM-ASD achieves state-of-the-art performance across five ASD benchmarks: 98.8% mAP on WASD, 87.9% on UniTalk, 96.5% on AVA, 99.3% on ASW, and 98.2% on Talkies, improving over previous best systems by 5.1, 4.7, 0.9, 1.0, and 2.1 mAP points, respectively. ROAM-ASD also substantially improves zero-shot cross-dataset generalization and remains robust to missing observations.

Tue 22 SeptMultimediaComputer Vision and Pattern RecognitionSound
The gist
Active speaker detection identifies who is speaking by linking what you hear with the faces you see. Existing methods struggle when video or audio is unclear or incomplete. The authors introduce ROAM-ASD, which looks at full faces, mouth details, and sounds together using a system that pays attention to all these inputs at once. This approach is better at detecting speakers even when some information is missing, working well across several test collections.
Open → 2609.26648v1