Papers for

interactive media platform engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Compact audio visual agent cuts toolset and costs by 70 percent

Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

Abstract: Fine-grained audio-visual question answering (AVQA) requires models to resolve complex cross-modal correspondences and temporally grounding from sparsely distributed evidence. Omni-modal agents, which autonomously orchestrate perception tools to actively collect evidence, achieve strong performance, but it remains unclear whether active perception from both audio and visual modalities depends on broad specialized toolsets, especially audio or visual modality-specific tools. Such systems may also introduce long execution trajectories and substantial inference cost. We investigate whether comparable performance can be retained with a compact workflow exposing only essential perceptual capabilities. Starting from the specialized-tool OmniAgent, we conduct staged controlled experiments that prune the toolset and progressively consolidate modality-specific tools into omni-modal tools, motivating a two-tool agent comprising only Omni-modal captioning and Omni-modal QA. Across three widely used AVQA benchmarks, our compact agent achieves comparable or better accuracy than the specialized-tool agent while reducing inference cost by around 70%.

Wed 28 OctArtificial IntelligenceComputation and Language
The gist
Answering questions about events or scenes that combine sound and visuals can be tricky because it requires understanding details across both senses. The authors studied a system that uses many specialized tools for audio and visual analysis to answer these questions, but such systems can be slow and costly. They found that by carefully reducing these tools to just two general-purpose ones that handle both senses, the system keeps or even improves accuracy while running much faster. This means simpler, cheaper systems can still do complex audio-visual reasoning well.
Open → 2610.38960v1