Hybrid retrieval boosts accuracy in analyzing long audio recordings
HARP: Agentic Hybrid Retrieval and Analysis for Long-Form Audio
Sound
Summary
Long audio recordings can be tricky to analyze because important information is spread out and sometimes found in sounds rather than just words. The authors created HARP, a system that combines searching with keywords and sound features to find evidence in audio more effectively. Their system performs better by using both text and audio clues, improving answer accuracy by about 10%. They also show that just looking at right answers isn't enough to judge performance—the reasoning behind answers matters too.
What this means in practice
- •For podcast producers: Find specific audio content and supporting evidence across lengthy podcast episodes for detailed content summaries or fact-checking.
- •For legal transcription teams: Improve retrieval of acoustically relevant evidence from long court recordings by combining text-based and audio-based searches.
Authors
Chin-Jou Li, Masao Someki, Woojeong Jin, Yashish M. Siriwardena, Tanmay Laud, Shanil Puri, Shinji Watanabe
Abstract
Long-form audio analysis requires systems to localize and integrate evidence distributed across extended recordings. While existing work primarily retrieves semantic content through structured textual representations, many real-world queries depend on acoustic evidence that is better preserved in continuous representations or raw audio. We introduce HARP (Hybrid Audio Retrieval Pipeline), an agentic framework and benchmark for systematically studying retrieval and evidence representations in long-audio analysis. Hybrid retrieval combining keyword and vector search shows the most robust performance. When paired with both metadata and retrieved audio as evidence, average answer accuracy improves by around 10% and rationale accuracy by around 6% over single-modality retrieval and evidence. Fine-grained evaluation shows that answer accuracy alone overestimates system capability and that HARP mostly follows human performance trends across query types. These results highlight the importance of combining structured retrieval with flexible access to audio evidence and evaluating long-audio systems beyond answer accuracy.