Audio systems segment and label extremely long recordings accurately
Logbook: Extremely Long-form Audio Event Understanding
Artificial IntelligenceComputation and LanguageMachine LearningSound
Summary
Most audio recognition systems work only on short sound clips, but this research introduces a new challenge to recognize events in recordings lasting from minutes up to days. The authors created Logbook, a benchmark where models must split long audio into segments and label each with an event name and description. They tested many systems, finding that while the task is possible, current systems still don't match human quality, often splitting recordings into too many segments. Fine-tuning helps reduce mistakes, and models that learn end-to-end often perform better except when handling very long audio.
What this means in practice
- •For security monitoring teams: Automatically identify and segment events over many hours of audio for continuous surveillance without manual clipping.
- •For media archive managers: Label and index long audio recordings like radio shows or podcasts with event descriptions to improve search and retrieval.
Authors
Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk, Guitang Lan, Chao-Wei Huang, Mohammad Sadegh Rasooli, Sangeeta Srivastava, Zhaojiang Lin, Saurabh Adya, Ming Sun
Abstract
Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we introduce Logbook, a benchmark for hour-scale audio understanding, with recordings ranging from ten minutes to six days. Given a continuous audio recording and an event label vocabulary, a system must predict a gap-free segmentation with an event label and a description per segment. We compare 52 systems, end-to-end and cascaded, and ablate fine-tuning, context length, and reasoning budget. We find the task tractable, though the best systems remain below the human reference. Also, over-segmentation is pervasive, and fine-tuning partially mitigates it. Finally, end-to-end are often better than cascaded systems, but degrades with longer context.