AudioMap: Cloze-and-Choice Reinforcement Learning for Time-Aware Dense Audio Captioning
2026-08-10 • Sound
Sound
AI summaryⓘ
The authors focus on a task called time-aware dense audio captioning, where the goal is to describe multiple sound events and their attributes precisely with time stamps. They identify problems with earlier methods and reinforcement learning approaches, especially difficulties in fine-grained rewards and linking captions to exact times. To fix this, they developed AudioMap, a new system that uses clever reward designs to ensure more detailed and accurate descriptions tied to time. They also created a large new dataset, AudioMapCap-44K, with many precise audio captions to help train and test their approach. Their experiments show that AudioMap performs very well compared to other models.
time-aware dense audio captioningreinforcement learningreward designaudio event detectiontemporal IoUcurriculum learningfine-grained audio descriptionaudio captioning dataset
Authors
Yan Rong, Fengji Ma, Xu Li, Jinting Wang, Chen Zhang, Li Liu
Abstract
Time-aware dense audio captioning (TDAC) aims to generate multiple fine-grained attributes (dense) of the audio with precise time boundaries (time-aware). Existing methods struggle to achieve these two goals and mainly rely on supervised fine-tuning, yielding sub-optimal performance. While reinforcement learning (RL) shows promise, applying it to TDAC faces two main challenges: (1) existing rewards are too coarse to supervise multi-event, multi-attribute, and multi-relation descriptions in a fine-grained manner; and (2) temporal supervision is difficult for free-form captions, where flexible event-time expressions make reliable event-time correspondence challenging. To address these challenges, we propose AudioMap, a novel RL-based TDAC framework, which shifts to a unified cloze-and-choice reward paradigm. Specifically, we introduce the Evidence Sufficiency Reward (ESR) with an asymmetric hierarchical scoring mechanism to promote fine-grained accuracy and descriptive richness across diverse acoustic dimensions. Furthermore, we design the Event-Conditioned Temporal Reward (ECTR) to structurally bind timestamps to event semantics via temporal IoU, accompanied by a dual-curriculum learning strategy to facilitate the training process. Finally, to support this task, we construct the first time-aware fine-grained audio captioning dataset, AudioMapCap-44K, which contains 44K carefully annotated captions. Extensive experiments across diverse benchmarks show that AudioMap achieves state-of-the-art (SOTA) performance among open-source models and delivers competitive or superior results relative to proprietary models. Project page and release updates are available at https://github.com/ryysayhi/AudioMap.