ACE-Cap: Active Evidence Acquisition via Agentic Co-Evolution for Long-Paragraph Fine-Grained Audio Captioning
2026-08-17 • Sound
Sound
AI summaryⓘ
The authors address the problem of creating detailed audio descriptions without missing important sounds or adding wrong details. They introduce ACE-Cap, a system where two parts—a Composer and an Instruct model—work together by asking and answering questions about the audio in multiple steps instead of just writing one quick caption. This back-and-forth helps the system decide what sounds need more attention, when it has enough information, and how to make a complete final description. Their approach uses special training techniques to improve this interactive process and make it more accurate.
audio captioningfine-grained descriptionevidence acquisitionmulti-turn interactionComposer modelInstruct modelpolicy optimizationLOOP-GRPOreward assignmentcaption synthesis
Authors
Fengji Ma, Yan Rong, Xu Li, Xuenan Xu, Chen Zhang, Li Liu
Abstract
Long-paragraph fine-grained audio captioning requires models to recover diverse acoustic facts while avoiding omissions and unsupported details. However, prevailing captioners remain passive one-shot generators: once a detail is overlooked, they cannot identify the evidence gap, query the audio for targeted information, or decide when sufficient evidence has been collected. We formulate this task as active evidence acquisition and introduce Agentic Co-Evolution for Captioning (ACE-Cap). The framework uses multi-turn interaction between a Composer and an Instruct model to form a closed evidence-acquisition loop. A Captioner first produces an initial description. Conditioned on this description and the interaction history, a text-only Composer asks targeted questions about unresolved acoustic attributes, while an audio-conditioned Instruct model provides grounded answers. The Composer then decides when to terminate and synthesizes the accumulated evidence into a final caption. ACE-Cap trains these roles through a unified gold-to-prediction reward derived from fixed, gold-grounded multiple-choice questions and a frozen caption-only judge. For credit assignment in variable-length interactions, LOOP-GRPO replaces the trajectory-wide scalar advantage with span-aligned signals: leave-one-out contributions of individual questions to the accumulated evidence, a quality-cost utility for stopping, and an evidence-preservation utility for final synthesis. Role-wise warm-up followed by alternating Composer and Instruct optimization keeps each update a well-defined single-policy problem while allowing the roles to co-evolve. ACE-Cap thus turns captioning from passive one-shot generation into an adaptive process that learns what evidence to acquire, when to stop, and how to preserve it in a long-paragraph caption.