Closing the Verification Loop: Self-Check Captioning for Long-Paragraph Detailed Audio Captioning
2026-08-31 • Sound
Sound
AI summaryⓘ
The authors studied how to create very detailed, long descriptions of sounds and speech in videos, which was hard before because of lack of good data and problems with current methods. They made a new dataset called LACap-50k with lots of long audio clips and detailed captions, including transcripts checked by speech recognition. They also designed a new training method to improve how the model learns and a way for the model to check its own work when generating captions. Their approach leads to better results compared to other open tools and competes well with private systems, helping future research in this area.
audio captioningmultimodal language modelsautomatic speech recognitionsupervised fine-tuningself-checksemantic entropy collapseaudio-visual datasetsquestion answeringcaption verificationLACap-50k dataset
Authors
Fengji Ma, Yan Rong, Xu Li, Chen Zhang, Pengfei Wan, Li Liu
Abstract
Long-paragraph detailed audio captioning, which requires dense and transcript-faithful descriptions of fine-grained audio content, remains unsolved for current audio-visual multimodal language models. We attribute this failure to two structural problems. The first is data poverty, as no public corpus jointly provides long clips, paragraph captions, and verbatim-transcript fidelity. The second is generation-mode failure, evidenced by a 44.8 to 46.4 percentage-point gap between right-audio and shuffled-audio multiple-choice question (MCQ) accuracy. We address both within Self-Check Captioning (SCC), a unified framework that instantiates audio-grounded question answering as the verification primitive at every lifecycle stage. SCC yields three artifacts. Long-paragraph Audio Caption 50k (LACap-50k) is a 50,222-clip audio-visual corpus with 491.5-word captions and a post-hoc automatic speech recognition (ASR) audit. Layer-Curvature Supervised Fine-Tuning (LC-SFT) is the first on-policy supervised fine-tuning method to weight tokens by intermediate-layer evidence, motivated by our identification of Late-Layer Semantic-Entropy Collapse (SEC). SCC-Verifier arbitrates among caption rollouts via audio-grounded self-answering at inference. Across multiple benchmarks, our system attains state-of-the-art among open-source captioners and is competitive with proprietary baselines. We release LACap-50k to fill the resource gap for long-paragraph detailed audio captioning research.