Audio text comparison tool describes differences between sound and words

AT-Differ: Dataset for Audio-Text Difference Captioning

Sound

Summary

It is important to know if a sound and its written description really match each other well. The existing tools only give a yes or no answer but cannot say how the sound and text are different. The authors created a new task and a special dataset to teach a computer model how to describe what is different between a sound and its text description. Training on this dataset helped the model improve at explaining those differences accurately.

What this means in practice

  • For audio software developers: Build tools that explain exactly how audio content mismatches its textual descriptions for better quality control.
  • For multimodal ai engineers: Improve models that link sound and language by training them to output human-readable difference explanations.

Authors

Akihiro Fujii, Yuki Okamoto, Yuki Saito, Hiroshi Saruwatari

Abstract

Evaluating audio-text semantic alignment is important in various audio-language tasks. Existing methods mainly quantify audio-text semantic alignment using a single score and cannot explicitly describe how audio and text differ. In this paper, we propose audio-text difference captioning, a task that generates textual descriptions of differences between audio and text. We also construct a new dataset AT-DIFFER containing audio-text pairs with systematically simulated differences and corresponding difference captions. Using AT-DIFFER, we fine-tune an existing multimodal large language model to describe differences between audio and text. The experimental results show that the original model has limited ability to describe such differences, whereas fine-tuning on AT-DIFFER substantially improves this ability.