Audio models miss speaker reactions in controlled sound scenes
CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound
SoundMachine Learning
Summary
Deciding which sounds in a recording are important to describe is tricky because people often disagree. This paper defines important sounds as those that cause people in the recording to react audibly. The authors created a large, controlled dataset of scenes with two speakers and sounds that cause reactions, using language models to write dialogues. When testing existing audio-language models, they found the models can recognize sounds but often fail to understand how speakers react to them.
What this means in practice
- •For audio software developers: Improve audio scene description tools by incorporating speaker reactions to better represent important sounds in recordings.
- •For assistive technology teams: Develop more accurate audio captioning systems that reflect human-like understanding of sound events and speaker responses.
Authors
Marcel Gibier, Thomas Thebaud, Olivier Boëffard, Jean-François Bonastre
Abstract
Automatic audio scene description turns a recording into a text account of a situation. One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all. Annotators disagree about this, making a ground truth hard to obtain. In this work, we first define the ground truth, then generate the data. We focus on audio events and define sound salience with a simple rule: a sound is salient when a speaker audibly reacts to it. For scale and variety, a controlled set of scenarios fixes the ground truth, and a language model writes the dialogues. The resulting corpus, CARES, contains 10,000 two-speaker scenes. We then benchmark six audio-language models on three tasks: identifying the scene, tagging the sounds present, and classifying reactions. We show that these models hear the sounds but miss how the speakers react to them.