Acoustic features predict speaker familiarity and kinship in conversations
Dyship: Leveraging Acoustic Features For Speaker Relationship Classification From Dyadic Conversations
Computation and LanguageArtificial Intelligence
Summary
The paper studies how different types of speech sounds help identify if two people talking know each other well or are related. The authors test five ways to represent speech, like spectrograms and specialized speech feature vectors, on a dataset of two-person conversations. They find some features work better for recognizing familiarity, while others are better for kinship. The results often beat previous human and machine attempts on the same dataset. This shows everyday speech sounds contain useful clues about relationships between speakers.
What this means in practice
- •For voice biometric engineers: Improve authentication systems by detecting speaker relationships to add an extra security layer in conversational access controls.
- •For social robotics developers: Enhance robot understanding in human interactions by identifying familiarity and kinship cues from speech to tailor responses accordingly.
Tested on one dataset.
Authors
Kieran Sturges, Thomas Thebaud, Yaohan Guan, Najim Dehak
Abstract
We introduce DyShip, a framework for classifying speaker familiarity and kinship from dyadic conversations in the Seamless Interaction corpus. We compare five speech representations: spectrograms, MFCCs, self-supervised WavLM representations, x-vectors, and estimated articulatory movements. Spectral features perform best for familiarity classification, reaching an accuracy of 87.81%, while WavLM achieves the highest kinship accuracy of 85.17%. X-vectors and articulatory features yield weaker results on familiarity classifiaction, but shine for kinship classification, with an AUC of 0.909 for x-vectors. Most classifiers exceed previously reported human and audio language model scores on the same dataset. These findings highlight the potential of conventional acoustic features for relationship inference and motivate further investigation of the conversational cues captured by different representations.