Multi-agent ai improves inference of speaker relationships in conversations
Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference
Multiagent SystemsComputation and LanguageSound
Summary
Understanding how people relate to each other by listening to conversations can be hard and taking time to train computers for this is expensive. The authors created a way for multiple AI agents to discuss and argue about relationships during conversations without extra training, making better guesses. They tested two approaches where agents take different roles or compete to find the best answer. Their method worked better than older ways, sometimes even beating people when text was included, but it finds sound cues harder to understand. This shows AI can benefit from teamwork when figuring out relationships in speech.
What this means in practice
- •For conversational ai developers: Enhance virtual assistants to better infer interpersonal relationships from conversations without needing task-specific retraining.
- •For call center support teams: Improve automated analysis of customer-agent interactions by detecting social relationships to better tailor responses or escalate issues.
Authors
Yaohan Guan, Yen-Ju Lu, Yuzhe Wang, Junhyeok Lee, Jesus Villalba, Laureano Moro Velazquez, Thomas Thebaud, Najim Dehak
Abstract
Inferring speaker relationships from spoken conversations is an important step towards socially aware speech understanding. However, this task remains underexplored, and supervised modeling is costly to train and scale. At the same time, existing inference-time LLM approaches provide limited structure for handling subtle, distributed, and multimodal relational cues that may support multiple plausible interpretations. To address these limitations, we introduce a training-free multi-agent reasoning framework that organizes inference through structured interaction among LLM agents, allowing relationship judgments to be proposed, challenged, and adjudicated without task-specific training. We instantiate this framework with two complementary designs. We propose Multi-Role Multi-Agent Debate as a task-specific adaptation of standard multi-agent debate for speaker relationship inference, assigning agents complementary roles or social-theory-grounded perspectives rather than a single undifferentiated viewpoint. In contrast, we introduce Multi-Agent Compete, a competition-based protocol that compares agent judgments through pairwise adjudication, eliminates weaker candidates, and retains the most defensible one. We evaluate these methods on the Seamless Interaction dataset across different modality settings, covering both binary classification and fine-grained relationship-detail prediction. Results suggest that they improve over zero-shot and existing multi-agent baselines in most cases. Human evaluation further suggests that this task is challenging even for people. LLM methods can sometimes outperform human annotators in text-included settings but are less competitive in the audio setting. Together, these findings suggest that relationship inference benefits from structured inference-time interaction among agents, while acoustic cues are not yet fully captured by current models.