Jarvis speaks up proactively in group conversations when needed
Jarvis: A Proactive Speech Agent for Multi-Party Conversations
Human-Computer InteractionArtificial Intelligence
Summary
Most speech assistants only talk when directly asked and only to one person. This paper presents Jarvis, a speech agent that listens to group conversations and speaks up only when it notices the group is wrong or missing information. Jarvis uses a shared document to verify facts and waits a few turns before intervening so it doesn’t interrupt unnecessarily. The authors created a special test dataset and showed that Jarvis is usually correct and rarely interrupts when the group fixes itself. They also tested Jarvis live with people and saw similar results.
What this means in practice
- •For virtual assistant developers: Create assistants that monitor group talks and help by correcting facts only when needed.
- •For conference support teams: Provide live fact-checking agents that intervene during multi-speaker discussions to improve accuracy.
- •For customer service call centers: Deploy agents that listen to multi-person calls and interject to correct critical errors, improving communication quality.$Commercial implications: Enables selling advanced speech agents that improve multi-party call accuracy and reduce misunderstandings in service interactions.
Authors
Seunghyun Oh, Hirotaka Hiraki, Shuyue Stella Li, Yulia Tsvetkov, Shyamnath Gollakota
Abstract
Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among several people and speak up only when it can help. We introduce Jarvis, a real-time proactive speech agent that audibly participates in multi-party human conversations. Grounded in a document shared beforehand, Jarvis follows the discussion and intervenes when the group misses or misstates a fact and does not correct itself within a few turns. We make three contributions: a problem setting based on epistemic breakdowns that makes proactive intervention measurable, realized as CHI-180-proactive, a synthetic multi-party dataset seeded with known gaps, errors, and self-corrections; a proactive backbone that harnesses a small, open-weight model with deterministic checks and grounds every claim in a source sentence; and interaction techniques for taking the floor in live speech and showing the cited evidence on screen. On CHI-180-proactive, Jarvis is correct on most events it addresses and stays silent 97% of the time when the group resolves an issue itself. A live study with 23 participants confirms these trends with real-time interventions.