Multi-party backchannel prediction struggles with unseen listeners
Multi-Party Backchannel Prediction: a Diagnosis, a Benchmark, and a Ceiling
SoundArtificial Intelligence
Summary
Backchannel cues are the small signals people give when listening during conversations, like nodding or saying "uh-huh." This paper finds that predicting these signals works well when only two people talk but fails when more people are involved, especially if the listener is new to the system. The authors created a new dataset from real meetings with many speakers to study this problem. They show that while some acoustic features help, understanding individual listener differences remains a big challenge for automatic prediction.
What this means in practice
- •For conversational ai developers: Use the new multi-party backchannel benchmark to improve models that predict listener responses in group meetings.
- •For human-computer interaction designers: Incorporate insights about listener individuality and acoustic cues to design better interactive systems for multi-person conversations.
Authors
Mohammed Hafsati, Ahmed Loughzali
Abstract
Backchannel prediction has been studied almost entirely in dyadic conversation. We introduce a multi-party benchmark based on the AMI corpus, comprising 682 masked-listener views from 171 meetings, 190 speakers, and 18,697 backchannel events, with a person-disjoint held-out split. A state-of-the-art dyadic model applied zero-shot to meeting audio performs at chance (AUROC 0.499); nevertheless, its frozen acoustic features remain informative: a linear probe reaches 0.704, and retraining the predictor raises performance to 0.751. Retraining reveals a second limitation. Listener conditioning improves prediction for listeners seen during training but not for unseen listeners, and the gap remains under capacity reduction, listener-adversarial training, per-listener adaptation, and oracle lexical conditioning. Adversarial training removes only part of the speaker-identity information, while stronger removal hurts prediction, suggesting that identity is entangled with cues that are useful for backchanneling. A within-model control helps explain this pattern: with the same features and data splits, turn-onset prediction transfers to unseen listeners, while backchannel prediction does not. Backchannel rates also vary about twice as much across individuals as turn-onset rates. Since backchannels occupy only about 1% of frames, frame-level F1 is strongly affected by the base rate. We therefore report AUROC alongside event-F1 on listener-active regions. We release the benchmark and evaluation tools at https://github.com/HafsatiMohammed/bc_multiparty_release.