Nonverbal signals reveal when people say no in conversations
Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue
Computation and Language
Summary
People usually understand 'no' when they hear it spoken, but there are also many nonverbal clues like facial expressions and gestures. The authors looked at conversations and found that you can detect when someone says no just by looking at these nonverbal behaviors without hearing the words. They discovered that facial movements were especially important for this, and even the other person’s behavior gave hints about negation, though less clearly. This means that detecting ‘no’ in dialogue isn’t only about the words but also about how people behave physically around that moment.
What this means in practice
- •For virtual reality developers: Improve avatar responsiveness by detecting negation cues from user’s facial and body movements in VR conversations.
- •For call center trainers: Use nonverbal behavior analysis to identify when speakers are rejecting or negating offers for better customer interaction training.
Authors
Leon Hammerla, Patrick Schrottenbacher, Alexander Mehler
Abstract
Negation is typically modeled through its linguistic realization, although spoken interaction is accompanied by tightly coordinated nonverbal behavior. We ask whether contexts centered on spoken negation cues contain measurable multimodal behavioral information: whether they can be distinguished from matched control contexts without lexical or acoustic input, where this information occurs in time, which modalities carry it, and whether it extends to the dialogue partner. We study 27 human-human interviews conducted in virtual reality, comprising temporally aligned gaze, facial, head, body, hand, and finger behavior and 964 annotated negation cues. Treating classification as a predictive probe, we compare 20 time-series models while excluding lexical and acoustic information, and then systematically vary temporal context, interactional source, modality availability, and event timing. Across grouped 10-fold cross-validation, the strongest probes reach up to .75 mean held-out AUROC from speaker-side behavior. Temporal analyses show that predictive information is concentrated around cue onset but remains detectable over a broader surrounding interval, while dialogue-partner behavior carries weaker predictive information with a comparatively diffuse temporal profile. Ablation and timing perturbations further show that facial features produce the largest modality-ablation effect and that the trained probe is sensitive to the temporal organization of the observed events.