Real-time agent tracks and guides emotions across conversations
PAIR: Perceptual Affective Inference and Regulation in a Real-Time Multimodal Conversational Agent
Human-Computer Interaction
Summary
People often need emotional support that remembers their feelings over time. The authors created PAIR, a computer agent that guesses how someone feels during a conversation and offers advice using speech, colors, and an avatar. It keeps track of past talks to understand context better. In a two-week study, the agent’s guesses about feelings mostly matched people’s own reports, especially for positive or negative feelings. Over time, users found the help more useful and the conversations more natural.
What this means in practice
- •For mental health app developers: Create conversational agents that recognize and respond to users’ emotions over multiple sessions to provide ongoing emotional support.$Commercial implications: Enables personalized emotional support features in mental health apps to improve user engagement and care quality.
- •For virtual customer service teams: Equip chatbots with emotion tracking and regulation to make interactions more empathetic and responsive to customer feelings in real time.
Authors
Kexin Quan, Zijian Ding, Jiaye Yong, Qinshi Zhang, Dong Wang, Jessie Chin
Abstract
Sustained emotional support requires generative agents to connect momentary emotion inference and regulation with continuity across encounters. We present PAIR (Perceptual Affective Inference and Regulation), a real-time multimodal agent that reconstructs how an event is appraised into an emotional state. Appraisal scaffolds produce a valence-arousal-dominance estimate and select regulation guidance, delivered through conversation with coordinated speech, color, and avatar cues. Rolling memory carries context across sessions, and the scaffold re-runs after guidance. In a 14-day deployment with 19 participants, 1,093 sessions paired initial and post-guidance estimates with unanchored self-reports. Initial valence reached MAE 1.20 on the 9-point SAM scale (r=.68), dominance reached MAE 1.30, and arousal showed weak agreement even after coarsening. Self-reported emotional change varied with initial state, with the largest valence increases in sessions that began at negative valence. Perceived understanding was associated with greater valence increase and showed little correspondence with numerical prediction error. Over two weeks, helpfulness increased while input shortened; interviews traced personalization and companionship to relevant recall, context updates, and familiar dialogue. These findings connect inference accuracy to conversational and temporal patterns of support through per-event, first-person evaluation.