AffAdapt: AFFect-driven ADAPTive AI Personas for Seamless Conversations
2026-08-24 • Human-Computer Interaction
Human-Computer Interaction
AI summaryⓘ
The authors created AffAdapt, a system that helps AI characters talk and act more naturally by understanding when to speak, listen, and show emotions. It combines speech recognition, emotional understanding, and body language into one smooth process for better conversations. They tested it in sensitive practice talks and found it manages speaking turns well but still faces challenges like interruptions and keeping emotions aligned across different ways of communication. This design can be used in training or coaching where believable AI conversations are important.
AI personasspeech recognitionturn managementemotional statemultimodal interactionembodied AIresponse generationaffective computinghuman-AI conversationinteraction design
Authors
Nishanth Chidambaram, Kaustubh Paliwal, Kayla Hom, Shaoze Zhou, Chen Chen, Manas Satish Bedmutha, Nadir Weibel
Abstract
AI-generated personas are being increasingly used for support, training and simulations. While generative AI models possess abilities to generate affect-aware responses, their embodiment into visual personas is an active area of investigation. Naturalistic exchanges require understanding of the conversational partners' turn completions, whether the agent should respond or keep listening and rely on non-verbal cues aligned with one's emotional states. Seamless human-AI conversation in a multimodal setting requires all modalities being generated to act in coordination. We present AffAdapt, a seamless interaction design framework for AI-personas, which coordinates streaming speech recognition, proactive turn-management, persona-grounded response generation, a persistent emotional state, and synchronized embodied output into a single interaction loop. We demonstrate the architecture in the context of practicing sensitive, high-stakes conversations, and report an initial case study showing fluid turn management and adaptive, persona-consistent behavior, alongside open challenges in interruption handling, open-ended dialogue, and multimodal affective alignment. AffAdapt's interaction loop is a generalizable pattern for coordinating timing, identity, and affect in real-time AI personas - applicable to training, coaching, education, and simulation contexts wherever believable, responsive interaction matters.