Intentional agent method improves humanlike facial reactions in ai systems
Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation
Artificial Intelligence
Summary
Making computers show natural facial reactions is important for better conversations with humans. The authors propose a new method that makes AI agents think internally before showing expressions, rather than just copying what they see. Their approach uses a model that updates an inner emotional state even during pauses, leading to more realistic and timed facial reactions. Tests with human judges showed their system’s expressions were as believable as real ones, better than some older methods.
What this means in practice
- •For ai developers: Build conversational agents that generate realistic and context-aware facial expressions based on internal emotional modeling.
- •For virtual reality designers: Create immersive virtual characters with believable facial responses that evolve naturally even during silent moments in dialogue.
Authors
Hanzhong Zhang, Jindong Wang, Siyang Song
Abstract
Automatic human-like facial reaction generation (FRG) is essential for building intelligent systems that can engage in human-computer interaction (HCI). While diverse and context-appropriate facial reactions can reflect latent appraisal and affective processes in human interaction, most existing FRG methods rely on end-to-end architectures that directly map speaker behaviours to listener expressions without an explicit intermediate internal state. We reformulate FRG as generation mediated by a structured internal-state process and propose the \textbf{Intentional Agent}, which shifts FRG from direct stimulus-response mapping to stimulus-grounded generation through explicit intermediate states. To represent temporal internal-state evolution, we propose an internal dynamics model that integrates emotional drives with an iterative Inner Thought Flow (ITF) within a structured intermediate state used for subsequent generation. This state can continue to update during conversational silences. Furthermore, to bridge abstract internal states with physiological actions, we formulate FRG as a downstream affective mapping from this latent thought flow to facial expressions. Experiments on the REACT 2025 dataset show an FRDist of 72.39 and an FRDiv of 0.5057; perceptual plausibility is evaluated separately through blinded human ratings. A blinded human evaluation of 96 reactions found no significant difference in mean score between Full and ground truth ($5.527$ vs.\ $5.195$, $p_{\mathrm{Holm}}=.076$), while Full significantly outperformed Event-Triggered and Heuristic-Only (both $p_{\mathrm{Holm}}<.001$). The Reaction Quality Scorer (RQS) correlated strongly with human judgements (Pearson $r=.855$; Spearman $ρ=.821$, both $p<.05$), supporting its use as an automatic metric. These results underscore the immense potential of endogenous dynamics in building highly autonomous, human-like agents.