Expressive robot gestures generated and adapted using multimodal AI models
Context-Aware Generation and Modulation of Expressive Motion Behavior using Multimodal Foundation Models
Robotics
Summary
Robots that move expressively can make interactions with people more engaging and effective. This paper presents a method for creating complex robot gestures by using information from text, audio, and video together. The method builds on animation principles and can adjust motions based on the situation while working on different robot types. The authors tested their approach with people and found that the gestures matched the context better than other methods. The system also runs efficiently to work in real time.
What this means in practice
- •For robotics engineers: Create robots with adaptive, expressive full-body gestures that better fit the interaction context using combined text, audio, and video inputs.
- •For entertainment animators: Automatically generate and modify character motions on different platforms to enhance storytelling and engagement based on multimodal cues.
Authors
Till Hielscher, Fabio Scaparro, Kai O. Arras
Abstract
Expressive robot motion positively impacts human-robot interaction by improving user engagement, likability, or task performance. We present a novel approach to automatically generate and modulate complex, expressive full-body gesture sequences from a multimodal (text, audio, video) context description. Based on a unified mathematical implementation of the Principles of Animation using Dynamic Movement Primitives, we use multimodal foundation models to generate such sequences along with parametric motion variations that are highly context-aware. This method extends the state of the art in terms of flexibility and generality by being interpretable, composable, and working across different robot morphologies. Moreover, we integrate the system into a continuous control framework and leverage knowledge distillation to learn a much smaller model, significantly improving token efficiency and system latency. Results from a user study with a human-like platform indicate that participants judged our system's motions to be better aligned with the interaction context than motions produced under a non-modulated or API-level condition. We also demonstrate how the system generates expressive motion for robots with different kinematics, showcasing its versatility. Paper webpage: https://gen-mod-expressive-motion-behavior.github.io/.