Conversational system refines text guided motion with multi agent approach
Learn to Modify: Self-evolving Multi-agent Refinement Framework for Conversational Text-to-Motion Generation
Computer Vision and Pattern Recognition
Summary
When making animations from text, it can be hard to fix small details without remaking the whole animation. The authors created a system that lets users give feedback and only changes the parts they want fixed, keeping the rest the same. Their system uses multiple specialized computer agents that work together to understand instructions, edit motions, and check quality. It also learns from past fixes to get better over time. This approach helps make editing animations faster and more accurate.
What this means in practice
- •For animation studios: Create animated motion sequences that can be interactively refined through conversation without regenerating entire animations from scratch.$Commercial implications: Enables animation software to offer user-friendly conversational editing features improving efficiency and precision in production workflows.
- •For game developers: Incorporate conversational refinement tools to modify character motions on demand, maintaining context and reducing development time.
Authors
Jianan Li, Jiawei Zhao, Yangtao Zhou, Zhifu Zhao, Shaoxin Wu, Yutao Wang, Dezhen Wang, Hua Chu, Fu Li
Abstract
Despite rapid progress in text-to-motion generation, existing methods are commonly designed for single-shot generation. When users are dissatisfied with specific motion details, they are often forced to regenerate the entire sequence from scratch, which is computationally inefficient and potentially destroys previously established kinematic context. A practical generation system should instead support iterative user feedback and refine only the affected spatio-temporal regions while preserving the remaining motion. Therefore, we introduce a conversational motion generation paradigm that requires the system to coordinate instruction understanding, motion generation, localized editing, and quality assessment. Recent advances in LLM-driven multi-agent systems provide a promising foundation for such role-specialized coordination. To tackle these challenges, a Self-evolving Multi-Agent RefinemenT Framework (SMART) is proposed for conversational motion generation. Within SMART, to preserve semantic consistency, we design a semantic-guided masked motion editing mechanism, which precisely grounds natural language feedback into spatiotemporal motion regions to enable localized refinements without altering unedited regions. To reduce redundant exploration, we propose a memory-based self-evolving strategy that dynamically accumulates motion editing knowledge from interaction history and reuses past successful experience. Extensive experiments demonstrate that SMART effectively supports multi-turn conversational motion generation and enables efficient motion refinement.