HumanSkill 4D enables language-based selection and editing of 3D animatable avatars
HumanSkill-4D: Training-Free Language Querying of Animatable Gaussian Avatars in Space and Time
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
It can be hard to select specific parts or movements of 3D animated avatars just by giving a language instruction because parts look similar or move in ways that look alike. The authors developed HumanSkill-4D, a system that understands language queries linked to body parts, clothing, and poses without needing to be trained for each new avatar. It uses a shared knowledge base and geometric tools to identify and highlight the correct regions and motions reliably. Their system works quickly with little setup and performs better than previous methods in finding and selecting the right avatar parts and motion segments.
What this means in practice
- •For 3d animators: Select and edit specific avatar body parts or clothing using natural language commands during animation workflows.
- •For game developers: Inspect and trigger localized avatar motions or poses based on language queries to enhance character interaction testing.
- •For augmented reality app developers: Integrate natural language interfaces allowing users to select and manipulate animated avatar parts and movements in AR environments.$Commercial implications: Enables sale of AR apps with advanced language-driven avatar interaction features for consumer or enterprise use.
Authors
unsu Kim, Naeun Kim, Dongyoon Han, Seungryul Baek
Abstract
We introduce avatar referring segmentation, which selects regions of animatable Gaussian avatars from language queries. A query can also specify a pose or motion to select matching intervals. This lets users interact with avatars for local editing and motion inspection. However, making these selections with existing language Gaussian splatting methods poses two challenges: adjacent parts and garments are hard to separate and may be referred to indirectly (region ambiguity), while fine-grained motions can look alike in RGB (motion ambiguity). We address these challenges with HumanSkill-4D, a training-free framework that shares a multimodal large language model (MLLM) and geometric operators across avatars. For each avatar, a semantic memory links body-part and garment categories to persistent Gaussian regions and appearance descriptions. The MLLM uses category names and descriptions to select categories by appearance, relation, or function. To find matching intervals, it composes operators into programs that check query conditions on driving joint trajectories stored in a motion memory. These category choices and operator outputs make selections traceable, while composing or adding operators supports new motion conditions. To evaluate this task, we introduce three human-annotated benchmarks and show that HumanSkill-4D outperforms the baselines in part segmentation and 3D and 4D referring segmentation, with compact memories and only minutes of setup per avatar.