From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigation
2026-07-20 • Robotics
RoboticsComputer Vision and Pattern RecognitionHuman-Computer Interaction
AI summaryⓘ
The authors studied how to improve sign language motions generated for humanoid robots. They found that these motions often have body parts passing through each other, which causes problems when the robot tries to move. To fix this, they created a system that first adjusts the motions to avoid collisions without changing them too much. Then, they use a vision-language model to check the robot’s movements and make corrections when needed. This approach helps make robot signing motions more realistic and easier to execute safely.
sign language generation3D body representationhumanoid robotscollision detectioninverse kinematicsSMPL-X modelvision-language modelsmotion retargetingtask-space corrections
Authors
Nabeela Khan, Bowen Wu, Runwu Shi, Benjamin Yen, Takeshi Ashizawa, Carlos Toshinori Ishi, Takashi Minato, Kazuhiro Nakadai
Abstract
Recent sign language generation (SLG) systems increasingly output dense 3D body representations, which better preserve full-body kinematics and geometry for downstream embodiment on humanoid robots. However, these generated motions frequently exhibit self-intersections such as hand-hand and hand-torso penetration. While such artifacts may be tolerated in offline rendering, they become critical in humanoid execution as they lead to infeasible inverse-kinematics (IK) solutions, collisions, and unstable retargeted trajectories. We present a system-level framework that bridges SLG outputs to humanoid joint-space execution via two components. First, we introduce a volumetric SMPL-X collision-mitigation module that projects generated signing motions toward physically plausible configurations while minimally deviating from the original trajectory. Second, we propose a vision-language-guided retargeting algorithm built on an IK backbone: a VLM serves as a visual critic over rendered humanoid motion, identifies embodiment-specific failure modes, and triggers targeted task-space corrections. Our results highlight collision handling and perception-guided refinement as key missing components for reliable humanoid signing.