Humanoid robot sign language benchmark tracks precise motion errors
Traceable Human-to-Humanoid Sign Language Benchmarking
Robotics
Summary
Turning human sign language videos into robot motions is tricky because small errors can affect the robot’s signing. The authors created a large dataset with thousands of Chinese Sign Language sentences, showing different stages of motion processing from humans to robots. Their approach helps track and separate errors at each step to improve accuracy. This makes robot sign language movements clearer and easier to evaluate.
What this means in practice
- •For robotics engineers: Develop and test more accurate humanoid robot sign language motions using a detailed dataset that isolates errors.
- •For human-computer interaction teams: Improve communication interfaces by integrating humanoid robots that can physically perform sign language with higher fidelity.
Authors
Ao Liu, Shengeng Tang, Lechao Cheng, Yanbin Hao, Bingkun Bao, Richang Hong
Abstract
Sign data collection is costly, and teleoperation scales poorly, motivating reuse of large video corpora. Humanoid signing requires converting video-derived human motion into robot trajectories while preserving linguistic motion cues. Errors from fitting, human-motion repair, retargeting, robot geometry repair, and control are hard to separate from the final trajectory alone. We introduce HumanoidCSL-20K, a dataset and benchmark of 20,648 sentence-level Chinese Sign Language sequences, each with four aligned versions: the source, the repaired human motion, the direct robot reference, and the geometry-repaired robot reference. Observation-supported local human-motion repair, full-robot geometry repair, and cross-representation provenance make each transformation traceable. Paired evaluations measure human-motion continuity and content preservation, robot-reference feasibility, and physical execution. A sign-specific kinematic-reference protocol scores handshape, location, palm orientation, and inter-hand relation over the full planned motion. Full-corpus results show fewer abnormal arm / hand steps and less inter-hand and hand-body penetration after repair. Control experiments separate reference learnability from curriculum effects, while component scores expose remaining execution errors.