Robotic system produces detailed sign language motions for humanoid use
RoboSTAR: Next-Scale Autoregressive Sign Language Translation for Humanoid Robots
Computer Vision and Pattern Recognition
Summary
Sign-language interpretation usually needs skilled human interpreters, which can be hard to provide everywhere. The authors created RoBoSTAR, a system that turns written text into detailed sign language motions that humanoid robots can perform. Unlike previous methods that treat the motion as one uniform sequence, RoBoSTAR builds the motion step-by-step, refining the signs with a mix of coarse and fine details. This approach helps make the robotic signing more natural and accurate. The system was tested and shown to work well both visually and quantitatively.
What this means in practice
- •For public communication providers: Create robotic sign language interpreters that can perform accurate, human-like signing for public accessibility.
- •For humanoid robot developers: Implement fine-grained sign language motion generation to enable humanoid robots to communicate using sign language.
Authors
Yujia Zeng, Chensheng Peng, Yuxin Chen, Alex Shao, Nathan Jew, Masayoshi Tomizuka
Abstract
Sign-language interpretation in public communication relies on qualified professional interpreters and can be difficult to scale, motivating robotic signing as a complementary accessibility interface. We present RoBoSTAR, a text-conditioned sign language production (SLP) framework for generating human-centric sign motion that can be retargeted for robotic execution, with speech supported optionally through an external ASR front end. Conventional autoregressive approaches flatten motion into a single full-resolution token sequence, forcing long-range and local dependencies to be modeled at a uniform temporal granularity. RoBoSTAR instead combines part-wise Finite Scalar Quantization with next-scale autoregression, generating motion over progressively finer temporal resolutions while predicting synchronized body and hand tokens in parallel within each step. This coarse-to-fine formulation provides compact long-range context before progressively refining motion details, while self-conditioning and context corruption improve robustness to cross-scale prediction errors. The generated motion is subsequently retargeted for physical humanoid execution. Extensive qualitative and quantitative evaluations are conducted to demonstrate the effectiveness of RoBoSTAR.