SignGPT unifies sign language translation and generation using AI models
SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation
Computer Vision and Pattern Recognition
Summary
Communicating in sign language with computers usually requires different tools for understanding and producing signs. The authors created SignGPT, a single system that can both translate sign language into text and generate sign language from text, without relying on written sign annotations called glosses. This system uses detailed models of body, hand, and facial movements to understand and create signs. They tested SignGPT on American and German sign language datasets and showed it can help enable conversations using sign language mediated by AI.
What this means in practice
- •For sign language technology developers: Develop integrated tools that both understand and produce sign language without switching models, improving user experience.
- •For communication device manufacturers: Build devices that facilitate bidirectional sign language conversations mediated by AI for Deaf users.$Commercial implications: Enables AI-powered conversational interfaces that produce and interpret sign language for communication devices targeting Deaf consumers.
Authors
Ronghui Li, Jun Dong, Zhongyuan Hu, Zunnan Xu, Jun Zhou, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, Jie Guo, Xiu Li, Linchao Bao
Abstract
Large language models (LLMs) provide limited support for sign language interaction. Unifying sign language translation (SLT) and generation (SLG) to enable sign language as both input and output can reduce switching between separate models during sign-text interaction. We present SignGPT, a unified, pose-based framework for gloss-free SLT and SLG. SignGPT integrates part-aware hierarchical representations of body, hand, and facial motion into a shared language model and employs asymmetric multi-token prediction and progressive training for bidirectional modeling. We evaluate SignGPT on How2Sign (ASL) and Phoenix-2014T (DGS) through benchmark comparisons, qualitative analyses, and component ablations. An exploratory study with 12 Deaf ASL signers assesses an LLM-mediated sign-to-sign response pipeline, highlighting the potential of unified modeling to support sign language conversation (SLC).