Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production
2026-08-10 • Computation and Language
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern RecognitionMultimedia
AI summaryⓘ
The authors created a new system called Uni-SLTP that can both understand sign language and produce sign language from text. Unlike past work that only focused on understanding sign language, their system works in both directions: from sign to text and from text to sign. They designed a special tokenizer to translate continuous sign motions into tokens and a single model that generates sequences either way. Tests show their system makes accurate sign motions and keeps good translation quality. This helps unify different sign language tasks into one framework.
Sign Language RecognitionSign Language TranslationSign Language ProductionTokenizerAutoregressive ModelSequence GenerationModality GapContinuous Sign LanguageDiscrete Text TokensConditional Generation
Authors
Xiao Liu, Shiwei Gan, Yafeng Yin, Jiaxin Yin, Bowen Guo, Yaqi Sun, Zhiwei Jiang, Lei Xie
Abstract
Recent advances in sign language (SL) research have shown a trend toward unifying multiple sign language understanding (SLU) subtasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), within a single framework, leading to substantial progress. Meanwhile, sign language production (SLP), which generates sign sequences from text, has also attracted growing attention. This naturally raises an important question: can sign language understanding and production be unified within a single framework? Compared with unifying SLU subtasks, this problem is substantially more challenging. Existing SLU tasks largely share the same direction of mapping, namely from sign inputs to linguistic outputs, whereas SLT and SLP lie in opposite directions of sign-text mapping. A unified framework must therefore address two key challenges: (1) bridging the modality gap between continuous sign motions and discrete text tokens through a shared sign tokenizer that supports both linguistic abstraction and motion reconstruction; and (2) learning a single conditional autoregressive model that can take either sign or text as input and generate the corresponding target sequence in the opposite modality. To this end, we propose Uni-SLTP, a unified framework for SLT and SLP with two key components: (1) a shared sign tokenizer that converts sign sequences into discrete tokens and latent representations, capturing both semantic and reconstructive information; and (2) a unified autoregressive generation model that formulates both tasks as conditional sequence generation. Experiments on widely used public datasets show that Uni-SLTP achieves superior motion accuracy for SLP while maintaining competitive SLT performance.